The root cause is likely an infinite loop or corrupted data somewhere, but not too many
vmcore dmessages. The fix of limiting vmcore-dmesg.txt size is mainly to defend kdump
servers against the existing problem, as well as other future problems. It doesn’t prevent
us from further investigating the root cause of the problem that is hard to reproduce
either.
Thanks,Jun
Sent from Yahoo Mail for iPhone
On Thursday, August 8, 2019, 10:45 PM, Dave Young <dyoung(a)redhat.com> wrote:
On 08/06/19 at 07:22pm, Lianbo Jiang wrote:
From: Jun Wang <junw99(a)yahoo.com>
With some corrupted vmcore files, the vmcore-dmesg.txt file may grow
forever till the kdump disk becomes full, and also probably causes
the disk error messages as follow:
...
sd 0:0:0:0: [sda] tag#6 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] tag#6 CDB: Read(10) 28 00 08 06 4c 98 00 00 08 00
blk_update_request: I/O error, dev sda, sector 134630552
sd 0:0:0:0: [sda] tag#7 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] tag#7 CDB: Read(10) 28 00 08 06 4c 98 00 00 08 00
blk_update_request: I/O error, dev sda, sector 134630552
...
Rethink about the problem, if the log_buf_len is not corrupted, it will
be fine, in kernel code log_buf_len is a u32 int. so it will be not possible to fill disk
forever
unless vmcore-dmesg is buggy.
Also about the implementation, it looks not very elegant. Can you try
to add some limitation in vmcore-dmesg.c?
Actually in upstream kernel there is a macro LOG_BUF_LEN_MAX which is
defined as 2G, so limit the file output to within 2G should be fine.
Lets limit the size of vmcore-dmesg.txt to avoid such problems.
Signed-off-by: Jun Wang <junw99(a)yahoo.com>
Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
---
Changes since v1:
[1] Add dump_fs path to limit the size of vmcore-dmesg.txt
[2] Add the option 'iflag=fullblock' for the dd command.
dracut-kdump.sh | 4 ++--
kdump-lib-initramfs.sh | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh
index 2ae1c7c5d70d..ddd96efb5184 100755
--- a/dracut-kdump.sh
+++ b/dracut-kdump.sh
@@ -102,8 +102,8 @@ save_vmcore_dmesg_ssh() {
local _opts="$3"
local _location=$4
- echo "kdump: saving vmcore-dmesg.txt"
- $_dmesg_collector /proc/vmcore | ssh $_opts $_location "dd
of=$_path/vmcore-dmesg-incomplete.txt"
+ echo "kdump: saving vmcore-dmesg.txt, up to 100MB"
+ $_dmesg_collector /proc/vmcore | ssh $_opts $_location "dd
of=$_path/vmcore-dmesg-incomplete.txt bs=512 count=204800 iflag=fullblock"
_exitcode=$?
if [ $_exitcode -eq 0 ]; then
diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh
index 608dc6efc07e..44f9ae4dfb8f 100755
--- a/kdump-lib-initramfs.sh
+++ b/kdump-lib-initramfs.sh
@@ -137,8 +137,8 @@ save_vmcore_dmesg_fs() {
local _dmesg_collector=$1
local _path=$2
- echo "kdump: saving vmcore-dmesg.txt"
- $_dmesg_collector /proc/vmcore > ${_path}/vmcore-dmesg-incomplete.txt
+ echo "kdump: saving vmcore-dmesg.txt, up to 100MB"
+ $_dmesg_collector /proc/vmcore | dd of=$_path/vmcore-dmesg-incomplete.txt bs=512
count=204800 iflag=fullblock
_exitcode=$?
if [ $_exitcode -eq 0 ]; then
mv ${_path}/vmcore-dmesg-incomplete.txt ${_path}/vmcore-dmesg.txt
--
2.17.1
Thanks
Dave