The root cause is likely an infinite loop or corrupted data somewhere, but not too many vmcore dmessages. The fix of limiting vmcore-dmesg.txt size is mainly to defend kdump servers against the existing problem, as well as other future problems. It doesn’t prevent us from further investigating the root cause of the problem that is hard to reproduce either. Thanks,Jun
Sent from Yahoo Mail for iPhone
On Thursday, August 8, 2019, 10:45 PM, Dave Young dyoung@redhat.com wrote:
On 08/06/19 at 07:22pm, Lianbo Jiang wrote:
From: Jun Wang junw99@yahoo.com
With some corrupted vmcore files, the vmcore-dmesg.txt file may grow forever till the kdump disk becomes full, and also probably causes the disk error messages as follow: ... sd 0:0:0:0: [sda] tag#6 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK sd 0:0:0:0: [sda] tag#6 CDB: Read(10) 28 00 08 06 4c 98 00 00 08 00 blk_update_request: I/O error, dev sda, sector 134630552 sd 0:0:0:0: [sda] tag#7 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK sd 0:0:0:0: [sda] tag#7 CDB: Read(10) 28 00 08 06 4c 98 00 00 08 00 blk_update_request: I/O error, dev sda, sector 134630552 ...
Rethink about the problem, if the log_buf_len is not corrupted, it will be fine, in kernel code log_buf_len is a u32 int. so it will be not possible to fill disk forever unless vmcore-dmesg is buggy.
Also about the implementation, it looks not very elegant. Can you try to add some limitation in vmcore-dmesg.c?
Actually in upstream kernel there is a macro LOG_BUF_LEN_MAX which is defined as 2G, so limit the file output to within 2G should be fine.
Lets limit the size of vmcore-dmesg.txt to avoid such problems.
Signed-off-by: Jun Wang junw99@yahoo.com Signed-off-by: Lianbo Jiang lijiang@redhat.com
Changes since v1: [1] Add dump_fs path to limit the size of vmcore-dmesg.txt [2] Add the option 'iflag=fullblock' for the dd command.
dracut-kdump.sh | 4 ++-- kdump-lib-initramfs.sh | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index 2ae1c7c5d70d..ddd96efb5184 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -102,8 +102,8 @@ save_vmcore_dmesg_ssh() { local _opts="$3" local _location=$4 - echo "kdump: saving vmcore-dmesg.txt" - $_dmesg_collector /proc/vmcore | ssh $_opts $_location "dd of=$_path/vmcore-dmesg-incomplete.txt" + echo "kdump: saving vmcore-dmesg.txt, up to 100MB" + $_dmesg_collector /proc/vmcore | ssh $_opts $_location "dd of=$_path/vmcore-dmesg-incomplete.txt bs=512 count=204800 iflag=fullblock" _exitcode=$? if [ $_exitcode -eq 0 ]; then diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 608dc6efc07e..44f9ae4dfb8f 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -137,8 +137,8 @@ save_vmcore_dmesg_fs() { local _dmesg_collector=$1 local _path=$2 - echo "kdump: saving vmcore-dmesg.txt" - $_dmesg_collector /proc/vmcore > ${_path}/vmcore-dmesg-incomplete.txt + echo "kdump: saving vmcore-dmesg.txt, up to 100MB" + $_dmesg_collector /proc/vmcore | dd of=$_path/vmcore-dmesg-incomplete.txt bs=512 count=204800 iflag=fullblock _exitcode=$? if [ $_exitcode -eq 0 ]; then mv ${_path}/vmcore-dmesg-incomplete.txt ${_path}/vmcore-dmesg.txt -- 2.17.1
Thanks Dave