kexec August 2021

kexec@lists.fedoraproject.org

7 participants
10 discussions

[PATCH] mkdumprd: allow spaces after 'path' config phrase when network dump
by Kazuhito Hagio 26 May '22

26 May '22

Without this patch, when there are two or more spaces after 'path' configuration phrase with ssh or nfs setting, SAVE_PATH is set to '/var/crash' in mkdumprd, and in most cases kdump service fails to start. ssh kdump(a)192.168.122.1 path /kdump ^^ This behavior would be too sensitive and different from the other configurations. With this patch, mkdumprd allows such spaces. Signed-off-by: Kazuhito Hagio <k-hagio(a)ab.jp.nec.com> --- mkdumprd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdumprd b/mkdumprd index a6f7fe8..aa0abfd 100644 --- a/mkdumprd +++ b/mkdumprd @@ -13,7 +13,7 @@ export IN_KDUMP=1 conf_file="/etc/kdump.conf" SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" -SAVE_PATH=$(grep ^path $conf_file| cut -d' ' -f2) +SAVE_PATH=$(awk '/^path/ {print $2}' $conf_file) [ -z "$SAVE_PATH" ] && SAVE_PATH=$DEFAULT_PATH # strip the duplicated "/" SAVE_PATH=$(echo $SAVE_PATH | tr -s /) -- 2.18.0

4 3

Re: [dm-crypt] Kdump with full-disk LUKS encryption
by Kairui Song 05 Nov '21

05 Nov '21

Hi, Thanks a lot, these info are very helpful. Better to keep it for debugging for now, and ask users to use it very carefully. On Tue, Apr 20, 2021 at 3:54 PM Milan Broz <gmazyland(a)gmail.com> wrote: > > Hi, > > TL;DR what you are trying to do is to actually reverse many security measures > we added. It is perhaps acceptable for debugging but hardly for real generic system. > > - using memory-hard function increases cost of dictionary and brute-force > attacks > You can always decrease amount of memory needed, but you should do it only > if you know that security margin is ok (like password is randomly generated > with enough entropy). > > - key is in keyring to remove possibility for normal userspace to receive > the key from kernel. Moreover, there is no need to retain kernel in keyring once > dm-crypt device is activated. (It is still in kernel memory but only in crypto > functions context). (Systemd also uses keyring to cache passphrase but that's > different thing.) > > You can still use old way for activation with --disable-keyring activation, > but then you disable this possibility. > > More comments below. > > On 19/04/2021 12:00, Kairui Song wrote: > > Hi all, > > > > I'm currently trying to add kdump support for systemd with full-disk > > LUKS encryption. vmcores contain sensitive data so they should also be > > protected, and network dumps sometimes are not available. So kdump has > > to open the LUKS encrypted device in the kdump environment. > > > > I'm using systemd/dracut, my work machine is running Fedora 34, and > > there are several problems I'm trying to solve: > > 1. Users have to input the password in the kdump kernel environment. > > But users often don't have shell access to the kdump environment. > > (headless server, graphic card not working after kexec, both are very > > common) > > 2. LUKS2 prefers Argon2 as the key derivation function, designed to > > use a lot of memory. kdump is expected to use a minimal amount of > > memory. Users will have to reserve a huge amount of memory for kdump > > to work (eg. 1G reserve for kdump with 4G total memory which is not > > reasonable). > > When I added Argon2 to LUKS2, I actually expected such issues. Despite > some people beats me that they cannot use arbitrary amount of memory, > we have some hard limits that were selected that it should work on most recent > systems. Maybe kdump can live with it. > > - maximum memory cost limit is 4GB, no LUKS2 device can use more for Argon2 > - we never use more than half of available physical memory > (measured on the host where the device was formatted) > - required amount of memory is visible in LUKS2 metadata (luksDump) > for the particular keyslot (Memory: the value is in kB) > - we use benchmark to calculate memory cost with prefered unlocking > time 2 seconds (again, on the device where LUKS was formatted) > Small systems (like RPi2) the uses much smaller acceptable values. > You can configure all costs (time, memory, threads) during format > or even set them to predefined values. > > I am sorry, but there is really no way around this - and the requeired > memory must be physical memory (otherwise it slows down extremely). > This is a feature, not a bug :-) > > > > To fix these problems, I tried to pass the master key to the second > > kernel directly via initramfs. Kdump will modify the initramfs in > > ramfs to include the key, kexec_load it, and never write to any actual > > back storage. This worked with old LUKS configurations. > > Well, passing volume key this way is quite insecure, but perhaps > acceptable for debugging. > > > > > But LUKS2/cryptsetup now stores the key in the kernel keyring by > > default. The key is accessible from userspace. > > If you are talking about volume key (not passsphrase), it is not > available from userspace. Only reference to it. But you can use > this reference to construct in-kernel dm-crypt device. > Please read https://gitlab.com/cryptsetup/cryptsetup/-/blob/master/docs/Keyring.txt > > > Users can enter the password to start kdump manually and then it will > > work, but usually people expect kdump service to start automatically. > > > > (WIP patch series: > > https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org…) > > > > I've several ideas about how to improve it but not sure which one is > > better, and if this is a good idea after all: > > 1. Simply introduce a config to let systemd-cryptsetup disable kernel > > keyring on setup, there is currently no such option. > > Well, that option could be useful anyway and we have support for it > in cryptsetup (--disable-keyring CLI option) and libcryptsetup, so why not. > Just this should not be a default option. > > This is should be patch for systemd-cryptsetup only as libcryptsetup supports it. > > ... > > > 2. If we can let the key stay in userspace for a little longer, eg. > > for systems booted with dracut/systemd, when > > systemd-cryptsetup(a)%s.service opens the crypt device, keep the key in > > dm-crypt. And later when services like kdump have finished loading, > > cryptsetup can refresh the device and store the key in the kernel > > keyring again. > > We invalidate volume key in keyring after libceyposetup operation > is finished (and kernel removes the reference once keyring garbage collection > is run). > > I can imagine to add some option to keep key inside keyring even after > call is finished, but as said above, this removes some security margin > we intentionally introduced here. I agree with your comments, thanks! These two approaches seem not a good idea now. > > ... > > Milan > How about plan 3 and 4? > 3. Let kdump use some custom helper/service to load all needed > resources in the early initrd boot stage, prior to > systemd-cryptsetup(a)%s.service. It will ask the password / parse the > keyfile and load kdump, then provide info for systemd-cryptsetup or > just do the setup. Or maybe let systemd-cryptsetup support some kind > of "plugins" so other tools can use it. Some details could be changed/improved, but systemd-cryptsetup(a)%s.service will prompt for a password or use a keyfile anyway. So I think at this point, loading kdump with the volume key should be safe? At least long as the kdump kernel/environment itself isn't compromised. Loaded kdump resources can be restricted to be only accessible from the kernel side. After panic, kernel kexec jumps to kdump kernel, and that's an minimized emergency environment that only lives for a very short period. > 4. A better and safer solution seems to keep a consistent key ring > between kexec boots but also more complex and difficult as different > arch implements kexec differently. Maybe plan 4 will be a good idea if doable? Since that keeps the key consistent in the kernel between kexec boots, and cryptsetup can just reuse it. -- Best Regards, Kairui Song

3 4

[PATCH v4] selftest: kill VM reliably by recursively kill children processes
by Coiby Xu 15 Oct '21

15 Oct '21

qemu is launched in nested subprocess and can't be killed by simply killing the job ids, PID Command 2269634 │ ├─ sshd: root [priv] 2269637 │ │ └─ sshd: root@pts/0 2269638 │ │ └─ -bash 2269744 │ │ └─ make test-run V=1 2273117 │ │ └─ /bin/bash /root/kexec-tools-300/tests/scripts/run-test.sh 2273712 │ │ ├─ /bin/bash /root/kexec-tools-300/tests/scripts/run-test.sh 2273714 │ │ │ └─ /bin/bash /root/kexec-tools-300/tests/scripts/run-test.sh 2273737 │ │ │ └─ timeout --foreground 10m /root/kexec-tools-300/tests/scripts/run-qemu -nodefaults -nographic -smp 2 -m 768M -monitor no 2273738 │ │ │ └─ /usr/bin/qemu-system-x86_64 -enable-kvm -cpu host -nodefaults -nographic -smp 2 -m 768M -monitor none -serial stdio 2273746 │ │ │ ├─ /usr/bin/qemu-system-x86_64 -enable-kvm -cpu host -nodefaults -nographic -smp 2 -m 768M -monitor none -serial std 2273797 │ │ ├─ /bin/bash /root/kexec-tools-300/tests/scripts/run-test.sh 2273798 │ │ │ └─ /bin/bash /root/kexec-tools-300/tests/scripts/run-test.sh 2273831 │ │ │ └─ timeout --foreground 10m /root/kexec-tools-300/tests/scripts/run-qemu -nodefaults -nographic -smp 2 -m 768M -monitor no 2273832 │ │ │ └─ /usr/bin/qemu-system-x86_64 -enable-kvm -cpu host -nodefaults -nographic -smp 2 -m 768M -monitor none -serial stdio 2273840 │ │ │ ├─ /usr/bin/qemu-system-x86_64 -enable-kvm -cpu host -nodefaults -nographic -smp 2 -m 768M -monitor none -serial std This led to the error "qemu-system-x86_64: can't bind ip=0.0.0.0 to socket: Address already in use". This patch will kill qemu by killing all the children of the job id. Signed-off-by: Coiby Xu <coxu(a)redhat.com> --- Thanks to Kairui for finding out an error and an issue about previous version, - "$_job" should be used instead "$job" - "Ctrl-c" leads to the failure of parsing pstree result This version stops parsing pstree and use pgrep to get the children of a process and kill them recursively instead. --- tests/scripts/run-test.sh | 37 +++++++++++++++++++++++++++++++++---- tests/scripts/test-lib.sh | 3 +-- 2 files changed, 34 insertions(+), 6 deletions(-) diff --git a/tests/scripts/run-test.sh b/tests/scripts/run-test.sh index a68504d..7113be8 100755 --- a/tests/scripts/run-test.sh +++ b/tests/scripts/run-test.sh @@ -1,9 +1,38 @@ #!/bin/bash -_kill_all_jobs() { - local _jobs=$(jobs -r -p) +_kill_if_valid_pid() { + local _pid="$1" + if ps -p $_pid > /dev/null + then + kill $_pid + fi +} + +_recursive_kill() { + local _pid="$1" + local _children _child + + _children=$(pgrep -P $_pid) + if [ -n "$_children" ]; then + for _child in $_children + do + _recursive_kill $_child + _kill_if_valid_pid $_child + done + fi + _kill_if_valid_pid $_pid +} - [ -n "$_jobs" ] && kill $_jobs +_kill_all_jobs() { + local _jobs=$(jobs -r -p) + local _job + + if [ -n "$_jobs" ]; then + for _job in $_jobs + do + _recursive_kill $_job + done + fi } trap ' @@ -121,7 +150,7 @@ for test_case in $testcases; do [ $? -ne 0 ] && ret=$(expr $ret + 1) results[$test_case]="$res" - + _kill_all_jobs echo -e "-------- Test finished: $test_case $res --------" for script in $scripts; do script="$testdir/$script" diff --git a/tests/scripts/test-lib.sh b/tests/scripts/test-lib.sh index f8a2249..8b24b2a 100644 --- a/tests/scripts/test-lib.sh +++ b/tests/scripts/test-lib.sh @@ -146,8 +146,7 @@ watch_test_outputs() { ret=$? if [ $ret -ne 255 ]; then - # Test finished, kill VMs - kill $(jobs -p) + # Test finished break 2 fi done -- 2.32.0

2 1

[PATCH] mkdumprd: automatically create dump path for user specified target
by Kairui Song 17 Sep '21

17 Sep '21

When user only provided a "path" value in kdump.conf without explicitly specify a dump target, covering all cases with "path" config value is complex. "path" could be a mount point, a dir insidse a mount point, or nested mount, bind mount etc. kdump need to detect the real dump target based on the "path" value, so user have to be carefull with it and in charge of creating it. But for user specified dump target (dump target explicitly specified with nfs/ext4/xfs/...), the "path" config have only one sole meaning: the absolute dump path on a dump target. In this case it's safe for kdump to create it automatically. Signed-off-by: Kairui Song <kasong(a)redhat.com> --- mkdumprd | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mkdumprd b/mkdumprd index cc37ae18..0dc4459e 100644 --- a/mkdumprd +++ b/mkdumprd @@ -242,7 +242,8 @@ check_user_configured_target() # For user configured target, use $SAVE_PATH as the dump path within the target if [[ ! -d "$_mnt/$SAVE_PATH" ]]; then - perror_exit "Dump path \"$_mnt/$SAVE_PATH\" does not exist in dump target \"$_target\"" + dwarn "Dump path \"$SAVE_PATH\" does not exist on dump target \"$_target\", kdump will create this path automatically." + mkdir -p "$_mnt/$SAVE_PATH" || perror_exit "Failed to create dump path \"$SAVE_PATH\" on dump target." fi check_size fs "$_target" -- 2.31.1

2 1

[PATCH v2 00/49] Batch update and refactor
by Kairui Song 13 Sep '21

13 Sep '21

This mass update is done with the assistant of shellcheck and shfmt. This patch series fixed most syntax issues found by shellcheck, tons of corner cases and issues are fixed. Also simplified a lot of code according to suggestions by shellcheck. Related shellcheck warn/error/suggest info links are included in the commit message. Since shellcheck can help to detect non-POSIX syntax issue in POSIX script (#!/bin/sh), and the coding style in kexec-tools package is very messy, this series refactored the code structure in following way: - kdumpctl, mkdumprd, mkfadumprd, *-module-setup.sh and kdump-lib.sh is "bash only" script. We rely on bash only syntax heavily, and they will only be used in the first kernel, so just let them leverage bash syntax for better maintainability and better performance. - kdump-lib-initramfs.sh, and dracut-kdump.sh is now POSIX compatible, and will be used in the second kernel. This refactoring also enables kdump to use dash or busybox, instead of bash in the second kernel, which improves performance and memory usage. Tested with latest dracut on Fedora 34, now "dash" in second kernel works well, it's faster than bash and save a lot of memory. We may test dash or busybox as default kdump shell in the future for better performance. For reviewers: Patch 25, 26, 30, 41, 46 is automatic code update using script and tools like shfmt, other patch may need careful review. Patch 1 added a editorconfig file for shfmt to work properly, which is Acked by Pingfan but still posting it to make it easier to review. Patch 2 - 29 focus on the bash part: Patch 2, 3, 4, 5, 8 introduced new config file read and format helper, which unified the config file reading in kexec-tools scripts. So space in config file should be no longer a problem (previously random problem may occur since kexec-tools scripts is using many different ways to read the config file). And this make it much easier to clean up the code. Patch 6, 7, 9 - 24 fixed many issues found with shellcheck in bash scripts manually. Some issues are already breaking kdump just no one noticed yet. Patch 25, 26, 30 is an automatic code update using sed and shfmt, there should be no code behaviour change. Patch 27, 28, 29 batch fixed some common bash script issues manually with hints from shellcheck. Patch 31 - 48 focus on the POSIX part: Patch 31 - 33 restructures the code, prepare to make kdump-lib-initramfs.sh a POSIX script and drop kdump-lib.sh in kdump kernel. Patch 34 - 38 clean up code in dracut-kdump.sh, and fix misc issues found by shellcheck manually, prepare to make it POSIX compatible. Patch 39 contains a lot of code refactoring that makes dracut-kdump.sh POSIX compatible. Patch 40 contains a lot of code refactoring that makes kdump-lib-initramfs.sh POSIX compatible. Patch 41 is an automatic code update using sed, there should be no code behaviour change. Patch 42, 43 reworked how nmcli is being called by kexec-tools, fixed word splitting issue and allow connection name to contain space. Patch 44 optimized a sed call. Patch 45 batch fixed some common bash script issues manually with hints from shellcheck. Patch 46 is an automatic code update using shfmt, there should be no code behaviour change. Patch 47, 48 contains a lot of change that makes kdump-logger.sh POSIX compatible. Patch 49 allows mkdumprd to use dash instead of bash in second kernel. Update from V1: - Fix many typos and code error, and fix a few more existing code issues found while reviewing the patch, many thanks to Philipp Rudo. - Add more info in commit message and cover letter. Kairui Song (49): Add a .editorconfig file kdump-lib.sh: add a config format and read helper kdump-lib.sh: add a config value retrive helper kdump-lib.sh: use kdump_get_conf_val to read config values kdumpctl: use kdump_get_conf_val to read config values kdumpctl: get rid of a `wc` call kdumpctl: fix fragile loops over find output mkdumprd: use kdump_get_conf_val to read config values mkdumprd: make dracut_args an array again mkdumprd: remove some redundant echo mkdumprd: fix multiple issues with get_ssh_size mkdumprd: use array to store ssh arguments in mkdir_save_path_ssh mkfadumprd: make _dracut_isolate_args an array dracut-module-setup.sh: rework kdump_get_ip_route_field dracut-module-setup.sh: remove an unused variable dracut-module-setup.sh: fix _bondoptions wrong references dracut-module-setup.sh: use "*" to expend array as string dracut-module-setup.sh: fix a ambiguous variable reference dracut-module-setup.sh: fix a loop over ls issue dracut-module-setup.sh: make iscsi check fail early if cd failed bash scripts: remove useless cat bash scripts: get rid of expr and let bash scripts: get rid of unnessary sed calls bash scripts: always use "read -r" bash scripts: use $(...) notation instead of legacy `...` bash scripts: replace '[ ]' with '[[ ]]' for bash scripts bash scripts: fix variable quoting issue bash scripts: fix redundant exit code check bash scripts: declare and assign separately bash scripts: reformat with shfmt Prepare to make kdump-lib-initramfs.sh a POSIX compatible lib Merge kdump-error-handler.sh into kdump.sh kdump-lib-initramfs.sh: move dump raleted functions to kdump.sh dracut-kdump.sh: don't put KDUMP_SCRIPT_DIR in PATH dracut-kdump.sh: remove add_dump_code dracut-kdump.sh: simplify dump_ssh dracut-kdump.sh: Use stat instead of ls to get vmcore size dracut-kdump.sh: POSIX doesn't support pipefail dracut-kdump.sh: make it POSIX compatible kdump-lib-initramfs.sh: make it POSIX compatible kdump-lib.sh: replace '[ ]' with '[[ ]]' and get rid of legacy `` kdump-lib.sh: fix and simplify get_nmcli_value_by_field kdump-lib.sh: fix word splitting of nmcli params kdump-lib.sh: avoid calling sed multiple times in get_system_size kdump-lib.sh: batch fix issues found with shellcheck kdump-lib.sh: reformat with shfmt kdump-logger.sh: refactor to remove some bash only syntax kdump-logger.sh: make it POSIX compatible mkdumprd: allow using dash .editorconfig | 32 + dracut-early-kdump-module-setup.sh | 11 +- dracut-kdump-emergency.service | 2 +- dracut-kdump-error-handler.sh | 10 - dracut-kdump.sh | 511 ++++++++--- dracut-module-setup.sh | 668 +++++++------- kdump-lib-initramfs.sh | 346 ++----- kdump-lib.sh | 1356 +++++++++++++--------------- kdump-logger.sh | 117 +-- kdumpctl | 604 ++++++------- kexec-tools.spec | 2 - mkdumprd | 600 ++++++------ mkfadumprd | 21 +- 13 files changed, 2104 insertions(+), 2176 deletions(-) create mode 100644 .editorconfig delete mode 100755 dracut-kdump-error-handler.sh -- 2.31.1

3 81

[PATCH] crash: add build requirement wget
by Philipp Rudo 23 Aug '21

23 Aug '21

wget is used to download the tar ball with the gdb sources. Add it as a build requirement. Signed-off-by: Philipp Rudo <prudo(a)redhat.com> --- crash.spec | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crash.spec b/crash.spec index 212d3c8..cd42075 100644 --- a/crash.spec +++ b/crash.spec @@ -11,7 +11,7 @@ Source1: http://ftp.gnu.org/gnu/gdb/gdb-7.6.tar.gz URL: https://crash-utility.github.io ExclusiveOS: Linux ExclusiveArch: %{ix86} ia64 x86_64 ppc ppc64 s390 s390x %{arm} aarch64 ppc64le -BuildRequires: ncurses-devel zlib-devel lzo-devel snappy-devel bison +BuildRequires: ncurses-devel zlib-devel lzo-devel snappy-devel bison wget BuildRequires: gcc gcc-c++ BuildRequires: make Requires: binutils -- 2.31.1

3 5

[PATCH] crash: add build requirement patch
by Philipp Rudo 23 Aug '21

23 Aug '21

the gdb sources have to be patched for crash. Add patch as a build requirement. Signed-off-by: Philipp Rudo <prudo(a)redhat.com> --- crash.spec | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crash.spec b/crash.spec index cd42075..75f8cee 100644 --- a/crash.spec +++ b/crash.spec @@ -11,7 +11,7 @@ Source1: http://ftp.gnu.org/gnu/gdb/gdb-7.6.tar.gz URL: https://crash-utility.github.io ExclusiveOS: Linux ExclusiveArch: %{ix86} ia64 x86_64 ppc ppc64 s390 s390x %{arm} aarch64 ppc64le -BuildRequires: ncurses-devel zlib-devel lzo-devel snappy-devel bison wget +BuildRequires: ncurses-devel zlib-devel lzo-devel snappy-devel bison wget patch BuildRequires: gcc gcc-c++ BuildRequires: make Requires: binutils -- 2.31.1

2 1

[PATCH] Remove hard requirement on grubby
by Kairui Song 19 Aug '21

19 Aug '21

Downgrade to "Recommends:" as suggested by CoreOS team. Signed-off-by: Kairui Song <kasong(a)redhat.com> --- kexec-tools.spec | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kexec-tools.spec b/kexec-tools.spec index c0425901..951c9d6e 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -70,7 +70,7 @@ Requires: dracut >= 050 Requires: dracut-network >= 050 Requires: dracut-squash >= 050 Requires: ethtool -Requires: grubby +Recommends: grubby BuildRequires: make BuildRequires: zlib-devel elfutils-devel glib2-devel bzip2-devel ncurses-devel bison flex lzo-devel snappy-devel BuildRequires: pkgconfig intltool gettext -- 2.31.1

2 1

[PATCH 00/47] Batch update and refactor
by Kairui Song 17 Aug '21

17 Aug '21

4 80

[PATCH] Clear old crashkernl=auto in comment and doc
by Kairui Song 05 Aug '21

05 Aug '21

Signed-off-by: Kairui Song <kasong(a)redhat.com> --- fadump-howto.txt | 9 ++++++--- kdumpctl | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/fadump-howto.txt b/fadump-howto.txt index 433e9a6..bc87644 100644 --- a/fadump-howto.txt +++ b/fadump-howto.txt @@ -344,9 +344,12 @@ or OR # grubby --update-kernel=/boot/vmlinuz-`uname -r` --args="fadump=off" -If KDump is to be used as the dump capturing mechanism, update the crashkernel -parameter (Else, remove "crashkernel=" parameter too, using grubby): +Remove "crashkernel=" from kernel cmdline parameters: - # grubby --update-kernel=/boot/vmlinuz-$kver --args="crashkernl=auto" + # grubby --update-kernel=/boot/vmlinuz-`uname -r` --remove-args="crashkernel" + +If KDump is to be used as the dump capturing mechanism, reset the crashkernel parameter: + + # kdumpctl reset-crashkernel `uname -r` Reboot the system for the settings to take effect. diff --git a/kdumpctl b/kdumpctl index 0fb73b6..f10b091 100755 --- a/kdumpctl +++ b/kdumpctl @@ -1232,7 +1232,7 @@ do_estimate() baseline=$((${baseline%Y} * 1048576)) fi - # The default value when using crashkernel=auto + # The default pre-reserved crashkernel value baseline_size=$((baseline * size_mb)) # Current reserved crashkernel size reserved_size=$(cat /sys/kernel/kexec_crash_size) -- 2.31.1

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

kexec August 2021