When run tests with 2 VMs, for example nfs/ssh kdump tests, client VM will do the crash and dump, server VM will do vmcore saving and if-vmcore-exists check.
Previously, when client VM finishes running, run-test.sh will kill the lead background process, and then check if server VM has outputted "TEST PASSED" or "TEST FAILED" string. However it didn't wait for server VM to finish. As a result, the server VM's final outputs are not collected and checked, leaving the test result as "TEST RESULT NOT FOUND" sometimes.
For example, the following is the pstree status of $(jobs -p) before it gets killed. We can see the server VM is still running:
run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh --console nfs-early-kdump └─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console... └─timeout,172480 --foreground 10m /root/kexec-tools/tests/scripts/run-qemu... └─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults... ├─{qemu-system-x86},172489 ├─{qemu-system-x86},172492 ├─{qemu-system-x86},172493 ├─{qemu-system-x86},172628 └─{qemu-system-x86},172629
In this patch, we will wait for $(jobs -p) to finish, in order to get the complete output of test results.
Signed-off-by: Tao Liu ltao@redhat.com --- tests/scripts/run-test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/scripts/run-test.sh b/tests/scripts/run-test.sh index c3e94a3..1501db4 100755 --- a/tests/scripts/run-test.sh +++ b/tests/scripts/run-test.sh @@ -140,7 +140,7 @@ for test_case in $testcases; do
if [ $console -eq 1 ]; then run_test_sync $script | tee $(get_test_console_file $script) - [ -n "$(jobs -p)" ] && kill $(jobs -p) + [ -n "$(jobs -p)" ] && wait $(jobs -p) else $(run_test_sync $script > $(get_test_console_file $script)) & watch_test_outputs $test_outputs
This patch will introduce early kdump test.
It reuses the code of nfs kdump test, in order to setup 2 seperated VMs, one(the client) for trigger the early kdump and crash, the other(the server) for saving vmcore and check if vmcore exists. In order to minimize the repetted code, a soft link is made to copy the same server side code.
Signed-off-by: Tao Liu ltao@redhat.com --- .../testcases/nfs-early-kdump/0-server.sh | 1 + .../testcases/nfs-early-kdump/1-client.sh | 46 +++++++++++++++++++ 2 files changed, 47 insertions(+) create mode 120000 tests/scripts/testcases/nfs-early-kdump/0-server.sh create mode 100755 tests/scripts/testcases/nfs-early-kdump/1-client.sh
diff --git a/tests/scripts/testcases/nfs-early-kdump/0-server.sh b/tests/scripts/testcases/nfs-early-kdump/0-server.sh new file mode 120000 index 0000000..3f888ce --- /dev/null +++ b/tests/scripts/testcases/nfs-early-kdump/0-server.sh @@ -0,0 +1 @@ +../nfs-kdump/0-server.sh \ No newline at end of file diff --git a/tests/scripts/testcases/nfs-early-kdump/1-client.sh b/tests/scripts/testcases/nfs-early-kdump/1-client.sh new file mode 100755 index 0000000..dfef4d8 --- /dev/null +++ b/tests/scripts/testcases/nfs-early-kdump/1-client.sh @@ -0,0 +1,46 @@ +# Executed before VM starts +on_build() { + img_inst_pkg "nfs-utils" + img_add_qemu_cmd "-nic socket,connect=127.0.0.1:8010,mac=52:54:00:12:34:57" +} + +on_test() { + local boot_count=$(get_test_boot_count) + local nfs_server=192.168.77.1 + local earlykdump_path="/usr/lib/dracut/modules.d/99earlykdump/early-kdump.sh" + local tmp_file="/tmp/.tmp-file" + + if [[ ! -f $earlykdump_path ]]; then + test_failed "early-kdump.sh not exist!" + fi + + if [ $boot_count -eq 1 ]; then + cat << EOF > /etc/kdump.conf +nfs $nfs_server:/srv/nfs +core_collector makedumpfile -l --message-level 7 -d 31 +final_action poweroff +EOF + + while ! ping -c 1 $nfs_server -W 1; do + sleep 1 + done + + kdumpctl start \ + || test_failed "Failed to start kdump" + grubby --update-kernel=ALL --args=rd.earlykdump + + cat << EOF > $tmp_file +echo 1 > /proc/sys/kernel/sysrq +echo c > /proc/sysrq-trigger +EOF + sed -i "/early_kdump_load$/r $tmp_file" $earlykdump_path + dracut -f --add earlykdump + kdumpctl restart \ + || test_failed "Failed to start earlykdump" + + sync + reboot + else + test_failed "Unexpected reboot" + fi +}
Thanks for the patch!
Acked-by: Coiby Xu coxu@redhat.com
On Thu, Dec 30, 2021 at 11:26:43AM +0800, Tao Liu wrote:
This patch will introduce early kdump test.
It reuses the code of nfs kdump test, in order to setup 2 seperated VMs, one(the client) for trigger the early kdump and crash, the other(the server) for saving vmcore and check if vmcore exists. In order to minimize the repetted code, a soft link is made to copy the same server side code.
Signed-off-by: Tao Liu ltao@redhat.com
.../testcases/nfs-early-kdump/0-server.sh | 1 + .../testcases/nfs-early-kdump/1-client.sh | 46 +++++++++++++++++++ 2 files changed, 47 insertions(+) create mode 120000 tests/scripts/testcases/nfs-early-kdump/0-server.sh create mode 100755 tests/scripts/testcases/nfs-early-kdump/1-client.sh
diff --git a/tests/scripts/testcases/nfs-early-kdump/0-server.sh b/tests/scripts/testcases/nfs-early-kdump/0-server.sh new file mode 120000 index 0000000..3f888ce --- /dev/null +++ b/tests/scripts/testcases/nfs-early-kdump/0-server.sh @@ -0,0 +1 @@ +../nfs-kdump/0-server.sh \ No newline at end of file diff --git a/tests/scripts/testcases/nfs-early-kdump/1-client.sh b/tests/scripts/testcases/nfs-early-kdump/1-client.sh new file mode 100755 index 0000000..dfef4d8 --- /dev/null +++ b/tests/scripts/testcases/nfs-early-kdump/1-client.sh @@ -0,0 +1,46 @@ +# Executed before VM starts +on_build() {
- img_inst_pkg "nfs-utils"
- img_add_qemu_cmd "-nic socket,connect=127.0.0.1:8010,mac=52:54:00:12:34:57"
+}
+on_test() {
- local boot_count=$(get_test_boot_count)
- local nfs_server=192.168.77.1
- local earlykdump_path="/usr/lib/dracut/modules.d/99earlykdump/early-kdump.sh"
- local tmp_file="/tmp/.tmp-file"
- if [[ ! -f $earlykdump_path ]]; then
test_failed "early-kdump.sh not exist!"
- fi
- if [ $boot_count -eq 1 ]; then
cat << EOF > /etc/kdump.conf
+nfs $nfs_server:/srv/nfs +core_collector makedumpfile -l --message-level 7 -d 31 +final_action poweroff +EOF
while ! ping -c 1 $nfs_server -W 1; do
sleep 1
done
kdumpctl start \
|| test_failed "Failed to start kdump"
grubby --update-kernel=ALL --args=rd.earlykdump
cat << EOF > $tmp_file
+echo 1 > /proc/sys/kernel/sysrq +echo c > /proc/sysrq-trigger +EOF
sed -i "/early_kdump_load$/r $tmp_file" $earlykdump_path
dracut -f --add earlykdump
kdumpctl restart \
|| test_failed "Failed to start earlykdump"
sync
reboot
- else
test_failed "Unexpected reboot"
- fi
+}
2.33.1
Hi Tao,
Thanks for the patch!
If we use wait, could the test hang because server VM somehow refuses to quit?
On Thu, Dec 30, 2021 at 11:26:42AM +0800, Tao Liu wrote:
When run tests with 2 VMs, for example nfs/ssh kdump tests, client VM will do the crash and dump, server VM will do vmcore saving and if-vmcore-exists check.
Previously, when client VM finishes running, run-test.sh will kill the lead background process, and then check if server VM has outputted "TEST PASSED" or "TEST FAILED" string. However it didn't wait for server VM to finish. As a result, the server VM's final outputs are not collected and checked, leaving the test result as "TEST RESULT NOT FOUND" sometimes.
For example, the following is the pstree status of $(jobs -p) before it gets killed. We can see the server VM is still running:
run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh --console nfs-early-kdump └─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console... └─timeout,172480 --foreground 10m /root/kexec-tools/tests/scripts/run-qemu... └─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults... ├─{qemu-system-x86},172489 ├─{qemu-system-x86},172492 ├─{qemu-system-x86},172493 ├─{qemu-system-x86},172628 └─{qemu-system-x86},172629
In this patch, we will wait for $(jobs -p) to finish, in order to get the complete output of test results.
Signed-off-by: Tao Liu ltao@redhat.com
tests/scripts/run-test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/scripts/run-test.sh b/tests/scripts/run-test.sh index c3e94a3..1501db4 100755 --- a/tests/scripts/run-test.sh +++ b/tests/scripts/run-test.sh @@ -140,7 +140,7 @@ for test_case in $testcases; do
if [ $console -eq 1 ]; then run_test_sync $script | tee $(get_test_console_file $script)
[ -n "$(jobs -p)" ] && kill $(jobs -p)
else $(run_test_sync $script > $(get_test_console_file $script)) & watch_test_outputs $test_outputs[ -n "$(jobs -p)" ] && wait $(jobs -p)
-- 2.33.1
Hi Coiby,
On Thu, Jan 20, 2022 at 4:06 PM Coiby Xu coxu@redhat.com wrote:
Hi Tao,
Thanks for the patch!
If we use wait, could the test hang because server VM somehow refuses to quit?
I think it won't hang if VM refuses to quit. You can see the process tree of $(jobs -p):
run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh --console nfs-early-kdump └─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console... └─timeout,172480 --foreground 10m /root/kexec-tools/tests/scripts/run-qemu... └─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults... ├─{qemu-system-x86},172489 ├─{qemu-system-x86},172492 ├─{qemu-system-x86},172493 ├─{qemu-system-x86},172628 └─{qemu-system-x86},172629
The qemu cmdline is wrapped by timeout. So even if the VM won't exit itself, it will be killed by timeout 10m later. So I think waiting for this process group won't hang.
Thanks, Tao Liu
On Thu, Dec 30, 2021 at 11:26:42AM +0800, Tao Liu wrote:
When run tests with 2 VMs, for example nfs/ssh kdump tests, client VM will do the crash and dump, server VM will do vmcore saving and if-vmcore-exists check.
Previously, when client VM finishes running, run-test.sh will kill the lead background process, and then check if server VM has outputted "TEST PASSED" or "TEST FAILED" string. However it didn't wait for server VM to finish. As a result, the server VM's final outputs are not collected and checked, leaving the test result as "TEST RESULT NOT FOUND" sometimes.
For example, the following is the pstree status of $(jobs -p) before it gets killed. We can see the server VM is still running:
run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh --console nfs-early-kdump └─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console... └─timeout,172480 --foreground 10m /root/kexec-tools/tests/scripts/run-qemu... └─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults... ├─{qemu-system-x86},172489 ├─{qemu-system-x86},172492 ├─{qemu-system-x86},172493 ├─{qemu-system-x86},172628 └─{qemu-system-x86},172629
In this patch, we will wait for $(jobs -p) to finish, in order to get the complete output of test results.
Signed-off-by: Tao Liu ltao@redhat.com
tests/scripts/run-test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/scripts/run-test.sh b/tests/scripts/run-test.sh index c3e94a3..1501db4 100755 --- a/tests/scripts/run-test.sh +++ b/tests/scripts/run-test.sh @@ -140,7 +140,7 @@ for test_case in $testcases; do
if [ $console -eq 1 ]; then run_test_sync $script | tee $(get_test_console_file $script)
[ -n "$(jobs -p)" ] && kill $(jobs -p)
[ -n "$(jobs -p)" ] && wait $(jobs -p) else $(run_test_sync $script > $(get_test_console_file $script)) & watch_test_outputs $test_outputs
-- 2.33.1
-- Best regards, Coiby _______________________________________________ kexec mailing list -- kexec@lists.fedoraproject.org To unsubscribe send an email to kexec-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Jan 20, 2022 at 04:59:37PM +0800, Tao Liu wrote:
Hi Coiby,
On Thu, Jan 20, 2022 at 4:06 PM Coiby Xu coxu@redhat.com wrote:
Hi Tao,
Thanks for the patch!
If we use wait, could the test hang because server VM somehow refuses to quit?
I think it won't hang if VM refuses to quit. You can see the process tree of $(jobs -p):
run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh --console nfs-early-kdump └─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console... └─timeout,172480 --foreground 10m /root/kexec-tools/tests/scripts/run-qemu... └─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults... ├─{qemu-system-x86},172489 ├─{qemu-system-x86},172492 ├─{qemu-system-x86},172493 ├─{qemu-system-x86},172628 └─{qemu-system-x86},172629
The qemu cmdline is wrapped by timeout. So even if the VM won't exit itself, it will be killed by timeout 10m later. So I think waiting for this process group won't hang.
Oh, I forgot the timeout wrapper. Thanks for the explanation!
Acked-by: Coiby Xu coxu@redhat.com