On 03/27/2017 at 04:38 PM, Xunlei Pang wrote:
On 03/27/2017 at 04:25 PM, Dave Young wrote:
> On 03/27/17 at 04:03pm, Xunlei Pang wrote:
>> On 03/27/2017 at 03:40 PM, Pratyush Anand wrote:
>>> Hi Xunlei,
>>>
>>> On Monday 27 March 2017 09:37 AM, Xunlei Pang wrote:
>>>> We met a problem that the kdump emergency service failed to
>>>> start when the target dump timeout(we passed "rd.timeout=30"
>>>> to kdump), it reported "Transaction is destructive" messages:
>>>>
>>>> [ TIME ] Timed out waiting for device
dev-mapper-fedora\x2droot.device.
>>>> [DEPEND] Dependency failed for Initrd Root Device.
>>>> [ SKIP ] Ordering cycle found, skipping System Initialization
>>>> [DEPEND] Dependency failed for /sysroot.
>>>> [DEPEND] Dependency failed for Initrd Root File System.
>>>> [DEPEND] Dependency failed for Reload Configuration from the Real
Root.
>>>> [ SKIP ] Ordering cycle found, skipping System Initialization
>>>> [ SKIP ] Ordering cycle found, skipping Initrd Default Target
>>>> [DEPEND] Dependency failed for File System Check on
/dev/mapper/fedora-root.
>>>> [ OK ] Reached target Initrd File Systems.
>>>> [ OK ] Stopped dracut pre-udev hook.
>>>> [ OK ] Stopped dracut cmdline hook.
>>>> Starting Setup Virtual Console...
>>>> Starting Kdump Emergency...
>>>> [ OK ] Reached target Initrd Default Target.
>>>> [ OK ] Stopped dracut initqueue hook.
>>>> Failed to start kdump-error-handler.service: Transaction is
destructive.
>>>> See system logs and 'systemctl status
kdump-error-handler.service' for details.
>>>> [FAILED] Failed to start Kdump Emergency.
>>>> See 'systemctl status emergency.service' for details.
>>>> [DEPEND] Dependency failed for Emergency Mode.
>>>>
>>>> This is because in case of root failure, initrd-root-fs.target
>>>> will trigger systemd emergency target which requires the systemd
>>>> emergency service actually is kdump-emergency.service, then our
>>>> kdump-emergency.service starts kdump-error-handler.service with
>>>> "systemctl isolate"(see 99kdumpbase/kdump-emergency.service,
we
>>>> replace systemd's with this one under kdump).
>>> I could not understand why kdump emergency.service needs isolate while, it is
not needed in first kernel (ie in systemd). If kdump would work without isolate then
probably it will be more closer to upstream systemd and we will need less modifications in
kdump.
>> Yes, what you said is another solution.
>>
>> Seems Dave has some concern about we may not stop some services like before if
replacing "isolate" with "start".
>> From my tests it is true that some services will be continuing.
>>
>> Hi Dave, what is you opinion?
> I do not remember why we use isolate previously. I thought it is for
> force run kdump error handler service so that kdump error handler ensure
> not to be blocked by other services.
>
> Suppose other service can run in an infinite loop, does the "start"
> still work for us? For example we expect "default shell" but other
> services runs in a loop or something else..
>
It may be due to our "default XXX" feature, we have to ensure it executes
reliably as possible as it can in case of emergency.
In my view, we have been using "isolate" since 2014, better no touch that,
that's why after some thinking I choose this method.
The following messages is an example I met during tests when I used "start"
instead of "isolate",
it triggered the Kdump Error Handler twice:
1) One was triggered by the initrd root fs service failure. 2) The other was triggered by
the kdump capture service failure going on
[ TIME ] Timed out waiting for device dev-mapper-fedora\x2droot.device.
[DEPEND] Dependency failed for Initrd Root Device.
[ SKIP ] Ordering cycle found, skipping dracut pre-pivot and cleanup hook
[ SKIP ] Ordering cycle found, skipping Initrd Default Target
[DEPEND] Dependency failed for File System Check on /dev/mapper/fedora-root.
[DEPEND] Dependency failed for /sysroot.
[DEPEND] Dependency failed for Initrd Root File System.
[DEPEND] Dependency failed for Reload Configuration from the Real Root.
[ SKIP ] Ordering cycle found, skipping dracut pre-pivot and cleanup hook
[ SKIP ] Ordering cycle found, skipping Initrd Default Target
[ SKIP ] Ordering cycle found, skipping dracut pre-pivot and cleanup hook
[ SKIP ] Ordering cycle found, skipping Initrd Default Target
[ OK ] Reached target Initrd File Systems.
[ OK ] Stopped dracut pre-udev hook.
[ 62.773496] audit_printk_skb: 48 callbacks suppressed
[ 62.773838] audit: type=1131 audit(1490601290.421:36): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=dracut-pre-udev comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Starting Kdump Emergency...
[ OK ] Stopped dracut cmdline hook.
[ 62.778038] audit: type=1131 audit(1490601290.425:37): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=dracut-cmdline comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Starting Setup Virtual Console...
[ OK ] Stopped dracut initqueue hook.
[ 62.784637] audit: type=1130 audit(1490601290.432:38): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 62.786329] audit: type=1131 audit(1490601290.434:39): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ OK ] Started Kdump Emergency.
[ 62.801906] audit: type=1130 audit(1490601290.449:40): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=emergency comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 62.805255] audit: type=1131 audit(1490601290.453:41): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=emergency comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
Starting dracut pre-pivot and cleanup hook...
[ OK ] Started Setup Virtual Console.
[ 62.856032] audit: type=1130 audit(1490601290.503:42): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Starting Kdump Error Handler...
Kdump: Executing default action kdump_emergency_shell
Warning: /dev/mapper/fedora-root does not exist
Generating "/run/initramfs/rdsosreport.txt"
[ OK ] Started dracut pre-pivot and cleanup hook.
[ 62.936605] audit: type=1130 audit(1490601290.584:43): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=dracut-pre-pivot comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Starting Kdump Vmcore Save Service...
[ OK ] Reached target Emergency Mode.
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.
kdump:/# kdump: dump target is
kdump: error: Dump target is not mounted.
kdump: saving vmcore failed
[FAILED] Failed to start Kdump Vmcore Save Service.
See 'systemctl status kdump-capture.service' for details.
[ SKIP ] Ordering cycle found, skipping dracut pre-pivot and cleanup hook
[ 62.995441] audit: type=1130 audit(1490601290.643:44): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=kdump-capture comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[ OK ] Stopped target Remote File Systems.
[ OK ] Stopped target Timers.
[ OK ] Stopped target Remote File Systems (Pre).
[ OK ] Stopped target Initrd Default Target.
[ OK ] Stopped target Basic System.
[ OK ] Stopped target Sockets.
[ OK ] Stopped target System Initialization.
[ 63.008279] systemd-journald[764]: Received SIGTERM from PID 1 (systemd).
[ 63.009140] systemd[1]: Stopping Journal Service...
Stopping Journal Service...
[ 63.013373] systemd[1]: Stopped Apply Kernel Variables.
[ OK ] Stopped Apply Kernel Variables.
[ 63.016630] audit: type=1131 audit(1490601290.664:45): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=systemd-sysctl comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 63.018835] systemd[1]: Stopped udev Coldplug all Devices.
[ OK ] Stopped udev Coldplug all Devices.
[ 63.021430] systemd[1]: Stopped target Local File Systems.
[ OK ] Stopped target Local File Systems.
[ 63.023394] systemd[1]: Starting Kdump Emergency...
Starting Kdump Emergency...
[ 63.025422] systemd[1]: Stopped target Paths.
[ OK ] Stopped target Paths.
[ 63.026916] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch.
[ OK ] Stopped Dispatch Password Requests to Console Directory Watch.
[ 63.028824] systemd[1]: Stopped target Initrd File Systems.
[ OK ] Stopped target Initrd File Systems.
[ 63.032734] systemd[1]: Stopped target Swap.
[ OK ] Stopped target Swap.
[ 63.033712] systemd[1]: Stopped target Slices.
[ OK ] Stopped target Slices.
Stopping udev Kernel Device Manager...
[ OK ] Stopped Journal Service.
[ OK ] Stopped udev Kernel Device Manager.
[ OK ] Started Kdump Emergency.
[ OK ] Stopped Kdump Error Handler.
[ OK ] Listening on Journal Audit Socket.
[ OK ] Started Dispatch Password Requests to Console Directory Watch.
[ OK ] Stopped Setup Virtual Console.
[ OK ] Reached target Slices.
Starting Apply Kernel Variables...
[ OK ] Reached target Timers.
Starting Journal Service...
[ OK ] Reached target Paths.
[ OK ] Stopped target Emergency Mode.
[ OK ] Reached target Swap.
[ OK ] Reached target Local File Systems.
Starting dracut cmdline hook...
[ OK ] Reached target Sockets.
[ OK ] Started Journal Service.
[ OK ] Started Apply Kernel Variables.
[ OK ] Started dracut cmdline hook.
Starting dracut pre-udev hook...
[ OK ] Started dracut pre-udev hook.
Starting udev Kernel Device Manager...
[ OK ] Started udev Kernel Device Manager.
Starting udev Coldplug all Devices...
[ OK ] Started udev Coldplug all Devices.
Starting dracut initqueue hook...
[ OK ] Reached target System Initialization.
[ OK ] Reached target Basic System.
[ TIME ] Timed out waiting for device dev-mapper-fedora\x2droot.device.
[DEPEND] Dependency failed for Initrd Root Device.
[ SKIP ] Ordering cycle found, skipping dracut pre-pivot and cleanup hook
[DEPEND] Dependency failed for File System Check on /dev/mapper/fedora-root.
[DEPEND] Dependency failed for /sysroot.
[DEPEND] Dependency failed for Initrd Root File System.
[DEPEND] Dependency failed for Reload Configuration from the Real Root.
[ SKIP ] Ordering cycle found, skipping dracut pre-pivot and cleanup hook
[ SKIP ] Ordering cycle found, skipping dracut pre-pivot and cleanup hook
[ OK ] Stopped target Basic System.
[ OK ] Reached target Initrd File Systems.
[ OK ] Stopped target System Initialization.
Starting Setup Virtual Console...
[ OK ] Stopped dracut pre-udev hook.
[ 93.273790] audit_printk_skb: 45 callbacks suppressed
[ 93.274120] audit: type=1131 audit(1490601320.921:61): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=dracut-pre-udev comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ OK ] Stopped dracut cmdline hook.
[ 93.283737] audit: type=1131 audit(1490601320.931:62): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=dracut-cmdline comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Starting Kdump Emergency...
[ OK ] Stopped dracut initqueue hook.
[ 93.295098] audit: type=1130 audit(1490601320.942:63): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 93.300060] audit: type=1131 audit(1490601320.947:64): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ OK ] Started Kdump Emergency.
[ 93.307304] audit: type=1130 audit(1490601320.955:65): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=emergency comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 93.311203] audit: type=1131 audit(1490601320.959:66): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=emergency comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
[ OK ] Reached target Emergency Mode.
[ OK ] Started Setup Virtual Console.
[ 93.344652] audit: type=1130 audit(1490601320.992:67): pid=1 uid=0 auid=4294967295
ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Starting Kdump Error Handler...
Kdump: Executing default action kdump_emergency_shell
Warning: /dev/mapper/fedora-root does not exist
Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.
kdump:/#