On 2021-07-17 1:18 p.m., Joe Zeff wrote: Here is the top of the current systemd-analyse blame output: $ systemd-analyze blame 27.625s plymouth-quit-wait.service 21.726s udisks2.service 13.932s libvirtd.service 13.711s systemd-journal-flush.service 12.946s abrtd.service 10.635s polkit.service 10.256s upower.service 9.109s NetworkManager-wait-online.service 9.030s lvm2-monitor.service 8.490s firewalld.service 8.208s sssd.service 7.309s accounts-daemon.service 6.730s initrd-switch-root.service 6.071s ModemManager.service 6.070s avahi-daemon.service 5.765s lm_sensors.service 5.761s rtkit-daemon.service 5.760s switcheroo-control.service 5.759s systemd-machined.service 5.758s thermald.service 5.757s systemd-homed.service 5.745s dbus-broker.service 4.511s fwupd.service 4.347s packagekit.service 4.107s chronyd.service 3.696s cups.service 2.850s systemd-udevd.service 2.661s geoclue.service 2.587s NetworkManager.service 2.040s wpa_supplicant.service 1.589s systemd-tmpfiles-setup-dev.service 1.516s systemd-resolved.service 1.207s gdm.service 1.105s systemd-logind.service 1.077s dracut-initqueue.service 904ms auditd.service 844ms systemd-fsck@dev-disk-by\x2duuid-bd603b31\x2d7ca0\x2d483e\x2daa91\x2d202c876783f2.service 669ms colord.service 649ms systemd-fsck@dev-disk-by\x2duuid-b572a339\x2d50fe\x2d4574\x2da299\x2d8d3fa9f3efa0.service 528ms systemd-zram-setup(a)zram0.service 494ms systemd-fsck@dev-disk-by\x2duuid-691af7ca\x2d5f30\x2d454b\x2d832f\x2d6ef37b60060a.service

I'm not sure if I'm able to correct you, but: cat /usr/lib/systemd/system/plymouth-quit-wait.service gives [Unit] Description=Hold until boot process finishes up After=rc-local.service plymouth-start.service systemd-user-sessions.service I'm unsure what systemd-user-sessions.service really is for. It's an binary file ! anyway, on an Intel i7-6700 with an Samsung 970 EVO Plus nmve and disabled services I don't need I get this: 2.653s plymouth-quit-wait.service 758ms upower.service 704ms firewalld.service 673ms sssd.service 639ms dracut-initqueue.service 450ms udisks2.service 417ms initrd-switch-root.service 340ms vboxdrv.service 312ms smb.service 301ms httpd.service 199ms cups.service 183ms systemd-resolved.service 165ms polkit.service 147ms avahi-daemon.service 139ms user(a)1000.service 138ms power-profiles-daemon.service 138ms initrd-parse-etc.service 136ms rtkit-daemon.service 132ms switcheroo-control.service 129ms systemd-homed.service 129ms thermald.service 100ms systemd-udev-trigger.service 77ms auditd.service 76ms dbus-broker.service 72ms systemd-vconsole-setup.service 68ms systemd-tmpfiles-setup.service 67ms lm_sensors.service 66ms smartd.service 59ms systemd-logind.service 58ms systemd-journal-flush.service 54ms sssd-kcm.service 50ms dev-zram0.swap 45ms systemd-udevd.service 44ms dracut-cmdline.service 42ms radicale.service 42ms systemd-tmpfiles-setup-dev.service 38ms colord.service 37ms accounts-daemon.service 37ms chronyd.service 36ms plymouth-switch-root.service 34ms dracut-pre-pivot.service 34ms uresourced.service 32ms systemd-random-seed.service 31ms gdm.service 27ms NetworkManager.service 25ms systemd-userdbd.service 24ms plymouth-start.service 23ms rc-local.service 23ms systemd-tmpfiles-clean.service 22ms import-state.service 22ms systemd-fsck@dev-disk-by\x2duuid-ac2d3c23\x2d941f\x2d427a\x2d9038\x2d735bdc0a4636.service 18ms dracut-pre-udev.service 17ms systemd-fsck-root.service 17ms initrd-cleanup.service 16ms netcf-transaction.service 16ms systemd-journald.service 15ms systemd-zram-setup(a)zram0.service 13ms dev-hugepages.mount 13ms plymouth-read-write.service 13ms dev-mqueue.mount 12ms sys-kernel-debug.mount 12ms systemd-user-sessions.service 11ms kmod-static-nodes.service 11ms systemd-remount-fs.service 11ms user-runtime-dir(a)1000.service 10ms systemd-update-utmp.service 10ms home-ron-DATA-devel-CCACHE.mount 10ms wpa_supplicant.service 10ms systemd-sysctl.service 9ms vboxballoonctrl-service.service 7ms initrd-udevadm-cleanup-db.service 7ms systemd-modules-load.service 7ms modprobe(a)drm.service 6ms systemd-update-utmp-runlevel.service 5ms boot.mount 4ms home.mount 3ms modprobe(a)configfs.service 3ms sys-fs-fuse-connections.mount 3ms home-ron-DATA.mount 2ms dracut-shutdown.service 2ms sys-kernel-config.mount 2ms modprobe(a)fuse.service 2ms tmp.mount 2ms var-tmp.mount and systemd-analyze plot > /tmp/boot-graph.svg && firefox /tmp/boot-graph.svg with an homebrewed 5.13.2 kernel gives Startup finished in 1.755s (kernel) + 1.313s (initrd) + 4.063s (userspace) = 7.131s graphical.target reached after 4.054s in userspace

Thanks for the info. I'll test to disable plymouth on my brother's box. He is also suffering from long boot times on an aged Intel i3.

On Sun, Jul 18, 2021 at 11:12 AM Joe Zeff <joe(a)zeff.us&gt; wrote: This counts the time to get to the login window, plymouth itself doesn't have a meaningful amount of startup time cost. It's just getting dinged because it's waiting for everything else required for startup prior to it being able to exit. So you can ignore it. All three of these are suspiciously long. Post /etc/fstab, /proc/cmdline, and the output from 'journalctl -b -o short-monotonic --no-hostname > journal.log' -- Chris Murphy

OK I'm skipping ahead because I got an email offlist from John with the journal. Hopefully he will post a pastebin or use file sharing service to share the journal so others can pitch in on the diagnosis. I'm seeing quite a lot of gaps in the time stamps (why I like monotonic time, makes such gaps obvious), related to audit... just two examples: [ 6.466561] systemd-sysv-generator[503]: SysV service '/etc/rc.d/init.d/livesys-late' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. [ 10.084941] kernel: audit: type=1334 audit(1626714308.750:61): prog-id=17 op=LOAD [ 27.581467] avahi-daemon[684]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70). [ 36.211086] audit: BPF prog-id=30 op=LOAD That's huge. But I don't know what's going on in during that time. If it's really audit or something else? On the one hand I'd like to see the journal again with boot parameter 'systemd.log_level=debug' and maybe that will give more details in the gaps? The problem is that it's extremely verbose and makes boot much slower, so it's possible we'll start to see other issues, like even more delays from journald flushing to disk. The fact journald is flushing to disk so slowly makes me wonder if the SSD is becoming slow. There's a variety of pre-death behaviors for SSDs, slowness is one of them. Of course folks running Btrfs can see this show up initially as transient corruptions of data or metadata (or both). Given that data is huge in comparison to metadata, it has a giant bullseye on its back, so data corruptions on Btrfs are often an early warning sign of impending hardware issues. Another possibility is 'rd.debug' to see what's going on with dracut/initramfs but I think that's even more verbose and not yet the direction to go in. One of the pieces of nearly weird ancient junk I have is a 2011 macbook pro that I added a SATA SSD drive to ages ago, and it boots to a login window in about 9 seconds. No special tuning on my part. [ 44.282677] kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready [ 46.157461] audit[689]: NETFILTER_CFG table=firewalld:1;mangle_PREROUTING_POLICIES_pre:240 family=1 entries=2 op=nft_register_chain pid=689 subj=system_u:system_r:firewalld_t:s0 comm="firewalld" Only a couple of seconds, but again it's audit as the second listed item following the delay. And 2 seconds is a ton of time, really! [ 46.274996] audit[689]: NETFILTER_CFG table=firewalld:1;__set1:260 family=1 entries=0 op=nft_register_set pid=689 subj=system_u:system_r:firewalld_t:s0 comm="firewalld" [ 51.976223] audit[689]: NETFILTER_CFG table=firewalld:1;__set1:260 family=1 entries=260 op=nft_register_setelem pid=689 subj=system_u:system_r:firewalld_t:s0 comm="firewalld" 5-6 seconds, it's really extreme. But no messages during the delay period, so it's hard to tell if it's audit itself, or firewalld, or by extension something network related. Or none of the above. And at this point it doesn't matter what else the logs say because this is already too long... but... [ 65.033121] gnome-session[1034]: gnome-session-binary[1034]: WARNING: Falling back to non-systemd startup procedure due to error: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.freedesktop.systemd1" does not exist [ 68.291154] audit[1129]: NETFILTER_CFG table=filter family=2 entries=19 op=xt_replace pid=1129 subj=system_u:system_r:iptables_t:s0-s0:c0.c1023 comm="iptables" [ Another delay, audit and networking related. [ 77.687346] org.gnome.Shell.desktop[1523]: Errors from xkbcomp are not fatal to the X server [ 79.685364] abrt-server[1151]: Deleting problem directory oops-2021-07-19-13:06:04-904-0 (dup of oops-2021-06-15-09:54:58-776-0) [ 80.745381] systemd[1]: Created slice Slice /system/dbus-:1.13-org.freedesktop.problems. None of these are obviously networking or audit related. Just slow. 3 seconds delay to discover and delete an abrt directory? I'm getting suspicious of SSD performance (bandwidth and latency). What's the output from: smartctl -x /dev/sdX Even going back through the log very superficially, I'm getting suspicious that the times that aren't showing up as obvious gaps, are just taking too long - multiple seconds for gnome shell stuff? That's more like HDD bandwidth and (especially) latency. [ 129.512520] PackageKit[1220]: resolve transaction /979_adbedcab from uid 1000 finished with success after 10999ms [ 137.008831] realmd[1457]: quitting realmd service after timeout Yeah ok wow, so packagekit even says 11s to do some basic stuff? Okayyy... uhh what is this?? [ 1.903084] kernel: i915 0000:00:02.0: [drm] *ERROR* Stolen reserved area [mem 0xdf100000-0xdf1fffff] outside stolen memory [mem 0x9f200000-0xc11fffff] [ 1.903856] kernel: i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem hmmm. Chris Murphy

On 2021-07-27 9:08 a.m., old sixpack13 wrote: I ran the extended built-in BIOS diagnostics on this Lenovo P300 machine again last week for half a day, and no faults were found. I also ran the GTKstresstest and stress apps for a day, also with no faults found.  The machine is solid and I'm unhappy that it is Fedora that is unstable on this hardware. On a hunch, I disabled the Lenovo deep C-state handling in the BIOS, reinstalled using BTRFS again and mostly recovered from backups.  Its a home daily-driver, so the extra power utilization on the disabled C-states is not really important.  Since this issue seems to be restricted to onkly my main Lenovo machine and not the other two or any on my Dells, I'm assuming here that the kernel is incorrectly handling C-states on IBM/Lenovo motherboards, but no proof either way so far.  I'm waiting and watching, but so far have not seen the extreme delay problem reoccur. I have had the GUI lockup twice, and have attributed that to defective hardware optimizations in Firefox, which I have now disabled.  So far the forced power-button shutdown that is required to recover from the lockup issue has not resulted in a read-only BTRFS root fault like it has in the past.  I'm waiting and watching on this issue as well. -- John Mellor

... +++ EDIT +++ I should have read the first thread again: it's an Intel GPU. anyway, after Crtl+Alt+F3 you should be able to do "sync && sync && sudo systemctl reboot" saves the headache about an possible (?) brtfs filesystem corruption when doing a "hardcore power off" IIRC, a brtfs scrub ... afterwards could help

Maybe your problem is something else then just a "GPU hang" ? In my case I evertime could switch to tty3. mouse is moveable/visible but after a click nothing happens, as described in the bug report. Keyboard is working as usual. just the GUI is dead. maybe you could ssh into the box next time to see if it is the *not* GPU and save the money. or find something investigating the log's after the reboot (but I guess you checked that alraedy)

... thanks chris for detailed info's, it helps to lower my fear. but as I mentioned in my last comment to John, I always was able to shut the box down in an civilized manner.

On 7/28/21 8:07 AM, John Mellor wrote: Regarding btrfs corruption issues... I recently upgraded to a new desktop PC, my first with NVMe drives (no spinning disks). FC33 installed OK, then I'd copy files from an old PC to the new one. Before too long, I'd start to see messages about I/O errors and the NVMe-based file system would be remounted as read-only. dmesg always showed I/O errors. The PC was under warranty so I shipped it back to the manufacturer. They discovered that the NVMe drive was OK _but_ one of the memory DIMMs was faulty. They replaced the DIMM and shipped it back to me. I've not had a single disk I/O error since then. Therefore, if you're experiencing file system corruption, you might want to run a memory test to see if perhaps your system has faulty RAM. Dave

On Sun, 2021-08-15 at 20:30 -0400, John Mellor wrote: This would merit a BZ report surely? poc

Saw on reddit about a 90 second stall. https://www.reddit.com/r/Fedora/comments/ogr3pq/f34_boot_stalls_for_addit... Edit /usr/lib/systemd/system/dbus-broker.service and remove "sysinit.target" from the "After=" line so it is just "After=dbus.socket" Then do a 'dracut -fv --regenerate-all' On 7/17/21 9:35 AM, John Mellor wrote: -- Brian Millett "Smooth. You're getting good at this. Keep working on it. One of these days I might even be convinced that you're human." -- [ Garibaldi (to Bester), "A Race Through Dark Places"]