As promised during yesterday's meeting, I looked at the proposed test case: https://fedoraproject.org/wiki/User:Tablepc/Draft_testcase_reboot I have multiple comments: - It is trying to check two things at the same time: 1. that reboot/shutdown works as expected 2. that filesystems are properly unmounted. I believe there should be two separate test cases for these. So please split these into two. - Formatting needs be fixed to make it look like our other test case pages. Currently it looks like a copy-paste from email without effort to use wiki formatting. - Basics don't need to be explained, because the required knowledge level for performing the test case guarantees that the tester knows e.g. how to log in. So for example the first two points can be shortened to "Switch to a free virtual console using Ctrl+Alt+F<n> shortcut and log in". - We don't need to mandate a particular disk layout in test case setup. It is more useful for different testers to have different environments, so that they have a higher chance to detect a bug. - I don't like checking pre-shutdown messages using halt very much. The first problem is that with plymouth installed (the default), you won't see the messages, but a frozen plymouth screen (unless you're quite fast and switch it to console messages before it freezes). The second problem is that it relies too much on user intuition in how to distinguish a success or a failure state. There is no example of either. Additionally, do we know for sure that a system that can't unmount filesystems will halt eventually? I'd expect it to hang forever. I'd rather leave all the error checking for the subsequent boot (or in the case of a hanging boot, it's obviously broken - but we won't need a test case for it, because you'll easily see it with regular interaction with the system). - When checking boot logs for fsck fixes, it's important to show an example of not just a successful case, but also of a failed case. And I seem to be lucky today [1]. - When testing system shutdown methods, I'd only use reboot and poweroff. Halt is very niche and shutdown is old and replaced by poweroff. Kamil [1] -- Logs begin at Tue 2019-08-27 09:26:40 CEST, end at Tue 2019-11-26 14:50:14 CET. -- Nov 25 10:25:20 phoenix systemd-fsck[684]: root: recovering journal Nov 25 10:25:20 phoenix systemd-fsck[684]: root: Clearing orphaned inode 12325283 (uid=1000, gid=1000, mode=0100644, size=641092) Nov 25 10:25:20 phoenix systemd-fsck[684]: root: Clearing orphaned inode 12331101 (uid=1000, gid=1000, mode=0100644, size=641092) .. Nov 25 10:25:20 phoenix systemd-fsck[684]: root: clean, 1023215/26869760 files, 46957728/107451392 blocks Nov 25 09:25:22 phoenix systemd-fsck[877]: boot: recovering journal Nov 25 09:25:22 phoenix systemd-fsck[878]: fsck.fat 4.1 (2017-01-24) Nov 25 09:25:22 phoenix systemd-fsck[878]: 0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt. Nov 25 09:25:22 phoenix systemd-fsck[878]: Automatically removing dirty bit. Nov 25 09:25:22 phoenix systemd-fsck[878]: Performing changes. Nov 25 09:25:22 phoenix systemd-fsck[878]: /dev/nvme0n1p1: 34 files, 6897/51145 clusters Nov 25 09:25:22 phoenix systemd-fsck[877]: boot: clean, 103/65536 files, 67833/262144 blocks _______________________________________________ test mailing list -- test(a)lists.fedoraproject.org To unsubscribe send an email to test-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org

On Wed, Nov 27, 2019 at 9:17 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote: > > A fair point about VM testing is whether the disk cache mode affects > the outcome. I use unsafe because it's faster and I embrace misery. I > think QA bots are now mostly using unsafe because it's faster too. So > depending on the situation it may be true that certain corruptions are > expected if unsafe is used, but I *think* unsafe is only unsafe in the > event the host crashes or experiences a power failure. I do forced > power offs of VMs all the time and never lose anything, in the case of > ext4 and XFS, journal replay always makes the file system consistent > again. And journal replay in that example is expected, not a bug. By "that example", do you mean the story you just described, or the "bad result example" from the test case?

Because in that test case example, if the machine was correctly powered off/rebooted, there should be no reason to reply journal or see dirty bits.

> > > How to test, step 2 and 3: > This only applies to FAT and ext4. XFS and Btrfs have no fsck, both > depend on log replay if there was an unclean shutdown. Also, there are > error messages for common unclean shutdowns, and error messages for > uncommon problems. I think we only care about the former, correct? I believe so. Is there a tool that could tell us whether the currently mounted drives were mounted cleanly, or some error correction had to be performed? Because this is quickly getting into territory where we will need to provide a large amount of examples and rely on people parsing the output, comparing and (mis-)judging. The wording in journal output can change any time as well. And I don't really like that.

> Steps 4-7: I'm not following the purpose of these steps. What I'd like > to see for step 4, is, if we get a bad result (any result 2 messages), > we need to collect the journal for the prior boot: `sudo journalctl > -b-1 > journal.log` and attach to a bug report; or we could maybe > parse for systemd messages suggesting it didn't get everything > unmounted. But offhand I don't know what those messages would be, I'd > have to go dig into systemd code to find them. I think the purpose is to verify that both reboot and poweroff shut down the system correctly without any filesystem issues (which means fully committed journals and no dirty bits set).

FAT, ext4, and XFS all have a kind of "dirty bit" set upon mount. It's removed when cleanly unmounted. Therefore if the file system isn't mounted, but the "dirty bit" is set, it can be assumed it was not cleanly unmounted. Both kernel code and each file system's fsck can detect this, and the message you see depends on which discovers the problem first. The subsequent messages about how this problem is handled, I think we can ignore. As you say, it will be variable. All we care about is the indicator that it was not properly unmounted. Here are those indicators for each file system: FAT fsck (since /etc/fstab sets EFI system partition fs_passno to 2, this is what's displayed for default installations) Nov 28 12:04:21 localhost.localdomain systemd-fsck[681]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt. FAT kernel [ 205.317346] FAT-fs (vdb1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck. ext4 fsck (since /etc/fstab sets /, /boot, /home fs_passno to 1 or 2, this is what's displayed for default installations) Nov 28 12:07:21 localhost.localdomain systemd-fsck[681]: /dev/vdb2: recovering journal ext4 kernel [ 316.756778] EXT4-fs (vdb2): recovery complete XFS kernel (since /etc/fstab sets / fs_passno to 0, we should only see this message with default installations) [ 372.027026] XFS (vdb3): Starting recovery (logdev: internal) If the test case is constrained only to default installations, the messages to test for: "0x41: Dirty bit is set" "recovering journal" "XFS" and "Starting recovery" If the test case is more broad, to account for non-default additional volumes that may not be set in fstab or may not have fs_passno set, also include: "EXT4-fs" and "recovery complete" "FAT-fs" and "Volume was not properly unmounted" In each case I'm choosing the first message that indicates previously unclean shutdown happened. Whether fsck or kernel message, they should be fairly consistent in that I'm not expecting them to change multiple times per year.

The gotcha is, how would we know? Failure to automatically parse for these messages, should they change, will indicate a clean shutdown. *shrug*

>> Steps 4-7: I'm not following the purpose of these steps. What I'd like >> to see for step 4, is, if we get a bad result (any result 2 messages), >> we need to collect the journal for the prior boot: `sudo journalctl >> -b-1 > journal.log` and attach to a bug report; or we could maybe >> parse for systemd messages suggesting it didn't get everything >> unmounted. But offhand I don't know what those messages would be, I'd >> have to go dig into systemd code to find them. > > > I think the purpose is to verify that both reboot and poweroff shut down the system correctly without any filesystem issues (which means fully committed journals and no dirty bits set). Gotcha. Yeah, I think it's reasonable to test the LiveOS reboot as well as the installed system's reboot, to make sure they are both properly unmounting file systems.

On Fri, Nov 29, 2019 at 5:10 AM Kamil Paral <kparal(a)redhat.com&gt; wrote: > > Sounds reasonable to test both LiveOS and the installed system. Does it make sense to test both installed system's reboot and poweroff, though? Are there any meaningful differences? i'm not certain.

On Fri, Nov 29, 2019 at 7:30 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote: > On Fri, Nov 29, 2019 at 5:10 AM Kamil Paral <kparal(a)redhat.com&gt; wrote: >> >> Sounds reasonable to test both LiveOS and the installed system. Does it > make sense to test both installed system's reboot and poweroff, though? Are > there any meaningful differences? > > i'm not certain. > In that case I'd probably start with the simpler version, and extended it if we detect it's not sufficient. So one reboot after installation -> fsck check -> one reboot from the installed system -> fsck check. Pat, can you update the test case according to the changes we've discussed above? Thanks.

I have the update ready. I think I interpolated the discussion correctly, but probably leaned toward the "keep it simple for now" case. Please let me know if there are further changes needed. Here's the link: https://fedoraproject.org/wiki/User:Tablepc/Draft_testcase_reboot

On Mon, Dec 2, 2019 at 7:43 PM pmkellly(a)frontier.com <pmkellly(a)frontier.com&gt; wrote: > I have the update ready. I think I interpolated the discussion > correctly, but probably leaned toward the "keep it simple for now" case. > Please let me know if there are further changes needed. Here's the link: > > https://fedoraproject.org/wiki/User:Tablepc/Draft_testcase_reboot I was hoping you would include `journalctl | grep` commands that would look for the strings Chris mentioned, so that it's easy for people or for scripts to confirm whether the filesystem was or wasn't properly unmounted. It's still helpful to include the examples (they don't necessarily need to be Expected Results, but it's fine there), so that people can compare the full text if they want or have doubts, but those grep commands would make the comparison much simpler.

I believe the poweroff cycle can be left out, and we can ask just for a single reboot.

Also, can you please fix the formatting in Expected Results?

Here's the email I had in mind, containing the important journal messages: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.or... We need to create a command or commands that will detect any of the failure states (and ideally another command for any of the success states, for confirmation). It doesn't really matter if you grep the journal directly or save the journal and grep the file. But it's one fewer command to grep the journal directly. And if a failure is found, ask the user to save the journal and report a bug.

On 12/5/19 03:56, Kamil Paral wrote: > > Here's the email I had in mind, containing the important journal messages: > https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.or... > > We need to create a command or commands that will detect any of the failure > states (and ideally another command for any of the success states, for > confirmation). > > > It doesn't really matter if you grep the journal directly or save the > journal and grep the file. But it's one fewer command to grep the journal > directly. And if a failure is found, ask the user to save the journal and > report a bug. > I have the test case updated, but I ran into a problem with the test case template. The "|" character is apparently a reserved character in the template code. I couldn't get it to let me have the pipe from the journal to the grep. Nor could I use the -E option with the grep because that character is used to separate the match patterns.