Continued from the 'background and summary' email: https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject... Adding Hans and Matthew and Lennart to cc. Questions I have are: - WG is considering dropping creation of swap partitions by default, in favor of swap-on-ZRAM. Any concerns? (We do know it's not possible to use a ZRAM device for hibernation, but the kernel will look to another swap device for contiguous free space to write out a hibernation image.) - What's the status of s2idle in the kernel?

- What sort of work is needed outside the kernel to properly support s2idle, or is this predominantly kernel work? Microsoft documents on Modern Standby suggest minimal application effort. [1]

- Prospect of kernel support to separate swap and hibernation partitions (and/or swap files)? Or systemd method of creating then activating swapfiles on demand? - Prospect of hibernation supported with UEFI Secure Boot? - Is hibernation a better fallback than poweroff, given the significant reliability differential? Why? Poweroff is universal, hibernation isn't. What's the argument that a non-universally available fallback is better than the universal fallback? - What are the implications of hibernation if Fedora will move to measured boot? (I'm not sure how mainstream that function is expected to be, or it's use case specific opt-in.) - There's some anecdotal evidence users are disabling UEFI Secure Boot, possibly quite a lot [2]. Does there need to be an effort at making the signing of user built kernel and module easier? Can it be made easier? I don't know custom or out of tree modules is a significant motive for disabling SB, vs other explanations.

- A systemd TODO makes me wonder: Does anyone have (corroborating) data on the reliability of firmware or battery reporting, when the system, and thus the battery, are under significant load? [3] I've discussed with a reliable source that on 2+ year old hardware, the vast majority of batteries are effectively broken and aren't likely to report anything reliably if they are under significant load, in particular waking from S3. By anything, I mean, time remaining, power remaining, and current power consumption rate. Would s2idle instead of S3 would make this more reliable? - It doesn't sound like S1 is really used at all, even though kernel docs say it's supported as shallow/standby. (?) Is it more or less reliable than S3?

- I'm inclined to think we should mimic what hardware vendors, Microsoft, Apple, and Google (with Chromebooks and Android) have been doing for a while: faster boots, and S0 low power idle - and skip the things making devs and users crazy. But I invite a persuasive contrary argument.

The reason, at least what I found out, is that the kernel first copies the whole memory into your memory (ouch), before writing it to the swap partition. So if you have 16 GB RAM, you can't hibernate if you use more than 8 GB, and therefore an 8 GB swap partition (fully unoccupied) is enough for you. I have tested this multiple times, it works for me exactly as written. So, actually, a swap device sized 1:1 with RAM is already an overkill (unless you use more than 50% of swap size with just regular usage), and 0.5:1 ratio would be perfectly fine if you made sure that swap got used just for hibernation.

Where I came up with 2:1 is from anaconda/blivet code: anaconda/pyanaconda/storage/utils.py:642: :param bool hibernation: calculate swap size big enough for hibernation https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/storage/ut... Note on line 673 it actually could be 3x RAM, if --hibernation were used, but this flag isn't used on Fedora Workstation so this computation never gets used.

And yet there is a 'resume=UUID' boot parameter included. Why is this boot parameter set as if we're supporting hibernation out of the box?

On Tue, Feb 11, 2020 at 3:00 AM Kamil Paral <kparal(a)redhat.com&gt; wrote: > > On Mon, Feb 10, 2020 at 9:43 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote: >> >> Where I came up with 2:1 is from anaconda/blivet code: >> anaconda/pyanaconda/storage/utils.py:642: :param bool hibernation: >> calculate swap size big enough for hibernation >> https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/storage/ut... >> >> Note on line 673 it actually could be 3x RAM, if --hibernation were >> used, but this flag isn't used on Fedora Workstation so this >> computation never gets used. > > > I guess that's because if you search the Internets, the swap recommendations are all over the place, so some median numbers were picked :-) I had no idea about the hibernation limitation either, until the web browsers started to eat 4+GBs RAM... :-) > > The problem is that you don't know how much swap the system will use during regular usage, so you don't know how much headroom you need on top of the 0.5x mem size needed for hibernation. I think nowadays a new formula could be devised, something like `swap size = 0.5x mem size + 2 GB`. I think that's plenty for regular usage (even SSDs are too slow to be used as RAM, not to mention it will wear them out quickly) and it should allow for hibernation most of the time. I don't think that's enough to be reliable. Example system: 32G RAM, all of it used, plus 2G of page outs (into the swap device). + 2G already paged out to swap + 16GB needs to be paged out to swap, to free up enough memory to create the hibernation image

+ 8-16GB for the (compressed) hibernation image to be written to a *contiguous* range within swap That's 26-34G needed for the swap device. Since the swap device is shared for pages and hibernation image, the actual size isn't knowable until the approximate time hibernation is called.

As Bastien noted in the pagure/workstation issue: https://pagure.io/fedora-workstation/issue/121#comment-620831 I've started a followup thread upstream to get more information about all of this. Kamil, can you do a brief write up of your use case or reproduce steps? I want upstream to know that your case is real world, and not some already known contrived case. You can either reply here and I'll reference it on linux-mm@ or you can post directly to this thread if you prefer. Thanks. https://lore.kernel.org/linux-mm/CAA25o9TvFMEJnF45NFVqAfdxzKy5umzHHVDs+SC...

>> And yet there is a 'resume=UUID' boot >> parameter included. Why is this boot parameter set as if we're >> supporting hibernation out of the box? > > > Probably because of this? > https://bugzilla.redhat.com/show_bug.cgi?id=1206936 > https://github.com/rhinstaller/anaconda/pull/1360 Right but anaconda is not called with --hibernation is what I mean. The higher partition requirements for hibernation are tied to the --hibernation flag, but the resume= boot parameter insertion is not. And it seems to me those things should be tied to --hibernation, and then also whether --hibernation is used should be an edition specific decision, and that it should be explicit because otherwise we're just setting users up for misadventure.

> And yes, we currently support hibernation out of the box, and it works. If you don't have SecureBoot enabled, and if you use non-GNOME or know how to call it from GNOME. It definitely doesn't work in all cases, and we don't have any evidence that it works in most cases.

I have a laptop that 100% of the time fails to resume from hibernation because the hibernation image is considered corrupt by the kernel. I don't know if it's corrupt during image write out, or read in. S4 relies on ACPI, and isn't even reliable on Windows or macOS 100% of the time. And as you've experienced, S3 isn't certainly reliable. Whereas S0 low power mode is reliable, which is why so much effort is going there. Just leave ACPI and the (logic board) firmware out of consideration. I would rather hibernation work. But I don't think it's OK to, by default, create huge swap partitions when this very clearly is not at all reliable as evidenced by your own experience, which requires either luck or esoteric knowledge to get the active page amount below some ~50% threshold in order for hibernation image creation to succeed.

On Thu, Feb 13, 2020 at 9:32 AM Kamil Paral <kparal(a)redhat.com&gt; wrote: > > On Tue, Feb 11, 2020 at 11:48 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote: >> Kamil, can you do a brief write up of your use case or >> reproduce steps? I want upstream to know that your case is real world, >> and not some already known contrived case. You can either reply here >> and I'll reference it on linux-mm@ or you can post directly to this >> thread if you prefer. Thanks. >> >> https://lore.kernel.org/linux-mm/CAA25o9TvFMEJnF45NFVqAfdxzKy5umzHHVDs+SC... > > > Hmm, what exactly do you want to know? :) Why I use hibernation? Reproduce steps for hibernation failing. I don't know if it's as easy as "Launch Firefox, and load 34 pages of bbc.com, then try to hibernate with 'systemctl hibernate'" - and voila, not enough memory error.

> Yes, in my experience it is that easy. If I look into htop before hibernating and I'm over 50% usage (not counting buffers/cache I'm not familiar with htop. If I run it without options, I see on the top left my CPUs followed by "Mem" which seems to be the same thing as /proc/meminfo's "Active(file):" value (if I divide by 1MiB). Is your 50% usage based on this "Mem" reporting? There isn't a corresponding value in top, the "Mem ... Used" value is totally different there.

> Feb 14 12:03:28 titan kernel: PM: hibernation exit > Feb 14 12:03:28 titan systemd-sleep[3424]: Failed to suspend system. System resumed again: Cannot allocate memory This is a systemd error, without a corresponding kernel message. So is it really a kernel complaint or problem? Looking at systemd ./src/sleep/sleep.c 220 r = write_state(&f, states); 221 if (r < 0) 222 log_struct_errno(LOG_ERR, r, 223 "MESSAGE_ID=" SD_MESSAGE_SLEEP_STOP_STR, 224 LOG_MESSAGE("Failed to suspend system. System resumed again: %m"), 225 "SLEEP=%s", arg_verb); I don't see where "Cannot allocate memory" comes from in systemd, but in the kernel: ./Documentation/admin-guide/sysctl/vm.rst:864:"fork: Cannot allocate memory". which is in https://www.kernel.org/doc/html/v5.5/admin-guide/sysctl/vm.html under user_reserve_kbytes Default value on Fedora is $ cat /proc/sys/vm/user_reserve_kbytes 131072 $ cat /proc/sys/vm/overcommit_memory 0 Hmm. Anyway, I'm not sure what's running out of memory or why. Right before this, hibernation exit comes from ./power/hibernate.c:781: pr_info("hibernation exit\n"); Looks like it got to here: 732 error = hibernation_snapshot(hibernation_mode == HIBERNATION_PLATFORM); 733 if (error || freezer_test_done) 734 goto Free_bitmaps; *shrug* It definitely didn't get to writing out the hibernation image.

But maybe you can beat me to it, and provide a /proc/meminfo prior to hibernation attempt, after it fails (used memory above 50%); and then use whatever technique you normally use to get used memory below 50%, then capture another /proc/meminfo, successfully hibernate and resume, then take a fourth /proc/meminfo.

My test: Fedora Workstation 31, laptop with 8G RAM, 8G swap partition, fill up memory using Firefox tabs pointed to various websites, and then I followed [1] to issue two commands: # echo reboot > /sys/power/disk # echo disk > /sys/power/state I experience twice as many failures as successes. Curiously, the successes show pageout does happen. Before hibernate there is no swap in-use, but after resume ~2GiB swap is in-use and RAM usage is about 50%.

Nope, it's a major clue! This comes up all the time with performance complaints about swap, oh hey throw this vm.swappiness=0 spaghetti at the wall. It's entirely plausible this is the origin of this 50% business. I'll ask about it on linux-mm@. And I think you're correct to point out that this needs to be a documented consequence of using this value. But how in the world did you get suspicious of your custom vm.swappiness value? :D

Even in the idealized VM environment, I've had a couple failures just after hibernation entry where the VM hangs indefinitely and has to be killed off. I didn't have a serial console setup to see if the problem can be captured via virsh console.

On Wed, Feb 19, 2020 at 10:13 PM Chris Murphy <lists(a)colorremedies.com&gt;

> > My test: Fedora Workstation 31, laptop with 8G RAM, 8G swap partition, > fill up memory using Firefox tabs pointed to various websites, and > then I followed [1] to issue two commands: > > # echo reboot > /sys/power/disk > # echo disk > /sys/power/state > > I experience twice as many failures as successes. Curiously, the > successes show pageout does happen. Before hibernate there is no swap > in-use, but after resume ~2GiB swap is in-use and RAM usage is about > 50%. Sigh. Turns out this is "my" mistake. 🤦 Hibernation apparently gets

https://lore.kernel.org/linux-mm/CAA25o9Q=36fiYHtbpcPPmGEPnORm2ZM7MfqRcsv...

contributor's workload has a lot of file pages, so that discarding them is enough for the image allocator to succeed. In that case "sync; echo 1 > /proc/sys/vm/drop_caches" would be a better way of achieving the same result.

On Tue, Feb 11, 2020 at 11:48 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote: > On Tue, Feb 11, 2020 at 3:00 AM Kamil Paral <kparal(a)redhat.com&gt; wrote: > > > > On Mon, Feb 10, 2020 at 9:43 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote: > >> > >> Where I came up with 2:1 is from anaconda/blivet code: > >> anaconda/pyanaconda/storage/utils.py:642: :param bool hibernation: > >> calculate swap size big enough for hibernation > >> https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/storage/ut... > >> > >> Note on line 673 it actually could be 3x RAM, if --hibernation were > >> used, but this flag isn't used on Fedora Workstation so this > >> computation never gets used. > > > > > > I guess that's because if you search the Internets, the swap recommendations are all over the place, so some median numbers were picked :-) I had no idea about the hibernation limitation either, until the web browsers started to eat 4+GBs RAM... :-) > > > > The problem is that you don't know how much swap the system will use during regular usage, so you don't know how much headroom you need on top of the 0.5x mem size needed for hibernation. I think nowadays a new formula could be devised, something like `swap size = 0.5x mem size + 2 GB`. I think that's plenty for regular usage (even SSDs are too slow to be used as RAM, not to mention it will wear them out quickly) and it should allow for hibernation most of the time. > > I don't think that's enough to be reliable. > > Example system: 32G RAM, all of it used, plus 2G of page outs (into > the swap device). > > + 2G already paged out to swap > + 16GB needs to be paged out to swap, to free up enough memory to > create the hibernation image If I understood the kernel discussions correctly, currently there's no simple and reliable mechanism to achieve this (move the excess memory to swap). So I wouldn't count it like this. If you have more than 50% memory occupied, you're out of luck, hibernation will be a no-op operation (it would be nice if it returned some user-friendly GUI message outside of system journal). If you are under 50% memory utilization, but your swap space is insufficient, the hibernation will also be aborted (this time not immediately, though, but only after it compresses the memory and finds out it doesn't fit into the free swap space). Again, some visible error message would be nice. It sounds like there are too many cases where the functionality fails, and that's true, but that's what we have, and often I consider it better to close a few extra tabs than having to power off. But it's annoying and far from polished, yes. > + 8-16GB for the (compressed) hibernation image to be written to a > *contiguous* range within swap > > That's 26-34G needed for the swap device. Since the swap device is > shared for pages and hibernation image, the actual size isn't knowable > until the approximate time hibernation is called. Windows, AFAIK, pre-allocates the hibernation image and doesn't share it with swap space. It's very inefficient regarding disk size, but it improves reliability, obviously. Btw, you have a very good remark that the memory image gets compressed. So for hibernation image, you can know its max size, but you can't know the optimal size - that depends on how well your memory gets compressed. And that's why you can often hibernate even with quite small swap - it just compressed well. > > As Bastien noted in the pagure/workstation issue: > https://pagure.io/fedora-workstation/issue/121#comment-620831 > > I've started a followup thread upstream to get more information about > all of this. Kamil, can you do a brief write up of your use case or > reproduce steps? I want upstream to know that your case is real world, > and not some already known contrived case. You can either reply here > and I'll reference it on linux-mm@ or you can post directly to this > thread if you prefer. Thanks. > > https://lore.kernel.org/linux-mm/CAA25o9TvFMEJnF45NFVqAfdxzKy5umzHHVDs+SC... Hmm, what exactly do you want to know? :) Why I use hibernation? On my desktop: * Due to some hardware/firmware flaky-ness I can't suspend to RAM, because I sometimes get random memory corruption on resume (note: this is not faulty hardware in the usual sense, the problem occurs *only* when suspending to RAM, everything is super reliable otherwise). So I hibernate. I don't want to log in and open all the windows every time I return to the computer (that would be the case if I powered it off). Even if I didn't have flaky hardware, I'd want sometimes to suspend to RAM and sometimes hibernate, depending on whether I want to turn off the power strip at that time (e.g. when leaving the flat for a few days, during a big storm, when having other appliances in the same power strip and wanting to save power, etc). On my laptop: * I use suspend to RAM, and it works well. But occasionally the power drain is surprisingly high during suspend (perhaps a firmware issue) and the battery can be depleted overnight. Usually it can last 2-3 days, which is fine during work-week, but it might be risky over the weekend. It requires you to keep track of the battery level when suspending (especially on Fridays), or simply power it off. I don't want to say it's a problem, it's not, just not as comfortable as on Windows. A hybrid sleep would work wonders here. You simply suspend it and it hibernates after a few hours automatically. However, GNOME doesn't actually allow me configure it and it even overrides systemd configs. I'd ideally want to pick the right action that suits my current need (suspend, hybrid sleep, hibernate) with a configurable default for closing the lid.

> >> And yet there is a 'resume=UUID' boot > >> parameter included. Why is this boot parameter set as if we're > >> supporting hibernation out of the box? > > > > > > Probably because of this? > > https://bugzilla.redhat.com/show_bug.cgi?id=1206936 > > https://github.com/rhinstaller/anaconda/pull/1360 > > Right but anaconda is not called with --hibernation is what I mean. > The higher partition requirements for hibernation are tied to the > --hibernation flag, but the resume= boot parameter insertion is not. > And it seems to me those things should be tied to --hibernation, and > then also whether --hibernation is used should be an edition specific > decision, and that it should be explicit because otherwise we're just > setting users up for misadventure. I somewhat disagree. The --hibernation flag just affects swap size. Larger swap makes it a bit more likely that hibernation will work, yes. But that doesn't mean it wouldn't work even with smaller size swap. The resume= parameter is required in all cases, whether you target a system that has large swap or medium swap size by default. Also, don't forget these are just default values, the users can change them during partitioning. It would be sad if the user intentionally configured large swap because she knows she wants to hibernate, and it wouldn't work just because the resume= parameter was missing, right? If some edition wants to prevent hibernation (that alone sounds like a bad idea), they can do it better by not offering the GUI option, or in the extreme case by somehow overriding "systemctl hibernate" functionality (let's not do that). If you allow to hibernate but break the resume (by omitting resume=), then the user just have lost some data.

I mean insofar as default/automatic partitioning, where the tentative idea is, no swap partition, and use swap-on-ZRAM instead. But if that seems dramatic,

it's effectively the case already with UEFI Secure Boot systems - the resume parameter is pointless, these users get a power off when the battery reaches a low threshold. That's data loss.

If we're not going to take hibernation functionality seriously anyway, I question carving out so much space from people's systems by default. I have no complaint about custom partitioning making a swap partition by default at 1:1 ratio or whatever is discovered to be most reliable without being wasteful, and also setting a resume parameter.