https://fedoraproject.org/wiki/Changes/EnableSystemdOomd == Summary == Provide a better experience for Fedora users in out-of-memory (OOM) situations by enabling [https://www.freedesktop.org/software/systemd/man/systemd-oomd.html systemd-oomd] by default. Actions taken by systemd-oomd operate on a per-cgroup level, aligning well with the life cycle of systemd units. systemd-oomd primarily uses [https://facebookmicrosites.github.io/psi/ Linux pressure stall information (PSI)] to make decisions based on wasted productivity due to resource shortages; in addition to that, it also supports swap based actions. == Owners == * Name: [[User:anitazha|Anita Zhang]], [[User:Dcavalca|Davide Cavalca]], [[User:Salimma|Michel Salim]], [[User:Htejun|Tejun Heo]], [[User:3ki|Rik van Riel]] * Email: the.anitazha(a)gmail.com, dcavalca(a)fb.com, michel(a)michel-slm.name, htejun(a)fb.com, riel(a)fb.com == Detailed description == The primary mechanism used by systemd-oomd for detecting when the system is out of memory is memory pressure. Memory pressure measures the percentage of time a cgroup has “wasted” due to lack of memory. This includes time spent reclaiming free memory, faulting in recently resident pages, and loading in anonymous pages from swap. When a monitored cgroup’s memory pressure exceeds the specified thresholds, systemd-oomd will perform action(s) on the targeted cgroup’s descendants, starting from the cgroups with the most reclaim scans. Reclaim activity is used here, rather than the largest consumer, as it reflects values set in the cgroup memory controller for memory protection (such as memory.low). For memory pressure configuration, this will be ManagedOOMMemoryPressure=kill and ManagedOOMMemoryPressureLimit=4% on user@.service to have systemd-oomd send SIGKILLs to all processes under a selected cgroup when total memory pressure on all tasks exceeds 4% for 10 seconds. For swap based actions, systemd-oomd will monitor the system-wide swap space and act when available swap falls below the configured threshold, starting with the cgroups with the highest swap usage to the least. Keeping some amount of swap (if enabled) available will prevent the kernel OOM killer from killing processes unpredictably and spending an unbounded amount of time afterwards. For swap configuration, this will be SwapUsedLimitPercent=90% in oomd.conf and ManagedOOMSwap=kill on -.slice (root cgroup slice) to have systemd-oomd send SIGKILLs to all processes under a cgroup when swap used exceeds 90%. == Benefit to Fedora == * Addressing the issue of improving user feedback in https://pagure.io/fedora-workstation/issue/202, systemd-oomd currently logs to the journal if pressure or swap action is about to occur. There are also debug logs, for each process that is sent a SIGKILL, that can be bumped up in priority. Further notification mechanisms (i.e. over dbus) can also be implemented depending on feedback. * While systemd-oomd is simpler in configuration to the oomd used at Facebook, the algorithm is largely the same. As such, the following case study can be used as an example of how PSI and cgroup killing can release memory not normally resolved with process killing and lead to better utilization: https://facebookincubator.github.io/oomd/docs/oomd-casestudy.html * OOM killing in userspace, before the kernel OOM killer kicks in, has been shown to be effective at keeping a system functional. An OOM kill in the kernel is slow, possibly leading to an unbounded amount of time swapping in and out pages and evicting the page cache. * PSI based actions, versus looking at raw memory consumption numbers, better reflect memory protection policies set for cgroup resource control limits (e.g. memory.low). == Scope == * Proposal owners: ** Implement and land additional refinements to systemd-oomd *** Remove swap as a hard requirement to running systemd-oomd *** Expand ManagedOOM*= properties to user units (currently only usable on system units) *** Configurable memory pressure time window knob ** Enable oomd by default with sensible configuration ** Test days ** Aid with documentation * Other developers: ** systemd: review PRs as needed * Release engineering: https://pagure.io/releng/issue/9913 * Policies and guidelines: N/A * Trademark approval: N/A == Upgrade/compatibility impact == Existing systems running earlyoom will not be modified. One can transition to systemd-oomd via: <pre>sudo systemctl disable --now earlyoom sudo systemctl enable --now systemd-oomd</pre> Systems that were previously not running earlyoom will have systemd-oomd enabled by default. == How to test == systemd 247 build for Fedora includes all the artifacts for systemd-oomd. It is disabled by default but can be started with: <pre>sudo systemctl enable --now systemd-oomd</pre> At this point you can decide which units to set properties on. For example, to enable swap-based killing on all units below the root slice: <pre>sudo systemctl edit --force -- -.slice [Slice] ManagedOOMSwap=kill # save and exit</pre> Note that the following memory pressure example requires the changes listed in “Scope” to work as expected, as systemd-oomd shipped with systemd v247 does not support changing the time window for memory pressure. This example was run on a system with swap: <pre>systemctl edit user@.service [Service] ManagedOOMMemoryPressure=kill ManagedOOMMemoryPressureLimit=4% # save and exit systemd-run --user tail /dev/zero # will lead to a lot of reclaim and then OOM if not killed</pre> == User experience == This should be a fully transparent change for users. == Dependencies == None. If changes to oomd are required to address feedback to this proposal, they will need to be merged in systemd. == Contingency plan == * Contingency mechanism: For workstation, owner will revert all changes and we’ll go back to using earlyoom instead * Contingency deadline: Final freeze * Blocks release? No * Blocks product? No == Documentation == https://www.freedesktop.org/software/systemd/man/systemd-oomd.html<br /> https://www.freedesktop.org/software/systemd/man/oomctl.html<br /> https://www.freedesktop.org/software/systemd/man/oomd.conf.html == Release Notes == systemd-oomd is enabled by default. Depending on which systemd units have ManagedOOMSwap=kill or ManagedOOMMemoryPressure=kill, systemd-oomd will SIGKILL all the processes under the appropriate descendant cgroups when the configured limits are exceeded. To revert back to earlyoom, run: <pre>sudo systemctl disable --now systemd-oomd sudo systemctl enable --now earlyoom</pre> See man oomd.conf for configuration options.

I think so. I designed the systemd-oomd interface to be general enough to eventually support sending d-bus signals in addition to or in place of killing cgroups with SIGKILL. But I'm uncertain if it can be implemented before the freeze.

== Upgrade/compatibility impact == Existing systems running earlyoom will not be modified. One can transition to systemd-oomd via: <pre>sudo systemctl disable --now earlyoom sudo systemctl enable --now systemd-oomd</pre> Systems that were previously not running earlyoom will have systemd-oomd enabled by default.

On Mon, 2020-12-21 at 10:57 -0600, Michael Catanzaro wrote: > On Mon, Dec 21, 2020 at 11:28 am, Ben Cotton <bcotton(a)redhat.com&gt; > wrote: > > == Upgrade/compatibility impact == > > > > Existing systems running earlyoom will not be modified. One can > > transition to systemd-oomd via: > > > > <pre>sudo systemctl disable --now earlyoom > > sudo systemctl enable --now systemd-oomd</pre> > > Systems that were previously not running earlyoom will have > > systemd-oomd enabled by default. > > I suggest we enable systemd-oomd for everyone on upgrade, to follow > our > design goal of ensuring upgraded systems match fresh installs as > closely as practical. We had thought about that, but one concern was migrating custom configuration that one might have for earlyoom, which could be tricky. If we're ok with unconditionally migrating to oomd with its default config, that should be pretty straightforward to do.

On Mon, Dec 21, 2020 at 9:59 am, Davide Cavalca via devel <devel(a)lists.fedoraproject.org&gt; wrote: > We had thought about that, but one concern was migrating custom > configuration that one might have for earlyoom, which could be > tricky. > If we're ok with unconditionally migrating to oomd with its default > config, that should be pretty straightforward to do. Yup, that's fine. It's expected that custom configuration goes away during a major architectural change like this.

On 12/21/20 8:28 AM, Ben Cotton wrote: > == Documentation == > > https://www.freedesktop.org/software/systemd/man/systemd-oomd.html<br /> > https://www.freedesktop.org/software/systemd/man/oomctl.html<br /> > https://www.freedesktop.org/software/systemd/man/oomd.conf.html > Be aware that if you intend to enable monitoring and actions on user.slice, user-$UID.slice, or their ancestor cgroups, it is highly recommended that your programs be managed by the systemd user manager to prevent running too many processes under the same session scope (and thus avoid a situation where memory intensive tasks trigger systemd-oomd to kill everything under the cgroup). If you're using a desktop environment like GNOME, it already spawns many session components with the systemd user manager. This makes me slightly very concerned. According to cgls, I have most of the apps running directly under user-$UID.slice -> session-X.scope. That includes a compositor (sway) and a few applications that consume uncomfortably close to 100% of available memory (firefox, thunderbird, clangd, gcc, etc...). My understanding is that unless I configure all of the above to run under dedicated scopes, an attempt to run a memory-intensive task would make systemd-oomd terminate the whole user slice with all my apps. Is there anything that could be improved for that scenario? I don't expect that all our desktop users would start using `systemd-run --scope --user` to launch their applications.

It's just that as a maintainer of one of those alternative desktop environments I have no idea how to make that work in default configuration.

And I wonder what will be the behavior for applications, which I start from my terminal? The most typical example for me is running GVim from gnome-terminal.

You see that I have executed FF from terminal. But I think that if any other process running in the app-org.gnome.Terminal.slice starts to grow memory usage, the whole tree will be killed.

On Tue, Dec 22, 2020, at 10:45 AM, Michael Catanzaro wrote: > Each gnome-terminal tab runs in its own cgroup: > > │ │ │ ├─app-org.gnome.Terminal.slice > │ │ │ │ > ├─vte-spawn-1d1d5519-71d2-4035-929a-8a9ae5bc3822.scope > │ │ │ │ │ ├─7848 bash > │ │ │ │ │ └─7889 less /etc/hosts > │ │ │ │ > ├─vte-spawn-03fe4cc9-0400-4723-ac9b-e929b850ca02.scope > │ │ │ │ │ ├─7892 bash > │ │ │ │ │ ├─7927 systemd-cgls > │ │ │ │ │ └─7928 less > │ │ │ │ └─gnome-terminal-server.service > │ │ │ │ └─7843 /usr/libexec/gnome-terminal-server > Glad to hear gnome-terminal separates tabs. Any idea if tilix does the same?

On Mon, Dec 21, 2020 at 1:51 pm, Aleksei Bavshin <alebastr89(a)gmail.com&gt; wrote: Hm good point. I think only GNOME and KDE create systemd scopes when launching apps; systemd-oomd is not going to work well in other desktops. Probably other desktop spins should opt-out of this change for now. Michael

If your desktop doesn't segregate apps and services into cgroups, systemd-oomd will kill the entire desktop whenever anything uses too much memory, because the desktop is going to be running in the same cgroup as the apps that it launches. So I think desktop spins (other than KDE) ought to opt out of this. It should be good for all Fedora editions, though (including Workstation, Server, Atomic, CoreOS), and also for KDE spin.

> If your desktop doesn't segregate apps and services into cgroups, > systemd-oomd will kill the entire desktop whenever anything uses too > much memory, because the desktop is going to be running in the same > cgroup as the apps that it launches. So I think desktop spins (other > than KDE) ought to opt out of this. It should be good for all Fedora > editions, though (including Workstation, Server, Atomic, CoreOS), and > also for KDE spin. How will this work on headless systems like Fedora Server, Atomic, and CoreOS? Will it be expected that users manually create their own cgroups?

So are you confirming that users are now going to need to place things in their own cgroup if they do not want systemd-oomd to potentially kill the single cgroup containing all of their running applications?

> If your desktop doesn't segregate apps and services into cgroups, > systemd-oomd will kill the entire desktop whenever anything uses too > much memory, because the desktop is going to be running in the same > cgroup as the apps that it launches. So I think desktop spins (other > than KDE) ought to opt out of this. It should be good for all Fedora > editions, though (including Workstation, Server, Atomic, CoreOS), and > also for KDE spin. How will this work on headless systems like Fedora Server, Atomic, and CoreOS? Will it be expected that users manually create their own cgroups?

Earlyoom maintainer here. I think it's too early to switch to systemd-oomd, because it was just merged to the systemd codebase and is still an experimental feature.

In earlyoom we have a list of processes that cannot be killed (eg. dnf/packagekit/etc.), so it is absolutely safe to use as a default userspace OOM solution. We currently don't know anything about the systemd-oomd safety for regular use on end user desktops. We can't even test it on the current stable Fedora release.

That's why I think this change need to be postponed to Fedora 35 (opt-in in F34 and default in F35).

On Tue, Dec 22, 2020 at 6:17 PM Anita Zhang <anitazha(a)fedoraproject.org&gt; wrote: Another variation on this theme: enable by default in Fedora 34 Server edition. And more broadly rolled out for Fedora 35. If it's broadly ready for Fedora 34, great. Otherwise, it seems like a good fit for Fedora Server edition, given sd-oomd's server origin and oomd2 been used in production for a number of years. It'd be a significant headline feature for Server edition, especially fitting for the in-progress reboot of that project. Any thoughts from Server WG folks?

On Mon, Dec 28, 2020 at 07:44:04PM -0000, Tom Seewald wrote: > > On Tue, Dec 22, 2020 at 6:17 PM Anita Zhang > > <anitazha(a)fedoraproject.org&gt; wrote: > > > > > > Another variation on this theme: enable by default in Fedora 34 > > Server > > edition. And more broadly rolled out for Fedora 35. > > > > If it's broadly ready for Fedora 34, great. Otherwise, it seems > > like a > > good fit for Fedora Server edition, given sd-oomd's server origin > > and > > oomd2 been used in production for a number of years. It'd be a > > significant headline feature for Server edition, especially > > fitting > > for the in-progress reboot of that project. Any thoughts from > > Server > > WG folks? > > I don't think enabling systemd-oomd for Fedora server by default > makes a lot of sense right now. Why would we want to automatically > enable systemd-oomd in cases where users either have to manually > manage cgroups or risk a worse experience than what currently > exists with earlyoom? If a user is already creating/managing > cgroups themselves, then manually enabling systemd-oomd would be a > minor extra step. But if the user isn't managing cgroups (which I > believe is the common case), then that user would be pretty > surprised if systemd-oomd wipes out a huge swath of running > programs that happen to be in the same cgroup. > > With Gnome and KDE Plasma most of the cgroup creation is done > automatically, so it makes more sense to enable systemd-oomd by > default as those systems are already set up for systemd-oomd to > work well. I'm a bit unclear on this and perhaps the change owners could comment: This operates on slices right? So in a server context each service is in it's own slice. So, while it might kill say a httpd slice, other services would be fine? That doesn't seem great if you are running a deadicated webserver, but perhaps not so bad if you are running a bunch of different services.

Also, perhaps there's some way to teach the web server applications to put say different wsgi applications in different slices, but no idea if any work has gone into that.

For swap based actions, systemd-oomd will monitor the system-wide swap space and act when available swap falls below the configured threshold, starting with the cgroups with the highest swap usage to the least. Keeping some amount of swap (if enabled) available will prevent the kernel OOM killer from killing processes unpredictably and spending an unbounded amount of time afterwards.

Robbie Harwood <rharwood(a)redhat.com&gt; writes: > Ben Cotton <bcotton(a)redhat.com&gt; writes: > >> For swap based actions, systemd-oomd will monitor the system-wide swap >> space and act when available swap falls below the configured >> threshold, starting with the cgroups with the highest swap usage to >> the least. Keeping some amount of swap (if enabled) available will >> prevent the kernel OOM killer from killing processes unpredictably and >> spending an unbounded amount of time afterwards. > > -1 from me. If the kernel behavior is a problem, fix it - don't kludge > around it in userspace. That is unlikely to happen. As far as I know, the kernel devs see the kernel oom killer as a kernel self protection measure and want it to fire as the last resort (and they are quite hesitant to touch userspace).

I believe you are assuming the consequent when you suggest that kernel developers should be somehow fixing this in userspace. To back up: the described problem is the manifestation of an interaction between swap and the OOM condition. The OOM killer is a popularly-understood piece of what goes on in the system during OOM, but it's not like the rest of the kernel can be ignored. (I would argue that part of the reason it's well understood is their insistence that it remain simple, but that's getting off into the weeds.)

So, several control points here: - OOM killer behavior. I think we're in agreement that this isn't the thing that needs changed.

- Enabling swap. Swap is really slow (by virtue of not being RAM...) and we don't seem willing to acknowledge that. If we want the system to be snappy and responsive... we shouldn't be swapping.

- Swap aggressiveness. Suggested by above, people want swap anyway. (Sometimes it's for hibernation (not supported, but that stops no one), sometimes it's for... historical reasons? Underprovisioning?) This could be tuned to the use cases we actually want.

- Education. Get people to a point where admins don't deploy swap on systems that aren't going to hibernate. I'll readily admit this one might be hardest.

And even possibly the (conceptually) simplest solution of all: - Swap usage monitoring as described for oomd... but in the kernel. This saves you on all the overhead of running in userspace, if nothing else.

But what really bothers me here is that, to my knowledge, no one has tried to actually make any of these happen in the kernel. There's a vague perception of what "the kernel devs" want, as if they're some other, but... has anyone asked? If so, we should be able to quote what the response was, and a good design proposal should include it as an explanation of why that route wasn't taken.