8:37 a.m.
I have 3 machines with clean F37 installs. One of the F37 machines has 4GB of RAM, and I
maintain it as a backup and normally only log in via ssh and do dnf updates via command
line. In the last few weeks this has become extremely difficult to do due to being
automatically logged out, presumably by systemd-oomd. It happens even if I boot in
multiuser, which ought to reduce memory use. From what little I've read and what
experimentation I've done so far, it appears that being logged into a DE (maybe only
GNOME or KDE?) protects against this, but non-DE logins (including ssh), and any commands
running in them, are not protected. This goes against the expectation that non-DE access
should be LESS likely to run out of memory, especially if there isn't even a DE
running. How hard would it be for systemd-oomd to be configured to protect non-DE logins
and anything running in them?
I've also read that configuring non-zram swap might be a cure. As I said, these are
clean F37 installs, and if that's necessary for reasonable behavior when there's
not enough RAM, the installer should be doing it automatically. In my case, I don't
think that's the cause, since the free command suggests that I'm only using a
fraction of both the memory and swap even when the automatic logging out is happening.
9:07 a.m.
On Sun, Mar 12, 2023 at 10:38 AM Andre Robatino <robatino(a)fedoraproject.org>
wrote:
I have 3 machines with clean F37 installs. One of the F37 machines
has 4GB
of RAM, and I maintain it as a backup and normally only log in via ssh and
do dnf updates via command line. In the last few weeks this has become
extremely difficult to do due to being automatically logged out, presumably
by systemd-oomd.
You should look for reasons with journalctl and monitor memory usage in a
terminal (using top, bpytop).
It happens even if I boot in multiuser, which ought to reduce memory
use.
Again. you should be monitoring memory use instead of guessing. It would
not be surprising that some recent
"improvement" increased memory use.
From what little I've read and what experimentation I've done so far, it
appears that being logged into a DE (maybe only GNOME or KDE?)
protects
against this, but non-DE logins (including ssh), and any commands running
in them, are not protected. This goes against the expectation that non-DE
access should be LESS likely to run out of memory, especially if there
isn't even a DE running. How hard would it be for systemd-oomd to be
configured to protect non-DE logins and anything running in them?
I've also read that configuring non-zram swap might be a cure. As I said,
these are clean F37 installs, and if that's necessary for reasonable
behavior when there's not enough RAM, the installer should be doing it
automatically. In my case, I don't think that's the cause, since the free
command suggests that I'm only using a fraction of both the memory and swap
even when the automatic logging out is happening.
Since you are using ssh, you should consider whether a network connection
is getting dropped. My
network monitoring software reports when a network interface goes offline.
Recently there have been
drops of inactive wifi connections with Fedora 37.
I suspect these might be a result of efforts to reduce power consumption.
--
George N. White III
9:24 a.m.
It's not just ssh. It happens even if I boot in multiuser and log in via the console,
so any non-DE login is affected. Sometimes, I can't even run a simple "dnf
check" command (after having a previous transaction aborted by a logout) without
being logged out again. And the free command indicates that I never come close to using
either all the memory or swap.
10:22 a.m.
On Sun, 12 Mar 2023 13:37:46 -0000
"Andre Robatino" <robatino(a)fedoraproject.org> wrote:
I have 3 machines with clean F37 installs. One of the F37 machines
has 4GB of RAM, and I maintain it as a backup and normally only log
in via ssh and do dnf updates via command line. In the last few weeks
this has become extremely difficult to do due to being automatically
logged out, presumably by systemd-oomd. It happens even if I boot in
multiuser, which ought to reduce memory use. From what little I've
read and what experimentation I've done so far, it appears that being
logged into a DE (maybe only GNOME or KDE?) protects against this,
but non-DE logins (including ssh), and any commands running in them,
are not protected. This goes against the expectation that non-DE
access should be LESS likely to run out of memory, especially if
there isn't even a DE running. How hard would it be for systemd-oomd
to be configured to protect non-DE logins and anything running in
them?
I've also read that configuring non-zram swap might be a cure. As I
said, these are clean F37 installs, and if that's necessary for
reasonable behavior when there's not enough RAM, the installer should
be doing it automatically. In my case, I don't think that's the
cause, since the free command suggests that I'm only using a fraction
of both the memory and swap even when the automatic logging out is
happening.
I don't have any problems with any of the things that you do of F37,
and I also initially log in to multiuser. This sounds like there is
some configuration issue on your system causing a problem. Or could
the memory be failing with intermittent faults? Or maybe you have a
setting that create the problems (seems like a longshot if you are
using a default install). If you can find a cause, it would be good to
let the maintainers of systemd-oomd know with a bugzilla.
You could, as root, run
systemctl stop systemd-oomd.service
If there is in fact an OOM condition, your system will hang. But, as
George said, it might be better to run one of the tops in a terminal,
and see what is happening with memory (top, or htop, or itop). Or
dmesg
or
journalctl -r
I took the additional step of running
systemctl mask systemd-oomd.service
so that it never will run. I have never had an issue, though I do have
disk based swap, so when I get close to memory issues (around 90%
usage), I notice the slowdown or hear the disk activating a lot.
10:47 a.m.
Does your machine have 4GB or less of RAM? If you have more, it may be much less likely to
trigger. I just verified that when I log into GNOME on the machine in question, an rsync
of a single large file that never works when done remotely works fine, it only fails when
attempted from a non-DE login (ssh or console). This should be easy to reproduce with a VM
with 1 or 2 GB of RAM (I don't know what the current minimum is). I have a F38 VM that
I normally allocate 2GB and boot in multiuser, just to update. I was starting to notice
the problem in that as well, so at first I increased the RAM to 4GB and didn't notice
it anymore. I don't know why the VM requires less RAM than the F37 bare metal machine.
Anyway, once I suspected that the problem was with the non-DE login, I lowered the
allocation to 2GB and booted in graphical, and logged into GNOME, and the problem was
gone. I don't like doing that, though, since the updating takes longer with all the
nonessential stuff going on in GNOME
(which is why I normally use multiuser). The VM is running on a different host with 16GB
so I doubt this has anything to do with hardware issues.
I did try "systemctl mask systemd-oomd" yesterday, but then discovered that if I
reinstall systemd-oomd-defaults, it starts running again, even though it's still
masked, so that's not reliable. The same thing would probably happen on any systemd
update, unless I just removed systemd-oomd-defaults itself.
10:58 a.m.
BTW, I did notice that the problem was gone during the time that systemd-oomd wasn't
running, so that's definitely the cause. Unfortunately, the mask command alone
isn't enough to prevent it from running, I'd have to either remove
systemd-oomd-defaults or edit some config files. And this really should be fixed, it's
just wrong that anything running in a DE is protected from being killed, while a non-DE
login, or any command run from within that, isn't.
11:11 a.m.
On 12 Mar 2023, at 15:58, Andre Robatino
<robatino(a)fedoraproject.org> wrote:
BTW, I did notice that the problem was gone during the time that systemd-oomd wasn't
running, so that's definitely the cause. Unfortunately, the mask command alone
isn't enough to prevent it from running, I'd have to either remove
systemd-oomd-defaults or edit some config files. And this really should be fixed, it's
just wrong that anything running in a DE is protected from being killed, while a non-DE
login, or any command run from within that, isn't.
It may be dnf that is using up the memory.
Depending on the complexity of the upgrade the dnf memory will go up or down.
I find on my 2GiB RPi i have update packages in groups to avoid the oom caused by dnf.
_______________________________________________
users mailing list -- users(a)lists.fedoraproject.org
To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
11:22 a.m.
I've heard dnf uses a lot of memory, but in my 2GB F38 VM, I can run a large DNF
transaction with no problem when logged into GNOME. On 4GB F37 bare metal, in multiuser,
even updating one letter at a time isn't enough, even "dnf check" can fail.
I read somewhere that GNOME and KDE are the only two DEs that have the necessary
protection, but don't quote me on that. I haven't even read up on what
"cgroup" means yet.
12:43 p.m.
Andre Robatino composed on 2023-03-12 15:58 (UTC):
BTW, I did notice that the problem was gone during the time that
systemd-oomd wasn't running, so that's definitely the cause. Unfortunately, the
mask command alone isn't enough to prevent it from running, I'd have to either
remove systemd-oomd-defaults or edit some config files. And this really should be fixed,
it's just wrong that anything running in a DE is protected from being killed, while a
non-DE login, or any command run from within that, isn't.
Just a thought: I've never found anything to be automatically running after which
I ran multiple systemctl commands in order:
stop > disable > mask
--
Evolution as taught in public schools is, like religion,
based on faith, not based on science.
Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!
Felix Miata
12:14 p.m.
On Sun, 12 Mar 2023 15:47:48 -0000
"Andre Robatino" <robatino(a)fedoraproject.org> wrote:
Does your machine have 4GB or less of RAM? If you have more, it may
be much less likely to trigger. I just verified that when I log into
Yeah, 16 GB.
GNOME on the machine in question, an rsync of a single large file
that never works when done remotely works fine, it only fails when
attempted from a non-DE login (ssh or console). This should be easy
to reproduce with a VM with 1 or 2 GB of RAM (I don't know what the
current minimum is). I have a F38 VM that I normally allocate 2GB and
boot in multiuser, just to update. I was starting to notice the
problem in that as well, so at first I increased the RAM to 4GB and
didn't notice it anymore. I don't know why the VM requires less RAM
than the F37 bare metal machine. Anyway, once I suspected that the
problem was with the non-DE login, I lowered the allocation to 2GB
and booted in graphical, and logged into GNOME, and the problem was
gone. I don't like doing that, though, since the updating takes
longer with all the nonessential stuff going on in GNOME (which is
why I normally use multiuser). The VM is running on a different host
with 16GB so I doubt this has anything to do with hardware issues.
This sounds like a bug, as if systemd-oomd is imposing much stricter
standards when login is sshd or console. I agree with you, how can a
resource intensive task like a desktop succeed when a minimal
implementation fails? I think the current minimum memory is 2GB (I
vaguely recall that I recently read on the test(?) list that people in
some situations are having issues with that amount). Like you, I
notice that dnf is using almost no memory when I do updates from
multiuser. Maybe systemd-oomd is somehow interpreting that
usage incorrectly?
I did try "systemctl mask systemd-oomd" yesterday, but
then
discovered that if I reinstall systemd-oomd-defaults, it starts
running again, even though it's still masked, so that's not reliable.
The same thing would probably happen on any systemd update, unless I
just removed systemd-oomd-defaults itself.
That hasn't happened to me. When I check with
systemctl -a -t service
it is still masked, and the status of systemd-oomd is failed, dead. And
that is after updates to systemd. But I don't have the package
systemd-oomd-defaults installed. Reinstalling that seems like it might
be a reset of the mask to the default. Or maybe, if that is the
package that actually sets up OOMD, I don't even have OOMD installed.
That would explain why my experience is different.
12:33 p.m.
I just tried removing systemd-oomd-defaults and it's still possible to run
systemd-oomd so I was wrong in thinking that would prevent it.
1:07 p.m.
Hi.
On Sun, 12 Mar 2023 17:33:42 +0000 "Andre Robatino" wrote:
I just tried removing systemd-oomd-defaults and it's still
possible to run
systemd-oomd so I was wrong in thinking that would prevent it.
Right: systemd-oomd-defaults only configures systemd-oomd.
rpm -ql --scripts systemd-oomd-defaults
/usr/lib/systemd/oomd.conf.d
/usr/lib/systemd/oomd.conf.d/10-oomd-defaults.conf
/usr/lib/systemd/system/system.slice.d/10-oomd-per-slice-defaults.conf
/usr/lib/systemd/system/user-.slice.d/10-oomd-per-slice-defaults.conf
/usr/lib/systemd/user/slice.d/10-oomd-per-slice-defaults.conf
I checked on a small VM.
With it installed, oomctl shows some CGroups monitored
root# oomctl
Dry Run: no
Swap Used Limit: 90.00%
Default Memory Pressure Limit: 60.00%
Default Memory Pressure Duration: 30s
System Context:
Memory: Used: 349.5M Total: 1.9G
Swap: Used: 0B Total: 0B
Swap Monitored CGroups:
Memory Pressure Monitored CGroups:
Path: /user.slice/user-0.slice
Memory Pressure Limit: 50.00%
<snip>
Path: /user.slice/user-0.slice/user(a)0.service
<snip>
root#
With it removes. and after a reboot (needed), no CGroups monitored
root# oomctl
Dry Run: no
Swap Used Limit: 90.00%
Default Memory Pressure Limit: 60.00%
Default Memory Pressure Duration: 30s
System Context:
Memory: Used: 457.5M Total: 1.9G
Swap: Used: 0B Total: 0B
Swap Monitored CGroups:
Memory Pressure Monitored CGroups:
root#
--
francis
1:45 p.m.
Thanks. I confirmed on the affected machine that immediately after removing
systemd-oomd-defaults, it was still monitoring the same CGroups, but after rebooting, it
was monitoring none, even though systemd-oomd was still enabled and running.
12:31 p.m.
On Sun, Mar 12, 2023 at 9:38 AM Andre Robatino
<robatino(a)fedoraproject.org> wrote:
I have 3 machines with clean F37 installs. One of the F37 machines has 4GB of RAM, and I
maintain it as a backup and normally only log in via ssh and do dnf updates via command
line. In the last few weeks this has become extremely difficult to do due to being
automatically logged out, presumably by systemd-oomd.
[...]
DNF is known to have excessive memory requirements lately. It started
around Fedora 35.
* https://bugzilla.redhat.com/show_bug.cgi?id=1907030
* https://www.reddit.com/r/Fedora/comments/isfw07/excessive_ram_usage_durin...
*
https://discussion.fedoraproject.org/t/dnf-operations-use-large-amount-of...
* etc.
Jeff
12:45 p.m.
I've heard that, but for me, with limited RAM, both DNF transactions and an rsync of a
very large file fail in multiuser (by causing logout while they're running) while they
both succeed in GNOME on the same machine.
4:28 a.m.
On Sun, 12 Mar 2023 13:37:46 -0000, Andre Robatino wrote:
I have 3 machines with clean F37 installs.
Dunno yet whether it's also F37 or just F38, but here on F38 even
gnome-terminal is getting killed when running a simple tar command in it
that works on creating a ~20 GB archive.
5:05 a.m.
I filed https://bugzilla.redhat.com/show_bug.cgi?id=2177722 for my original problem
(involving non-DE logins) and it was indeed fixed with the latest systemd updates in both
F37 and F38. I know other people have reported it happening in GNOME but have never seen
it personally. There are closed bugs and some open bugs for what you're describing,
for example
https://bugzilla.redhat.com/show_bug.cgi?id=2162708
https://bugzilla.redhat.com/show_bug.cgi?id=2173394
so check Bugzilla.
368
days inactive
416
days old
16 comments
8 participants
participants (8)
-
Andre Robatino -
Barry -
Felix Miata -
Francis.Montagnac＠inria.fr -
George N. White III -
Jeffrey Walton -
Michael Schwendt -
stan