4:41 a.m.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Bug ID: 1853261
Summary: sssd fails to start on read-only filesystem
Product: Fedora
Version: rawhide
Status: NEW
Component: sssd
Assignee: sssd-maintainers(a)lists.fedoraproject.org
Reporter: zbyszek(a)in.waw.pl
QA Contact: extras-qa(a)fedoraproject.org
CC: abokovoy(a)redhat.com, atikhono(a)redhat.com,
jhrozek(a)redhat.com, lslebodn(a)redhat.com,
mzidek(a)redhat.com, pbrezina(a)redhat.com,
rharwood(a)redhat.com, sbose(a)redhat.com,
ssorce(a)redhat.com,
sssd-maintainers(a)lists.fedoraproject.org
Target Milestone: ---
Classification: Fedora
Description of problem:
sssd refuses to start when /var/log is read-only. It is obviously a problem
when trying to recover
from a file system issue. Since logging is non-essential, sssd should just
continue. At most is
should emit a warning.
$ touch /var/tmp/foo
touch: cannot touch '/var/tmp/foo': Read-only file system
$ sudo systemctl start sssd.service
Job for sssd.service failed because the control process exited with error code.
See "systemctl status sssd.service" and "journalctl -xe" for details.
[fedora@workstation-uefi ~]$
Broadcast message from systemd-journald@workstation-uefi (Thu 2020-07-02
11:16:46 CEST):
sssd[900]: Could not open file [/var/log/sssd/sssd.log]. Error: [30][Read-only
file system]
Broadcast message from systemd-journald@workstation-uefi (Thu 2020-07-02
11:16:47 CEST):
sssd[903]: Could not open file [/var/log/sssd/sssd.log]. Error: [30][Read-only
file system]
Broadcast message from systemd-journald@workstation-uefi (Thu 2020-07-02
11:16:47 CEST):
sssd[905]: Could not open file [/var/log/sssd/sssd.log]. Error: [30][Read-only
file system]
Broadcast message from systemd-journald@workstation-uefi (Thu 2020-07-02
11:16:47 CEST):
sssd[906]: Could not open file [/var/log/sssd/sssd.log]. Error: [30][Read-only
file system]
Broadcast message from systemd-journald@workstation-uefi (Thu 2020-07-02
11:16:47 CEST):
sssd[907]: Could not open file [/var/log/sssd/sssd.log]. Error: [30][Read-only
file system]
We see multiple issues here:
- the main one is that sssd should not fail to start as described above
- but also, why is sssd not just logging to the journal? Why is it spamming
with broadcast messages?
If sssd would just log to the journal like any modern service, all those
problems
would be avoided.
Version-Release number of selected component (if applicable):
sssd-2.3.0-2.fc33.x86_64
How reproducible:
Deterministic
Steps to Reproduce:
1. Make /var/log read-only, either by not remounting the filesystems rw, or by
using udev.blockdev-read-only with systemd-246+
2. Try to start sssd
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
5:03 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Pavel Březina <pbrezina(a)redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Assignee|sssd-maintainers(a)lists.fedo |ppolawsk(a)redhat.com
|raproject.org |
Doc Type|--- |If docs needed, set a value
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
7:10 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #1 from Lukas Slebodnik <lslebodn(a)redhat.com> ---
(In reply to Zbigniew Jędrzejewski-Szmek from comment #0)
We see multiple issues here:
- the main one is that sssd should not fail to start as described above
I would say you should use root to recover from a file system issue
and root user is not handled by sssd
- but also, why is sssd not just logging to the journal? Why is it
spamming
with broadcast messages?
By default it try to log to files in /var/log/sssd. Obviously it failed due to
read-only file system.
that's the reason of broadcast emssage.
If sssd would just log to the journal like any modern service, all
those
problems would be avoided.
sssd can log to journald you just need to explicitely configure it.
sh$ systemctl cat sssd | head
# /usr/lib/systemd/system/sssd.service
[Unit]
Description=System Security Services Daemon
# SSSD must be running before we permit user sessions
Before=systemd-user-sessions.service nss-user-lookup.target
Wants=nss-user-lookup.target
[Service]
Environment=DEBUG_LOGGER=--logger=files
EnvironmentFile=-/etc/sysconfig/sssd
Just set DEBUG_LOGGER in /etc/sysconfig/sssd (details in `man sssd`)
But I doubt it will help if system is read-only.
sssd will still try to open rw files in /var/lib/sss/
--
You are receiving this mail because:
You are on the CC list for the bug.
7:23 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #2 from Alexey Tikhonov <atikhono(a)redhat.com> ---
Hi,
1) "basic" logs by default are written to files (/var/log/sssd) and one can
control this via "--logger" option (including option to redirect to journald or
stderr completely)
2) "important" messages are also (additionally) logged to journald
What you see as "Broadcast message from systemd-journald@workstation-uefi" is
broadcast from journald, not directly from SSSD.
You can control those broadcast messages via journald settings (namely
`ForwardToWall` and `MaxLevelWall`)
So the questions here are:
1) Does this message have appropriate log level (*perhaps* emergency level of
this specific message should be decreased)?
2) as you wrote, perhaps SSSD should fallback to "/dev/null" for
"basic" logs
output in case of fail to open files...
--
You are receiving this mail because:
You are on the CC list for the bug.
8:25 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #3 from Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl> ---
The goal of my bug report is to nudge Fedora toward a system which is resilient
to failure.
In particular this is motivated by making it easier for users to recover from
disk corruption
and other file system issues. Users are confused and unhappy when they boot to
a tty prompt.
I'm going through all services which reasonably should run fine with readonly
fs. Obviously,
in really bad cases, logging in at the tty will be necessary. But read-only
root is not one
of them.
There is really no reason to log to a private log file nowadays. Please just
log to the
journal by default. This is much nicer for users because they expect
'journalctl' to give
them system logs, and not to have to hunt for some obscure log file. This also
makes this
*much* more robust. In particular, journald is completely fine with read-only
root.
Log files are stored in /run, and if /var/log becomes available, logs are
flushed there.
Logs should "just work" from user's POV.
--
You are receiving this mail because:
You are on the CC list for the bug.
8:39 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #4 from Pavel Březina <pbrezina(a)redhat.com> ---
Well, I don't think users would be happy given the amount of messages sssd
produce. But I agree that being unable to produce logs should not be considered
fatal.
--
You are receiving this mail because:
You are on the CC list for the bug.
8:59 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #5 from Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl> ---
(In reply to Pavel Březina from comment #4)
Well, I don't think users would be happy given the amount of
messages sssd produce.
I didn't consider this... On my machine sssd seems to be very quiet, but that's
probably because I have just one local user. Users hate "chatty" services with
a
passion, so if sssd were to log the journal, it indeed shouldn't log too much
(by default).
But I agree that being unable to produce logs should not be
considered fatal.
--
You are receiving this mail because:
You are on the CC list for the bug.
3:29 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #6 from Pavel Březina <pbrezina(a)redhat.com> ---
Since you have only one local users you use sssd only marginally.
We use journald (well syslog) for the most important error messages (like
something that requires admin attention), but the rest goes to files when
logging is enabled (unless --logger journald).
--
You are receiving this mail because:
You are on the CC list for the bug.
11:53 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Michael Romero <romerom(a)gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |romerom(a)gmail.com
--- Comment #8 from Michael Romero <romerom(a)gmail.com> ---
Any change this is going to be addressed? I'm trying to get us completely away
from an older LDAP environment, but whenever a filesystem goes read-only, the
only account we can get in to investigate with is either the "root" user via
console, or a legacy LDAP user via "SSH". While these r/o FS issues don't
happen that often, when they do - it would be GREAT if authentication could
continue to function and not require logging in as root over console to
resolve.
--
You are receiving this mail because:
You are on the CC list for the bug.
1:56 p.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #9 from Alexey Tikhonov <atikhono(a)redhat.com> ---
(In reply to Michael Romero from comment #8)
Any change this is going to be addressed? I'm trying to get us
completely
away from an older LDAP environment, but whenever a filesystem goes
read-only, the only account we can get in to investigate with is either the
"root" user via console, or a legacy LDAP user via "SSH". While
these r/o
FS issues don't happen that often, when they do - it would be GREAT if
authentication could continue to function and not require logging in as root
over console to resolve.
Solving this for logs won't help: sssd need writable /var/lib/sss/ anyway.
--
You are receiving this mail because:
You are on the CC list for the bug.
8:31 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Paweł Poławski <ppolawsk(a)redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |CLOSED
Resolution|--- |WONTFIX
Last Closed| |2021-04-08 13:31:48
--- Comment #10 from Paweł Poławski <ppolawsk(a)redhat.com> ---
After some discussion it looks like there is no point of SSSD working on
read-only filesystem.
User can already redirect logs to journal using existing SSSD options.
The problem is our databases located at /var/lib/sss/db/* still needs to be RW
for SSSD to work.
--
You are receiving this mail because:
You are on the CC list for the bug.
3:38 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|CLOSED |NEW
Resolution|WONTFIX |---
Keywords| |Reopened
--- Comment #11 from Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl> ---
Sorry, but that is not an acceptable answer.
After some discussion it looks like there is no point of SSSD working
on read-only filesystem.
Let me quote the previous previous message:
but whenever a filesystem goes read-only, the only account we can get
in to investigate with is either the "root" user via console, or a legacy LDAP
user via "SSH".
If sssd was an opt-in thing, throwing up your hands and saying this is too hard
to support would fine.
But it is a default component in Fedora. This means that it also needs to solve
the hard edge cases.
The default authentication mechanism simply cannot stop working in a scenario
which is a fairly
common failure mode.
--
You are receiving this mail because:
You are on the CC list for the bug.
4:29 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Paweł Poławski <ppolawsk(a)redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Keywords| |Triaged
--
You are receiving this mail because:
You are on the CC list for the bug.
6:49 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Paweł Poławski <ppolawsk(a)redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED
--
You are receiving this mail because:
You are on the CC list for the bug.
8:44 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #12 from Ben Cotton <bcotton(a)redhat.com> ---
This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with
a
Fedora 'version' of '33'.
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.
Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 33 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
--
You are receiving this mail because:
You are on the CC list for the bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
9:14 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #13 from Ben Cotton <bcotton(a)redhat.com> ---
This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with
a
Fedora 'version' of '33'.
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.
Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 33 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
--
You are receiving this mail because:
You are on the CC list for the bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
10:12 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #14 from Ben Cotton <bcotton(a)redhat.com> ---
This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with
a
Fedora 'version' of '33'.
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.
Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 33 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
--
You are receiving this mail because:
You are on the CC list for the bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
9:13 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Paweł Poławski <ppolawsk(a)redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Assignee|ppolawsk(a)redhat.com |sssd-maint(a)redhat.com
--
You are receiving this mail because:
You are on the CC list for the bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
10:08 a.m.
New subject: [Bug 1853261] sssd fails to start on read-only filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Ben Cotton <bcotton(a)redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |CLOSED
Resolution|--- |EOL
Last Closed|2021-04-08 13:31:48 |2021-11-30 16:08:05
--- Comment #15 from Ben Cotton <bcotton(a)redhat.com> ---
Fedora 33 changed to end-of-life (EOL) status on 2021-11-30. Fedora 33 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.
If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.
Thank you for reporting this bug and we are sorry it could not be fixed.
--
You are receiving this mail because:
You are on the CC list for the bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
872
days inactive
1388
days old
sssd-maintainers@lists.fedoraproject.org
18 comments
1 participants
participants (1)
-
bugzilla＠redhat.com