https://bugzilla.redhat.com/show_bug.cgi?id=1995608
Bug ID: 1995608 Summary: sssd logging verbose by default results in huge log files Product: Fedora Version: 34 Hardware: x86_64 OS: Linux Status: NEW Component: sssd Severity: urgent Assignee: sssd-maintainers@lists.fedoraproject.org Reporter: brian@interlinx.bc.ca QA Contact: extras-qa@fedoraproject.org CC: abokovoy@redhat.com, atikhono@redhat.com, jhrozek@redhat.com, lslebodn@redhat.com, luk.claes@gmail.com, mzidek@redhat.com, pbrezina@redhat.com, sbose@redhat.com, ssorce@redhat.com, sssd-maintainers@lists.fedoraproject.org Target Milestone: --- Classification: Fedora
Created attachment 1815664 --> https://bugzilla.redhat.com/attachment.cgi?id=1815664&action=edit Example log spew
Description of problem: My /var/log/sssd/sssd_$domain.log grows huge.
Version-Release number of selected component (if applicable): sssd-2.5.2-2.fc34.x86_64
How reproducible: 100%
Steps to Reproduce: Should pretty obvious.
Actual results: Log grows large with useless debug info.
Expected results: Log should not grow as quick and large as it does.
Additional info: Every minute a spew similar to the attachment is added to the log.
I can squelch this log spam with:
# sssctl debug-level 0
Neither the words debug or log even appear in my configuration:
# grep -ri -e debug -e log /etc/sssd/ [nothing]
so this is appearing to be a rather verbose default setting.
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
Alexey Tikhonov atikhono@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Assignee|sssd-maintainers@lists.fedo |atikhono@redhat.com |raproject.org | Doc Type|--- |If docs needed, set a value
--- Comment #1 from Alexey Tikhonov atikhono@redhat.com --- Hi,
thanks for reporting this.
Did you cut off the lines like: ``` "********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:\n"; "********************** BACKTRACE DUMP ENDS HERE *********************************\n\n"; ``` from the log or are those missing?
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
--- Comment #2 from Brian J. Murrell brian@interlinx.bc.ca --- Ah, yes. Those lines evaded the grep used to try to pull out just one example of the frequently repeated pattern. So yes, those lines do appear in the log, just not the attached example. Apologies.
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
Alexey Tikhonov atikhono@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Triaged Status|NEW |ASSIGNED
--- Comment #3 from Alexey Tikhonov atikhono@redhat.com --- (In reply to Brian J. Murrell from comment #2)
Ah, yes. Those lines evaded the grep used to try to pull out just one example of the frequently repeated pattern. So yes, those lines do appear in the log, just not the attached example.
Ok, thanks for the confirmation.
As an immediate remediation you can reduce log level in `sssd.conf::[$domain]` to 0
As a mid term solution I will check if it makes sense to change debug level of this message.
A proper solution (already discussed internally) is to avoid printing duplicating backtraces.
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
--- Comment #4 from Brian J. Murrell brian@interlinx.bc.ca --- So this confirms that the default log/debug level is not 0? What is it if I may ask?
Is the situation that is causing the messages to be added to my log at the default log level something I should address? I.e. is the problem actually something other than an inappropriate log level and I could make the messages stop by fixing something?
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
--- Comment #5 from Alexey Tikhonov atikhono@redhat.com --- (In reply to Brian J. Murrell from comment #4)
So this confirms that the default log/debug level is not 0? What is it if I may ask?
It's 2: https://github.com/SSSD/sssd/blob/26654d3e5f5882dd1681116cb49228d108351d48/s...
But every error message with level up to 2 triggers a dump of a full log (all levels) to a certain depth: https://github.com/SSSD/sssd/pull/5585 This backtrace can be disabled via `debug_backtrace_enabled = false` (i.e. you can keep default general log level, disable backtrace and then you will get only single error line "(0x0020): ... Cannot open" without backtraces)
Is the situation that is causing the messages to be added to my log at the default log level something I should address? I.e. is the problem actually something other than an inappropriate log level and I could make the messages stop by fixing something?
That's what I mean in "As a mid term solution I will check if it makes sense to change debug level of this message."
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
--- Comment #6 from Brian J. Murrell brian@interlinx.bc.ca --- Is the entire spew of messages in the attachment all a result of the:
* (2021-08-19 9:29:01): [be[example.com]] [remove_tree_with_ctx] (0x0020): [RID#609] Cannot open /var/lib/sss/deskprofile/example.com/brian: [2]: No such file or directory
message? That missing directory seems to be related to Fleet Commander. Is that correct? If so, given that that is a completely optional management tool, it seems that this missing directory really ought not cause such a spew at the default logging level, which I suppose refers to your mid-term solution.
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
--- Comment #7 from Alexey Tikhonov atikhono@redhat.com --- (In reply to Brian J. Murrell from comment #6)
Is the entire spew of messages in the attachment all a result of the:
- (2021-08-19 9:29:01): [be[example.com]] [remove_tree_with_ctx]
(0x0020): [RID#609] Cannot open /var/lib/sss/deskprofile/example.com/brian: [2]: No such file or directory
message?
Right, 0x0020 message itself is logged because default log levels allows for it and then everything between "PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE" and "BACKTRACE DUMP ENDS HERE" is logged because `debug_backtrace_enabled` allows for it.
This is very new feature, there are some flaws. Sorry for inconvenience.
That missing directory seems to be related to Fleet Commander. Is that correct?
Correct.
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
--- Comment #8 from Brian J. Murrell brian@interlinx.bc.ca --- Yeah. I have a full understanding now. I appreciate your patience with my questions.
So indeed, the root issue here is if the debug level of that:
* (2021-08-19 9:29:01): [be[example.com]] [remove_tree_with_ctx] (0x0020): [RID#609] Cannot open /var/lib/sss/deskprofile/example.com/brian: [2]: No such file or directory
being at the default debug level of 0x20 is appropriate. Given that it's benign and only a result of not using an optional feature of sssd, my suggestion would be that it's not appropriate at 0x20 and the description of 0x20:
0x0020: Critical failures. An error that doesn't kill the SSSD, but one that indicates that at least one major feature is not going to work properly.
seems to bear that out. I'm not sure anything less than 0x0100 is even appropriate for such a benign message. But I will leave figuring out the right level to you folks, just so long as it's not at the default level any more.
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
Alexey Tikhonov atikhono@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Whiteboard| |sync-to-jira review
--- Comment #9 from Alexey Tikhonov atikhono@redhat.com --- Upstream PR: https://github.com/SSSD/sssd/pull/5758
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
Pavel Březina pbrezina@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST
--- Comment #10 from Pavel Březina pbrezina@redhat.com --- Pushed PR: https://github.com/SSSD/sssd/pull/5758
* `master` * bd2ccbf69af8e5836f6f6e09a893d54428d903c5 - file utils: reduce log level in remove_tree_with_ctx() Users of this function are responsible to decide if fail is critical.
https://bugzilla.redhat.com/show_bug.cgi?id=1995608
Alexey Tikhonov atikhono@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2022-03-14 16:51:46
--- Comment #11 from Alexey Tikhonov atikhono@redhat.com --- Fixed since sssd-2.6.0
sssd-maintainers@lists.fedoraproject.org
-
bugzilla＠redhat.com