8:53 a.m.
Hi all,
Let me introduce the symptoms that triggered this investigation.
First, versions:
- OS: Ubuntu 20.04
- SSSD: 2.2.3-3ubuntu0.7
- Platform: x86_64
Some weeks after starting using a new region/provider we noticed some intermittent issue
impacting SSH logins (ie: clients got `Connection closed by UNKNOWN port 65535` on `ssh
...`). When looking into it, I was able to reproduce it somewhat reliably. It happens
after some time of inactivity for that user, but if I constantly issue commands (ie:
`watch sssctl user-checks <user>` or `watch ssh <host> whoami` from a client)
the issue wont trigger. If I stopped trying out those commands and tried again ~30 minutes
later, it would fail.
With these symptoms I had these possibilities in mind: some caching issue, some
authentication expiring, connections pools issues...
I enabled detailed logging and compared logs from a successful request and a failed one
and I found what seems to be the source of the error from sssd logs perspective, but still
can't make much sense of it.
Log extracted from sssd_<domain>.log:
```
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [ipa_hbac_rule_info_next] (0x0400):
Sending request for next search base: [cn=hbac,dc=ipa,dc=corp,<dc
<domain>>][2][(&(objectclass=ipaHBACRule)(ipaenabledflag=TRUE)(accessRuleType=allow)(|(hostCategory=all)(...)))]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_print_server] (0x2000): Searching
<ip10.1.10.102:389
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with
[(&(objectclass=ipaHBACRule)(ipaenabledflag=TRUE)(accessRuleType=allow)(|(hostCategory=all)(...)][cn=hbac,dc=ipa,dc=corp,<dc
<domain>>].
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [objectclass]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [cn]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [ipauniqueid]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [ipaenabledflag]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [accessRuleType]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [memberUser]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [userCategory]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [memberService]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [serviceCategory]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [sourceHost]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [sourceHostCategory]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [externalHost]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [memberHost]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000):
Requesting attrs: [hostCategory]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x2000):
ldap_search_ext called, msgid = 18
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_op_add] (0x2000): New operation 18
timeout 60
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_process_result] (0x2000): Trace:
sh[0x55d349f96d00], connected[1], ops[0x55d349ff16e0], ldap[0x55d34a10cba0]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_process_result] (0x2000): Trace: end
of ldap_result list
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [sdap_op_timeout] (0x1000): Issuing
timeout for 18
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [sdap_op_destructor] (0x1000): Abandoning
operation 18
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [generic_ext_search_handler] (0x0040):
sdap_get_generic_ext_recv failed [110]: Connection timed out
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [ipa_hbac_rule_info_done] (0x0080): Could
not retrieve HBAC rules
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [ipa_pam_access_handler_done] (0x0020):
Unable to fetch HBAC rules [110]: Connection timed out
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [dp_req_done] (0x0400): DP Request [PAM
Account #21]: Request handler finished [0]: Success
```
I tried using `ldap_enumeration_search_timeout = 180` which updated the timeout in
(`[sdap_op_add] (0x2000): New operation 18 timeout 60 `) but same result (`enumerate` and
`subdomain_enumerate` config options are left to their defaults, `false` and `none`
respectively).
I looked at the IPA servers LDAP logs and saw now related requests regarding
`ipaHBACRules` so I assume the request didn't even left the server.
I tried tweaking some other caching options like `cached_auth_timeout` and
`pam_id_timeout` without any impact. Also tried clearing up the caches with `sss_cache -E`
or `systemctl stop sssd && rm /var/lib/sss/db/* && systemctl start sssd`
and nothing changed.
This is the error returned by `sssctl user-checks` when it fails:
```
$ sudo sssctl user-checks <user>
user: <user>
action: acct
service: system-auth
SSSD nss user lookup result:
- user name: <user>
- user id: <uid>
- group id: <gid>
- gecos: <name>
- home directory: /home/<user>
- shell: /bin/bash
InfoPipe operation failed. Check that SSSD is running and the InfoPipe responder is
enabled. Make sure 'ifp' is listed in the 'services' option in
sssd.conf.InfoPipe User lookup with [<user>] failed.
testing pam_acct_mgmt
pam_acct_mgmt: System error
PAM Environment:
- no env -
```
Any pointer here would be very useful.
Thank you!
Aitor