sssd-users October 2021

sssd-users@lists.fedorahosted.org

13 participants
14 discussions

https://bugzilla.redhat.com/show_bug.cgi?id=1984591 not understanding the nature of the sssd bug introduced recently…
by Spike White 11 Oct '21

11 Oct '21

All (but particularly Sumit since he wrote the comments on https://bugzilla.redhat.com/show_bug.cgi?id=1984591), There are at least two problems created by this recently-introduced sssd bug. One problem is solvable by the suggested work-around, the other is not. The work-around suggested is: [domain/name.of.joined.domain] ad_enabled_domains = dom1.example.com, dom2.example.com, dom3.example.com In order to query only the desired AD domains. What is the bug? the sssd-ad man page says "The AD provider can be used to get user information and authenticate users from trusted domains. Currently only trusted domains in the same forest are recognized.". What is happening is that untrusted AD domains are being discovered. A very specific type of untrusted domains. When the joined domain has no trust with that other domain, but that other domain trusts the original domain – that is a one-way trust (the wrong way). To the joined domain, this is an untrusted domain and should not be discovered. This is actually very common in corporate environments. You may have a main AD domain, call it CORP.COMPANY.COM. Then for testing and new production evaluation, you might have a test AD domain called LAB-TEST.COMPANY.COM. CORP.COMPANY.COM is tightly controlled, with full audits and corporate security. LAB-TEST.COMPANY.COM is a test AD domain – it’s the wild, wild west! So LAB-TEST.COMPANY.COM trusts the main AD domain (in order that users can log into this test domain with their CORP accounts). But CORP.COMPANY.COM does not trust LAB-TEST.COMPANY.COM – nor should it!! (That’s the wild, wild west, doing so would compromise corporate security.) Thus, a server joined to domain CORP.COMPANY.COM should discover CORP.COMPANY.COM and any domains trusted by CORP.COMPANY.COM. It should *NOT* discover LAB-TEST.COMPANY.COM, as CORP.COMPANY.COM does not trust this domain. A server joined to LAB-TEST.COMPANY.COM should discover LAB-TEST.COMPANY.COM and all domains trusted by LAB-TEST.COMPANY.COM. Including CORP.COMPANY.COM, as LAB-TEST.COMPANY.COM trusts CORP.COMPANY.COM. The bug is that a server joined to CORP.COMPANY.COM discovers LAB-TEST.COMPANY.COM, which it shouldn’t. What problems does this cause? Two problems. 1. Many of these untrusted discovered “lab” domains are accessible only to specific network locations. That is, they’re firewalled off to a particular lab. So sssd attempts to query these inaccessible AD domains and takes a long time to time out. This problem can be worked around by the suggested work-around in the Bugzilla: [domain/corp.company.com] ad_enabled_domains = corp.company.com So then, while LAB-TEST.COMPANY.COM is still erroneously discovered, it is no longer searched. Sssd is again fast. 2. Bogus messages in /var/log/sssd_nss.log file. Even with no debug level set in the [nss] stanza, these error messages appear multiple times a second. It quickly fills up the /var/log filesystem. [root@auspdfdlobv01 sssd]# cat sssd_nss.log |grep "The Data Provider returned an error" (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] From debug level 9, it is clear that this is arising from a query of these erroneously-discovered untrusted domains. Here’s an example of one instance of above with debug level 9 turned on. So emeaicmd.geodll.company.com is one of these erroneously-discovered untrusted lab domains, that happens to be firewalled off from this particular AD client: (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 0 errno: 0 error message: Success (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Looking up [oracle(a)company.com] in cache (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Object [oracle(a)company.com] was not found in cache (2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #9: Adding [oracle(a)company.com] to negative cache (2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle(a)company.com is a local user (2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache (2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000): Request type POSIX-only for domain EMEAICMD.geodll.company.com type POSIX is valid (2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #9: Using domain [EMEAICMD.geodll.company.com] (2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR #9: Preparing input data for domain [EMEAICMD.geodll.company.com] rules (2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #9: Looking up oracle(a)emeaicmd.geodll.company.com (2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: Checking negative cache for [oracle(a)emeaicmd.geodll.company.com] (2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com] (2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: [ oracle(a)emeaicmd.geodll.company.com] is not present in negative cache (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Looking up [oracle(a)emeaicmd.geodll.company.com] in cache (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Object [oracle(a)emeaicmd.geodll.company.com] was not found in cache (2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #9: Looking up [oracle(a)emeaicmd.geodll.company.com] in data provider (2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com] (2021-10-07 9:50:02): [nss] [sss_dp_get_account_msg] (0x0400): Creating request for [EMEAICMD.geodll.company.com ][0x3][BE_REQ_INITGROUPS][name=oracle@emeaicmd.geodll.company.com:-] (2021-10-07 9:50:02): [nss] [sbus_add_timeout] (0x2000): 0x564d6ccd6670 (2021-10-07 9:50:02): [nss] [sss_dp_internal_get_send] (0x0400): Entering request [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com] (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Looking up [oracle(a)company.com] in cache (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Object [oracle(a)company.com] was not found in cache (2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #12: Adding [oracle(a)company.com] to negative cache (2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle(a)company.com is a local user (2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache (2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000): Request type POSIX-only for domain EMEAICMD.geodll.company.com type POSIX is valid (2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #12: Using domain [EMEAICMD.geodll.company.com] (2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR #12: Preparing input data for domain [EMEAICMD.geodll.company.com] rules (2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #12: Looking up oracle(a)emeaicmd.geodll.company.com (2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12: Checking negative cache for [oracle(a)emeaicmd.geodll.company.com] (2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com] (2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12: [ oracle(a)emeaicmd.geodll.company.com] is not present in negative cache (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Looking up [oracle(a)emeaicmd.geodll.company.com] in cache (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Object [oracle(a)emeaicmd.geodll.company.com] was not found in cache (2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #12: Looking up [oracle(a)emeaicmd.geodll.company.com] in data provider (2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com] (2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Identical request in progress: [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com] (2021-10-07 9:50:02): [nss] [sss_dp_req_destructor] (0x0400): Deleting request: [0x564d6be36a70:3:oracle@company.com@company.com] (2021-10-07 9:50:02): [nss] [sbus_remove_timeout] (0x2000): 0x564d6ccd6670 (2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): dbus conn: 0x564d6ccc9300 (2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): Dispatching. (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed] The suggested work-around does not resolve problem #2. BTW, here is a listing of the domains discovered on that sssd client: [root@auspdfdlobv01 ~]# sssctl domain-list amer.company.com company.com japn.company.com emea.company.com apac.company.com EMEAICMD.geodll.company.com geodll.company.com EMEAICM.GEODLL.COMPANY.COM alienware.com corp.svcs perotsystems.net companyservices.dmz Beer.Town production.online.company.com jp-poclab.companypoc.com emea-poclab.companypoc.com oldev.preol.company.com olqa.preol.company.com ap-poclab.companypoc.com [root@auspdfdlobv01 ~]# This sssd client is joined to amer.company.com, so the only trusted domains are the first 5. The parent domain and the 4 regional domains. All those other domains below that are untrusted domains. More specifically, they trust company.com, but company.com does not trust them. (one way trust – the wrong way.) Some look like the real wild wild west (Beer.Town ?). Spike

2 6

sss_ssh_authorizedkeys returning "Error looking up public keys"
by Philip Colmer 10 Oct '21

10 Oct '21

We are using Ubuntu 18.04 with sssd 1.16.1-1ubuntu1.8. sssd is being used to allow users to sign in with SSH keys matched against keys stored in OpenLDAP. This has been used for several years but, just recently, a couple of users were reporting that they could not SSH to the server. Further investigation showed that running "sss_ssh_authorizedkeys" for these specific users returned "Error looking up public keys". The tool continues to work correctly for other users. I cannot find any information that might suggest why the tool is not managing to get the keys for these users. I have tried building the code from the GitHub source but it errors out with this: In file included from ./src/util/sss_pam_data.h:33:0, from src/util/sss_pam_data.c:27: ./src/util/debug.h:92:44: error: unknown type name ‘uid_t’; did you mean ‘__id_t’? int chown_debug_file(const char *filename, uid_t uid, gid_t gid); ^~~~~ __id_t ./src/util/debug.h:92:55: error: unknown type name ‘gid_t’; did you mean ‘__id_t’? int chown_debug_file(const char *filename, uid_t uid, gid_t gid); ^~~~~ __id_t Makefile:22359: recipe for target 'src/util/libsss_util_la-sss_pam_data.lo' failed Can anyone suggest how I can troubleshoot this tool in order to fix the error or how I can get it to build on Ubuntu? Thanks. Philip

2 2

Trouble-shooting sssd’s ‘Automatic Kerberos Host Keytab Renewal’ with AD back-end….
by Spike White 09 Oct '21

09 Oct '21

Sssd experts, *Short summary: * How can we troubleshoot sssd’s ‘Automatic Kerberos Host Keytab Renewal’ process? We have ~0.4% of our Linux servers dropping off the AD domain monthly. *Longer explanation:* Over the past two years, we have on-boarded sssd as our Linux AD integration component. Largely displacing a former commercial product that did the same. We have about ~20K Linux servers that are sssd-enabled. A mix of RHEL6, RHEL7, RHEL8, Oracle Linux 6, 7 and 8. We have ~7K Linux servers still on the old commercial product. (For certain edge-case scenarios, such as DMZs, the commercial product works better.) Our AD forest is a single AD forest, with 4 regional child domains. All with transitive trust. Sssd auto-discovers parent domain and all 4 child domains, no problem – whenever it’s adcli joined to its regional local domain. Why are I writing this? Because we are researching an ongoing problem reported by L1 server ops. About 70 – 80 sssd-enabled Linux servers / month drop off the domain. Out of our current sssd-enabled population of ~20K server, that’s not horrible. But still it should be better. (Our former commercial product did better.) It’s not limited to one particular OS, OS version, build location or region. We have surveyed; it seems to occur randomly among all OS versions, regions and locations. To be clear, it’s extremely likely that this behavior arising from some subtle misconfiguration on our part – not from any sssd or adcli or Kerberos bug. We have a couple of configuration improvements we’re pursuing. (Kerberos max ticket lifetime mismatch between AD and /etc/krb5.conf file for instance.) We are taking sssd’s default settings for ad_maximum_machine_account_password_age and ad_machine_account_password_renewal_opts. So after 30 days, sssd will attempt daily to renew the host Kerberos keytab file. It should re-attempt daily if not renewed. By company policy, our AD disables any machine accounts that have not renewed their credentials in 40 days. So when we find servers that have dropped off the domain, it’s because they have not renewed their AD machine accounts in 40 days. We have SR’s open with our OS vendors (Redhat and Oracle respectively) for months now. To no great help. (They gave a few suggestions, but none panned out.) We thought we were hitting this bug: https://github.com/SSSD/sssd/issues/4762 But packet captures proved that adcli update is using TCP on RHEL7/8. Thus, this might be a potential problem, but only on RHEL6. (BTW ‘udp_preference_limit = 0’ doesn’t force use of TCP for the kpasswd invocation in RHEL6 – it still uses UDP. Thus, the recommended work-around for this bug doesn’t work.) So that isn’t our underlying problem. We’re at a loss now – as you can see, we’re grasping at straws. How can we troubleshoot sssd’s ‘automatic Kerberos Host keytab renewal’ process? Whenever we inspect a particular server it works. We can’t run all sssd clients at debug level 9; it fills up /var/log filesystem after a few days of that. We’re interested in troubleshooting that one particular sssd process on all clients; not all parts of sssd. Other than a steep learning curve (on our part), obscure situations (like DMZ auto-discovery of AD controllers) and exotic scenarios (like above), we’re quite happy with our 2 yr journey of direct AD integration with sssd. Obviously, the troubleshooting tools on RHEL6 are very minimal. But certainly, overall the quality of sssd on RHEL7/8 is excellent. AD integration has innumerable devils in the details; I’m amazed that sssd performs as well as it does against our multi-domain forest. Spike PS the problem with sssd auto-discovery of AD controllers in DMZs has been fixed in a recent sssd release. The better discovery algorithm was implemented – same one used by Windows clients and commercial products. It’s just that recent sssd version is not on RHEL7 or 8.

8 14

Any way to auto create groups while still mapping UIDs to uidNumber?
by Kurt Stine 05 Oct '21

05 Oct '21

I was told this would be a better place than github issues. We're moving from an ldap environment to an AD environment. This means we have a large amount of users who are still linked with their original ldap UIDs. Unfortunately, changing the UIDs to the automatically assigned AD UIDs is not an option. Is there a way for SSSD to both inherit a UID if it exists or create its own if it doesn't? Also, is it possible to sync AD groups as well? Right now we have this: #ldap_id_mapping = True ldap_user_uid_number = uidNumber ldap_user_gid_number = gidNumber Unfortunately, even removing the last line doesn't sync AD groups. Are there any options to sync them? Thanks, Kurt

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

sssd-users October 2021