Hi,
I have a weird situation when a user launches a Slurm interactive job on a new compute
node with empty sssd cache. I'm not sure if its an issue with Slurm or SSSD/NSS.
Below is the output of the Slurm session originated on node ip-0A0B0004 and starting
interactive node ip-0A0B0006 by user1 with uid 705601104:
[user1@ip-0A0B0004 ~]$ srun -p lowmem -w lowmem-2 --pty bash
/usr/bin/id: cannot find name for group ID 705600513
/usr/bin/id: cannot find name for user ID 705601104
[I have no name!@ip-0A0B0006 ~]$
From this node I can successfully id user1 or any other AD user (this populates my cache
and subsequent Slurm sessions for any user will resolve):
[I have no name!@ip-0A0B0006 ~]$ id user1
uid=705601104(user1) gid=705600513 groups=705600513,705601103(group1)
[I have no name!@ip-0A0B0006 ~]$
Here is the relevant session from my sssd_nss.log:
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_send] (0x0400): CR #0: New request
'User by ID'
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_select_domains] (0x0400): CR #0:
Performing a multi-domain search
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_domains] (0x0400): CR #0: Search
will check the cache and check the data provider
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_set_domain] (0x0400): CR #0: Using
domain [
jmorey.net]
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_send] (0x0400): CR #0: Looking up
UID:705601104@jmorey.net
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_ncache] (0x0400): CR #0: Checking
negative cache for [UID:705601104@jmorey.net]
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_ncache] (0x0400): CR #0:
[UID:705601104@jmorey.net] is not present in negative cache
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #0: Looking
up [UID:705601104@jmorey.net] in cache
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #0: Object
[UID:705601104@jmorey.net] was not found in cache
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_dp] (0x0400): CR #0: Looking up
[UID:705601104@jmorey.net] in data provider
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request
for [0x562848473820:1:705601104@jmorey.net]
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request
for [jmorey.net][0x1][BE_REQ_USER][idnumber=705601104:-]
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering
request [0x562848473820:1:705601104@jmorey.net]
(Fri Jul 10 22:47:58 2020)
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_common_dp_recv] (0x0040): CR #0: Data
Provider Error: 3, 0, Success
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_common_dp_recv] (0x0400): CR #0: Due to
an error we will return cached data
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #0: Looking
up [UID:705601104@jmorey.net] in cache
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #0: Object
[UID:705601104@jmorey.net] was not found in cache
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_process_result] (0x0400): CR #0:
Finished: Not found
To my inexperienced eye it looks like it is communicating with the backend but receives
some kind of error so it reverts to the cache, which is empty. The other oddity is the
'User by id' request that is trying to resolve based on the uid not the name.
While this is happening I can successfully resolve user1 using SSH to the same node.
The SSH session:
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_send] (0x0400): CR #6: New request
'User by name'
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_process_input] (0x0400): CR #6: Parsing
input name [user1]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_parse_name_for_domains] (0x0200): name
'user1' matched without domain, user is user1
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_set_name] (0x0400): CR #6: Setting name
[user1]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_select_domains] (0x0400): CR #6:
Performing a multi-domain search
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_domains] (0x0400): CR #6: Search
will check the cache and check the data provider
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_set_domain] (0x0400): CR #6: Using
domain [
jmorey.net] (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_prepare_domain_data]
(0x0400): CR #6: Preparing input data for domain [
jmorey.net] rules
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_send] (0x0400): CR #6: Looking up
user1(a)jmorey.net
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_ncache] (0x0400): CR #6: Checking
negative cache for [user1(a)jmorey.net]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_ncache] (0x0400): CR #6:
[user1(a)jmorey.net] is not present in negative cache
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #6: Looking
up [user1(a)jmorey.net] in cache
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #6: Object
[user1(a)jmorey.net] was not found in cache
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_dp] (0x0400): CR #6: Looking up
[user1(a)jmorey.net] in data provider
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request
for [0x564629422820:1:user1@jmorey.net@jmorey.net]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request
for [jmorey.net][0x1][BE_REQ_USER][name=user1@jmorey.net:-]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering
request [0x564629422820:1:user1@jmorey.net@jmorey.net]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data
Provider - DP error code: 0 errno: 0 error message: Success
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #6: Looking
up [user1(a)jmorey.net] in cache
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_ncache_filter] (0x0400): CR #6:
This request type does not support filtering result by negative cache
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_done] (0x0400): CR #6: Returning
updated object [user1(a)jmorey.net]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_create_and_add_result] (0x0400): CR #6:
Found 1 entries in domain
jmorey.net
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_req_destructor] (0x0400): Deleting request:
[0x564629422820:1:user1@jmorey.net@jmorey.net]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_done] (0x0400): CR #6: Finished: Success
Here is my sssd.conf:
[sssd]
config_file_version = 2
services = nss,pam
domains =
jmorey.net
[nss]
filter_users = root
filter_groups = root
debug_level = 7
[pam]
[
domain/jmorey.net]
id_provider = ldap
auth_provider = ldap
access_provider = ldap
debug_level = 7
dns_discovery_domain =
jmorey.net
enumerate = False
cache_credentials = True
case_sensitive = false
ldap_schema = ad
ldap_uri =
ldaps://jmorey-novo-ad.jmorey.net
ldap_user_search_base = cn=Users,dc=jmorey,dc=net
ldap_group_search_base = cn=group1,cn=Users,dc=jmorey,dc=net
ldap_referrals = False
ldap_tls_reqcert = never
ldap_use_tokengroups = True
ldap_id_mapping = True
override_homedir = /mnt/exports/shared/home/%u
fallback_homedir = /shared/home/%u
default_shell = /bin/bash
ldap_access_order = filter, expire
ldap_account_expire_policy = ad
ldap_access_filter = (|(memberOf=cn=group1,cn=Users,dc=jmorey,dc=net))
ldap_default_bind_dn = cn=user 1,cn=Users,dc=jmorey,dc=net
ldap_default_authtok_type = password
ldap_default_authtok =
thanks,
Jerry