hi,

i have added debug = 9 to the domain/idm.domain.local in the sssd.conf of the idm server and restarted sssd.

I have no hits on the server on the time when the client does a lookup using id user@domain and finding nothing (no such user)

this is the sss_domain log server file during the same time (as attachtment with name idm_rhel8server.txt)

During the same minute locally on the rhel8 server I could run the id user@domain command with a positive result:

(2024-03-25 11:17:55): [nss] [sss_domain_get_state] (0x1000): [CID#1] Domain domain.local is Active
(2024-03-25 11:17:55): [nss] [sss_parse_name_for_domains] (0x0200): [CID#1] name 'user@domain' matched expression for domain 'domain.local', user is user
(2024-03-25 11:17:55): [nss] [sss_nss_get_object_send] (0x0400): [CID#1] Client [0x55686dfef680][22]: sent cache request #166
(2024-03-25 11:17:55): [nss] [cache_req_set_name] (0x0400): [CID#1] CR #166: Setting name [user]
(2024-03-25 11:17:55): [nss] [cache_req_select_domains] (0x0400): [CID#1] CR #166: Performing a single domain search
(2024-03-25 11:17:55): [nss] [sss_domain_get_state] (0x1000): [CID#1] Domain idm.domain.local is Active
(2024-03-25 11:17:55): [nss] [sss_domain_get_state] (0x1000): [CID#1] Domain domain.local is Active
(2024-03-25 11:17:55): [nss] [cache_req_search_domains] (0x0400): [CID#1] CR #166: Search will check the cache and check the data provider
(2024-03-25 11:17:55): [nss] [cache_req_validate_domain_type] (0x2000): [CID#1] Request type POSIX-only for domain domain.local type POSIX is valid
(2024-03-25 11:17:55): [nss] [cache_req_set_domain] (0x0400): [CID#1] CR #166: Using domain [domain.local]
(2024-03-25 11:17:55): [nss] [cache_req_prepare_domain_data] (0x0400): [CID#1] CR #166: Preparing input data for domain [domain.local] rules
(2024-03-25 11:17:55): [nss] [cache_req_search_send] (0x0400): [CID#1] CR #166: Looking up user@domain.local
(2024-03-25 11:17:55): [nss] [cache_req_search_ncache] (0x0400): [CID#1] CR #166: Checking negative cache for [user@domain.local]
(2024-03-25 11:17:55): [nss] [sss_ncache_check_str] (0x2000): [CID#1] Checking negative cache for [NCE/USER/domain.local/user@domain.local]
(2024-03-25 11:17:55): [nss] [cache_req_search_ncache] (0x0400): [CID#1] CR #166: [user@domain.local] is not present in negative cache
(2024-03-25 11:17:55): [nss] [cache_req_search_cache] (0x0400): [CID#1] CR #166: Looking up [user@domain.local] in cache
(2024-03-25 11:17:55): [nss] [cache_req_search_send] (0x0400): [CID#1] CR #166: Returning [user@domain.local] from cache
(2024-03-25 11:17:55): [nss] [cache_req_search_ncache_filter] (0x0400): [CID#1] CR #166: This request type does not support filtering result by negative cache
(2024-03-25 11:17:55): [nss] [cache_req_create_and_add_result] (0x0400): [CID#1] CR #166: Found 1 entries in domain domain.local
(2024-03-25 11:17:55): [nss] [cache_req_done] (0x0400): [CID#1] CR #166: Finished: Success
(2024-03-25 11:17:55): [nss] [sss_nss_protocol_done] (0x4000): [CID#1] Sending reply: success


So unclear why I can resolve the user locally on the idm server but not on the client.

What else can I try?

Thanks for your support.

--
regards,
Natxo

On Fri, Mar 22, 2024 at 5:15 PM Alexander Bokovoy <abokovoy@redhat.com> wrote:
On Пят, 22 сак 2024, Natxo Asenjo via FreeIPA-users wrote:
>hi,
>
>thanks for replying. Yes, we had seen this already, that is also why I was
>pointing to the rhel 8 idm server in krb5.conf en sssd.conf
>
>I missed the dns_lookup_kdc = false directive though, in krb5.conf
>[libdefaults] section, I modified it, tried but still not working.
>
>The rest points to the one host.
>
>#File modified by ipa-client-install
>
>includedir /etc/krb5.conf.d/
>[libdefaults]
>  default_realm = IDM.DOMAIN.LOCAL
>  dns_lookup_realm = true
>  rdns = false
>  dns_canonicalize_hostname = false
>
>  dns_lookup_kdc = false
>  ticket_lifetime = 24h
>  forwardable = true
>  udp_preference_limit = 0
>  default_ccache_name = KEYRING:persistent:%{uid}
>
>[realms]
>
>  IDM.DOMAIN.LOCAL = {
>    kdc =ds.idm.domain.local:88
>    master_kdc = ds.idm.domain.local:88
>    admin_server = ds.idm.domain.local:749
>    kpasswd_server = ds.idm.domain.local:464
>    default_domain = idm.domain.local
>    pkinit_anchors =
>FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
>    pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
>  }
>
>
>and this:
>
># grep ipa_server /etc/sssd/sssd.conf
>ipa_server = ds.idm.domain.local
>
>When I try ssh from a remote host, I see that sssd tries getting info for a
>trusted user:
>
>(2024-03-22 16:42:35): [be[idm.domain.local]] [dp_get_account_info_send]
>(0x0200): Got request for [0x1][BE_REQ_USER][name=user@ad]
>(2024-03-22 16:42:35): [be[idm.domain.local]] [dp_attach_req] (0x0400):
>[RID#8] DP Request [Account #8]: REQ_TRACE: New request. [sssd.nss CID #4]
>Flags [0x0001].
>(2024-03-22 16:42:35): [be[idm.domain.local]] [dp_attach_req] (0x0400):
>[RID#8] Number of active DP request: 1
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sss_domain_get_state]
>(0x1000): [RID#8] Domain idm.domain.local is Active
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sss_domain_get_state]
>(0x1000): [RID#8] Domain domain.local is Active
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sss_domain_get_state]
>(0x1000): [RID#8] Domain idm.domain.local is Active
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sss_domain_get_state]
>(0x1000): [RID#8] Domain domain.local is Active
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sss_domain_get_state]
>(0x1000): [RID#8] Domain idm.domain.local is Active
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sss_domain_get_state]
>(0x1000): [RID#8] Domain domain.local is Active
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sdap_id_op_connect_step]
>(0x4000): [RID#8] reusing cached connection
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sdap_id_conn_data_not_idle]
>(0x4000): [RID#8] Marking connection as not idle
>(2024-03-22 16:42:35): [be[idm.domain.local]] [ipa_s2n_get_acct_info_send]
>(0x0400): [RID#8] Sending request_type: [REQ_FULL_WITH_MEMBERS] for trust
>user [user@ad] to IPA server
>(2024-03-22 16:42:35): [be[idm.domain.local]] [ipa_s2n_exop_send] (0x0400):
>[RID#8] Executing extended operation
>(2024-03-22 16:42:35): [be[idm.domain.local]] [ipa_s2n_exop_send] (0x2000):
>[RID#8] ldap_extended_operation sent, msgid = 22
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sdap_op_add] (0x2000):
>[RID#8] New operation 22 timeout 6
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sdap_process_result]
>(0x2000): Trace: sh[0x55bcce14e400], connected[1], ops[0x55bcce119870],
>ldap[0x55bcce14c1c0]
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sdap_process_message]
>(0x4000): [RID#8] Message type: [LDAP_RES_EXTENDED]
>(2024-03-22 16:42:35): [be[idm.domain.local]] [sdap_call_op_callback]
>(0x20000): [RID#8] Handling LDAP operation [22][server: [xxx:389] IPA EXOP]
>took [0.752] milliseconds.
>(2024-03-22 16:42:35): [be[idm.domain.local]] [ipa_s2n_exop_done] (0x0400):
>[RID#8] ldap_extended_operation result: No such object(32), (null).
>
>
>but it finds nothing it seems.
>
>on the idm host everything seems to be running:
>
># ipactl status
>Directory Service: RUNNING
>krb5kdc Service: RUNNING
>kadmin Service: RUNNING
>named Service: RUNNING
>httpd Service: RUNNING
>ipa-custodia Service: RUNNING
>smb Service: RUNNING
>winbind Service: RUNNING
>ipa-otpd Service: RUNNING
>ipa-dnskeysyncd Service: RUNNING
>ipa: INFO: The ipactl command was successful
>
>
>And now I don't know what else to try. It seems that the trust agent is not
>answering properly, but how do I check this?

So this looks like a different problem. The client talks to IPA server
for s2n request and the server does not find anything. It means you need
to follow normal SSSD debugging process: collect SSSD debug logs at the
client and at the server at the time of the request, check what happened
on the SSSD on the IPA server that caused it to not find the object.

Usually there are:
  - issues talking to AD DCs
  - unresolvable SIDs of group members or groups one is a member of
  - use of domain-local groups

SSSD domain log would have more details, see https://sssd.io/troubleshooting/ipa_provider.html

>
>
>
>
>On Fri, Mar 22, 2024 at 4:12 PM Alexander Bokovoy <abokovoy@redhat.com>
>wrote:
>
>> On Пят, 22 сак 2024, Natxo Asenjo via FreeIPA-users wrote:
>> >hi,
>> >
>> >at work we are having *interesting* problems with our migration from idm
>> >servers running rhel 7 to new rhel 8 idm servers.
>> >
>> >We have a AD -> idm trust in place, this is working properly.
>> >
>> >AD is domain.local
>> >IDM is idm.domain.local
>> >
>> >We add replicas to the set, and this runs properly. No replication errors.
>> >
>> >Basically the problem is we cannot log in to newly joined systems running
>> >rhel 8 as trusted users from AD.
>> >
>> >We have a new rhel 8 idm server which has also the trust agent and trust
>> >controller role installed.
>> >
>> >We want to login as a trusted AD user to a rhel 8 host which has this new
>> >rhel 8 idm server as its ipa host, we have forced it using this sssd.conf:
>> >
>> >[domain/idm.domain.local]
>> >
>> >id_provider = ipa
>> >ipa_server = ds.idm.domain.local
>> >ipa_domain = idm.domain.local
>> >ipa_hostname = host.idm.domain.local
>> >auth_provider = ipa
>> >chpass_provider = ipa
>> >access_provider = ipa
>> >cache_credentials = True
>> >ldap_tls_cacert = /etc/ipa/ca.crt
>> >dyndns_update = True
>> >dyndns_iface = ens192
>> >krb5_store_password_if_offline = True
>> >[sssd]
>> >services = nss, pam, ssh, sudo
>> >
>> >domains = idm.domain.local
>> >full_name_format = %1$s
>> >debug = 9
>> >[nss]
>> >homedir_substring = /home
>> >override_homedir = /home/%u
>> >
>> >[pam]
>> >
>> >[sudo]
>> >
>> >[autofs]
>> >
>> >[ssh]
>> >
>> >[pac]
>> >
>> >[ifp]
>> >
>> >[session_recording]
>> >
>> >
>> >We also modified krb5.conf on the client to find the IDM realm only on the
>> >rhel 8 idm server, not the rhel 7. So we disabled srv dns autodiscovery
>> for
>> >the IDM.DOMAIN.LOCAL realm:
>> >
>> ># cat /etc/krb5.conf
>> >#File modified by ipa-client-install
>> >
>> >includedir /etc/krb5.conf.d/
>> >[libdefaults]
>> >  default_realm = IDM.DOMAIN.LOCAL
>> >dns_lookup_realm = true
>> >  rdns = false
>> >  dns_canonicalize_hostname = false
>> >dns_lookup_kdc = true
>> >  ticket_lifetime = 24h
>> >  forwardable = true
>> >  udp_preference_limit = 0
>> >  default_ccache_name = KEYRING:persistent:%{uid}
>> >
>> >
>> >[realms]
>> >  IDM.DOMAIN.LOCAL = {
>> >    kdc = ds.idm.domain.local:88
>> >    master_kdc = ds.idm.domain.local:88
>> >    admin_server = ds.idm.domain.local:749
>> >    kpasswd_server = ds.idm.domain.local:464
>> >    default_domain = idm.domain.local
>> >    pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
>> >    pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
>> >
>> >  }
>> >
>> >
>> >[domain_realm]
>> >  .idm.domain.local = IDM.DOMAIN.LOCAL
>> >  idm.domain.local = IDM.DOMAIN.LOCAL
>> >  host.idm.domain.local = IDM.DOMAIN.LOCAL
>> >
>> >On the rhel 8 client, I can kinit with the host keytab, this work, I get a
>> >ticket with the host principal.
>> >
>> >I can also kinit using a trusted AD user, this works, I get a ticket of
>> the
>> >AD domain.
>> >
>> >But as soon as I try logging in from ssh, it does not work. It does not
>> >recognize the user.
>> >
>> >Running id trusteduser@ad does not wok either (no such user)
>> >
>> >I have run out of ideas, to be honest. I do not know how to troubleshoot
>> >this anymore. The rhel8 idm server finds the users, I can log in there
>> >without any problem too thanks to the rbac rules, but this rhel8 client
>> >simpley will not see the users.
>>
>> This all sounds like you see SPAKE vs non-SPAKE behavior by the clients.
>>
>> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/migrating_to_identity_management_on_rhel_8/migrate-7-to-8_migrating
>> covers this in a yellow warning box:
>>
>> -------------------------------------
>> Important
>>
>> RHEL 8 supports SPAKE and IdP pre-authentication, but RHEL 7 does not.
>> Having RHEL 8 servers with SPAKE or IdP enabled in a RHEL 7 IdM
>> deployment may lead to problems such as users not being able to log in.
>>
>> Red Hat strongly recommends that all servers in an IdM deployment are
>> migrated as quickly as possible and that older systems should not be
>> left in operation for extended periods of time.
>>
>> For more information, see:
>>
>>      https://access.redhat.com/solutions/7053377
>>      https://access.redhat.com/solutions/3529911
>> -------------------------------------
>>
>> Specifically, https://access.redhat.com/solutions/7053377 should have
>> enough technical details to confirm it is the case.
>>
>>
>> --
>> / Alexander Bokovoy
>> Sr. Principal Software Engineer
>> Security / Identity Management Engineering
>> Red Hat Limited, Finland
>>
>>
>
>--
>--
>Groeten,
>natxo




--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland



--
--
Groeten,
natxo