6:42 p.m.
So yesterday we upgrade our older IPA 3.x servers (RHEL 6.8) to the latest
and greatest (RHEL 6.9) and it seemed to be working as expected. Came in
the next day and older IPA 4.2 server (RHEL 7.2) was having issues so
thought it would be a good time patch it up to the latest (IPA 4.4 and RHEL
7.3) seemed okay after it came back up but after a little while hosts
configured to use IPA stopped allowing access via ssh.
I can explicitly put in sssd.conf to use the patched up 4.4 server and I
seem to be able to log in via ssh but when a server is pointing to one of
the 3.x servers it just gives me an access denied. The strange part is
some accounts (like my test account) can log in to some servers via ssh and
can even sudo to root where available regardless of which IPA server the
host is using.
Any help would be appreciated because I'm stumped on this one. Just let
me know what logs would be helpful in troubleshooting.
Thanks!
--
John Bowman
11:28 p.m.
After upping the log levels on sssd on one of the failing servers I saw this in one of the
sssd log files:
from sssd_pamd.log:
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_ncache_check_str] (0x2000): Checking negative
cache for [NCE/USER/domain.tld/jbowman]
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_issue_request] (0x0400): Issuing request
for [0x41b5c0:3:jbowman@domain.tld]
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_get_account_msg] (0x0400): Creating request
for [domain.tld][3][1][name=jbowman]
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sbus_add_timeout] (0x2000): 0x20ef8a0
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_internal_get_send] (0x0400): Entering
request [0x41b5c0:3:jbowman@domain.tld]
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sbus_remove_timeout] (0x2000): 0x20ef8a0
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_get_reply] (0x1000): Got reply from Data
Provider - DP error code: 3 errno: 22 error message: Init Groups Failed
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [pam_check_user_dp_callback] (0x0040): Unable to
get information from Data Provider
Error: 3, 22, Init Groups Failed
from sssd_domain.tld.log
(Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [hbac_eval_user_element] (0x0080): Parse
error on [cn=system: manage service
principals+nsuniqueid=e8d2f834-512111e7-9205b5bf-43202000,cn=permissions,cn=pbac,dc=domain,dc=tld]
(Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [hbac_ctx_to_rules] (0x0020): Could not
construct eval request
(Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [ipa_hbac_evaluate_rules] (0x0020):
Could not construct HBAC rules
(Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_id_op_destroy] (0x4000): releasing
operation connection
(Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [be_pam_handler_callback] (0x0100):
Backend returned: (3, 4, <NULL>) [Internal Error (System error)]
(Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [be_pam_handler_callback] (0x0100):
Sending result [4][domain.tld]
(Wed Jun 14 22:55:37 2017) [sssd[be[domainn.tld]]] [be_pam_handler_callback] (0x0100):
Sent result [4][domain.tld]
(Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_process_result] (0x2000): Trace:
sh[0x7ea6b0], connected[1], ops[(nil)], ldap[0x844de0]
(Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_process_result] (0x2000): Trace:
ldap_result found nothing!
(Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_dispatch] (0x4000): dbus conn:
7B2A00
(Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_dispatch] (0x4000): Dispatching.
(Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_message_handler] (0x4000):
Received SBUS method [ping]
I saw a similar issue in a previous posting to the list:
https://www.redhat.com/archives/freeipa-users/2017-January/msg00286.html
I was wondering if these errors might be related to the issues I'm seeing currently
since they seem very similar so far...
1:09 a.m.
On Thu, Jun 15, 2017 at 04:28:13AM -0000, john.bowman--- via FreeIPA-users wrote:
After upping the log levels on sssd on one of the failing servers I
saw this in one of the sssd log files:
from sssd_pamd.log:
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_ncache_check_str] (0x2000): Checking negative
cache for [NCE/USER/domain.tld/jbowman]
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_issue_request] (0x0400): Issuing request
for [0x41b5c0:3:jbowman@domain.tld]
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_get_account_msg] (0x0400): Creating
request for [domain.tld][3][1][name=jbowman]
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sbus_add_timeout] (0x2000): 0x20ef8a0
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_internal_get_send] (0x0400): Entering
request [0x41b5c0:3:jbowman@domain.tld]
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sbus_remove_timeout] (0x2000): 0x20ef8a0
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_get_reply] (0x1000): Got reply from Data
Provider - DP error code: 3 errno: 22 error message: Init Groups Failed
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [pam_check_user_dp_callback] (0x0040): Unable to
get information from Data Provider
Error: 3, 22, Init Groups Failed
from sssd_domain.tld.log
(Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [hbac_eval_user_element] (0x0080):
Parse error on [cn=system: manage service
principals+nsuniqueid=e8d2f834-512111e7-9205b5bf-43202000,cn=permissions,cn=pbac,dc=domain,dc=tld]
Yes, only recent vresions of sssd can skip over the replication
conflicts. I would recommend to clear the conflicts manually.
> (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [hbac_ctx_to_rules] (0x0020): Could
not construct eval request
> (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [ipa_hbac_evaluate_rules] (0x0020):
Could not construct HBAC rules
> (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_id_op_destroy] (0x4000):
releasing operation connection
> (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [be_pam_handler_callback] (0x0100):
Backend returned: (3, 4, <NULL>) [Internal Error (System error)]
> (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [be_pam_handler_callback] (0x0100):
Sending result [4][domain.tld]
> (Wed Jun 14 22:55:37 2017) [sssd[be[domainn.tld]]] [be_pam_handler_callback]
(0x0100): Sent result [4][domain.tld]
> (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_process_result] (0x2000):
Trace: sh[0x7ea6b0], connected[1], ops[(nil)], ldap[0x844de0]
> (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_process_result] (0x2000):
Trace: ldap_result found nothing!
> (Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_dispatch] (0x4000): dbus
conn: 7B2A00
> (Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_dispatch] (0x4000):
Dispatching.
> (Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_message_handler] (0x4000):
Received SBUS method [ping]
>
> I saw a similar issue in a previous posting to the list:
> https://www.redhat.com/archives/freeipa-users/2017-January/msg00286.html
>
> I was wondering if these errors might be related to the issues I'm seeing
currently since they seem very similar so far...
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
8:07 a.m.
You'll have to forgive my ignorance here since I'm still fairly new to IPA and
fortunately haven't run in to many issues as of yet.
The three IPA 3.0 servers all have what look to be following conflicts:
$ ldapsearch -D "cn=directory manager" -w secret -b "dc=domain,dc=tld"
"nsds5ReplConflict=*" \* nsds5ReplConflict | grep nsds5ReplConflict
# filter: nsds5ReplConflict=*
# requesting: * nsds5ReplConflict
nsds5ReplConflict: namingConflict cn=ipa4-4.domain.tld,cn=masters,cn=ipa,cn
nsds5ReplConflict: namingConflict dnahostname=ipa4-4.domain.tld+dnaportnum=
nsds5ReplConflict: namingConflict cn=ipaservers,cn=hostgroups,cn=accounts,dc=z
nsds5ReplConflict: namingConflict cn=ipaservers,cn=ng,cn=alt,dc=domain,dc=tld
nsds5ReplConflict: namingConflict cn=domain,cn=topology,cn=ipa,cn=etc,dc=domain,
nsds5ReplConflict: namingConflict cn=locations,cn=etc,dc=domain,dc=tld
nsds5ReplConflict: namingConflict cn=dns administrators,cn=privileges,cn=pbac,
nsds5ReplConflict: namingConflict cn=dns servers,cn=privileges,cn=pbac,dc=domain
nsds5ReplConflict: namingConflict cn=cas,cn=ca,dc=domain,dc=tld
nsds5ReplConflict: namingConflict cn=dogtag,cn=custodia,cn=ipa,cn=etc,dc=domain,
nsds5ReplConflict: namingConflict cn=ca,cn=topology,cn=ipa,cn=etc,dc=domain,dc=u
nsds5ReplConflict: namingConflict cn=system: add ca,cn=permissions,cn=pbac,dc=
nsds5ReplConflict: namingConflict cn=system: delete ca,cn=permissions,cn=pbac,
nsds5ReplConflict: namingConflict cn=system: modify ca,cn=permissions,cn=pbac,
nsds5ReplConflict: namingConflict cn=system: read cas,cn=permissions,cn=pbac,d
nsds5ReplConflict: namingConflict cn=system: modify dns servers configuration,
nsds5ReplConflict: namingConflict cn=system: read dns servers configuration,cn
nsds5ReplConflict: namingConflict cn=system: add ipa locations,cn=permissions,
nsds5ReplConflict: namingConflict cn=system: modify ipa locations,cn=permissio
nsds5ReplConflict: namingConflict cn=system: read ipa locations,cn=permissions
nsds5ReplConflict: namingConflict cn=system: remove ipa locations,cn=permissio
nsds5ReplConflict: namingConflict cn=system: read locations of ipa servers,cn=
nsds5ReplConflict: namingConflict cn=system: read status of services on ipa se
nsds5ReplConflict: namingConflict cn=system: manage service principals,cn=perm
nsds5ReplConflict: namingConflict cn=system: manage user principals,cn=permiss
nsds5ReplConflict: namingConflict dnahostname=ipa4-4.domain.tld+dnaportnum=
While the IPA 4.4 server shows no conflicts:
$ ldapsearch -D "cn=directory manager" -w secret -b "dc=domain,dc=tld"
"nsds5ReplConflict=*" \* nsds5ReplConflict | grep nsds5ReplConflict
# filter: nsds5ReplConflict=*
# requesting: * nsds5ReplConflict
So I would need to delete/modify the conflicts on the IPA 3.0 servers but the IPA 4.4
server should be okay, correct? Is there any impact to running the ldapmodify command to
remove these conflicts while services are running? Would I need to do this on each of the
IPA 3.x servers?
Looking at one of the conflicts on one of the IPA 3.0:
$ ldapsearch -D "cn=directory manager" -w secret -b "dc=domain,dc=tld"
"cn=domain"
# extended LDIF
#
# LDAPv3
# base <dc=domain,dc=tld> with scope subtree
# filter: cn=domain
# requesting: ALL
#
# domain, topology, ipa, etc, domain.us
dn: cn=domain,cn=topology,cn=ipa,cn=etc,dc=domain,dc=tld
cn: domain
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
uccessfulauth krblastfailedauth krbloginfailedcount
objectClass: top
objectClass: iparepltopoconf
ipaReplTopoConfRoot: dc=domain,dc=tld
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
ternalModifyTimestamp
# domain + e8d2f70e-512111e7-9205b5bf-43202000, topology, ipa, etc, domain.us
dn: cn=domain+nsuniqueid=e8d2f70e-512111e7-9205b5bf-43202000,cn=topology,cn=ip
a,cn=etc,dc=domain,dc=tld
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
ternalModifyTimestamp
ipaReplTopoConfRoot: dc=domain,dc=tld
objectClass: top
objectClass: iparepltopoconf
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
uccessfulauth krblastfailedauth krbloginfailedcount
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
cn: domain
# search result
search: 2
result: 0 Success
# numResponses: 3
# numEntries: 2
Would I need to remove the "dn:
cn=domain+nsuniqueid=e8d2f70e-512111e7-9205b5bf-43202000" entry in this case? And do
that removal on each server?
Thank you and any help is appreciated!
9:41 a.m.
On Thu, Jun 15, 2017 at 01:07:27PM -0000, john.bowman--- via FreeIPA-users wrote:
You'll have to forgive my ignorance here since I'm still
fairly new to IPA and fortunately haven't run in to many issues as of yet.
The three IPA 3.0 servers all have what look to be following conflicts:
$ ldapsearch -D "cn=directory manager" -w secret -b
"dc=domain,dc=tld" "nsds5ReplConflict=*" \* nsds5ReplConflict | grep
nsds5ReplConflict
# filter: nsds5ReplConflict=*
# requesting: * nsds5ReplConflict
nsds5ReplConflict: namingConflict cn=ipa4-4.domain.tld,cn=masters,cn=ipa,cn
nsds5ReplConflict: namingConflict dnahostname=ipa4-4.domain.tld+dnaportnum=
nsds5ReplConflict: namingConflict cn=ipaservers,cn=hostgroups,cn=accounts,dc=z
nsds5ReplConflict: namingConflict cn=ipaservers,cn=ng,cn=alt,dc=domain,dc=tld
nsds5ReplConflict: namingConflict cn=domain,cn=topology,cn=ipa,cn=etc,dc=domain,
nsds5ReplConflict: namingConflict cn=locations,cn=etc,dc=domain,dc=tld
nsds5ReplConflict: namingConflict cn=dns administrators,cn=privileges,cn=pbac,
nsds5ReplConflict: namingConflict cn=dns servers,cn=privileges,cn=pbac,dc=domain
nsds5ReplConflict: namingConflict cn=cas,cn=ca,dc=domain,dc=tld
nsds5ReplConflict: namingConflict cn=dogtag,cn=custodia,cn=ipa,cn=etc,dc=domain,
nsds5ReplConflict: namingConflict cn=ca,cn=topology,cn=ipa,cn=etc,dc=domain,dc=u
nsds5ReplConflict: namingConflict cn=system: add ca,cn=permissions,cn=pbac,dc=
nsds5ReplConflict: namingConflict cn=system: delete ca,cn=permissions,cn=pbac,
nsds5ReplConflict: namingConflict cn=system: modify ca,cn=permissions,cn=pbac,
nsds5ReplConflict: namingConflict cn=system: read cas,cn=permissions,cn=pbac,d
nsds5ReplConflict: namingConflict cn=system: modify dns servers configuration,
nsds5ReplConflict: namingConflict cn=system: read dns servers configuration,cn
nsds5ReplConflict: namingConflict cn=system: add ipa locations,cn=permissions,
nsds5ReplConflict: namingConflict cn=system: modify ipa locations,cn=permissio
nsds5ReplConflict: namingConflict cn=system: read ipa locations,cn=permissions
nsds5ReplConflict: namingConflict cn=system: remove ipa locations,cn=permissio
nsds5ReplConflict: namingConflict cn=system: read locations of ipa servers,cn=
nsds5ReplConflict: namingConflict cn=system: read status of services on ipa se
nsds5ReplConflict: namingConflict cn=system: manage service principals,cn=perm
nsds5ReplConflict: namingConflict cn=system: manage user principals,cn=permiss
nsds5ReplConflict: namingConflict dnahostname=ipa4-4.domain.tld+dnaportnum=
While the IPA 4.4 server shows no conflicts:
$ ldapsearch -D "cn=directory manager" -w secret -b
"dc=domain,dc=tld" "nsds5ReplConflict=*" \* nsds5ReplConflict | grep
nsds5ReplConflict
# filter: nsds5ReplConflict=*
# requesting: * nsds5ReplConflict
Depends on whether you need to keep the data on the v3 machine and
whether the data on the v4 machine is correct...
But the general guide to managing replication conflicts is:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/...
>
> So I would need to delete/modify the conflicts on the IPA 3.0 servers but the IPA 4.4
server should be okay, correct? Is there any impact to running the ldapmodify command to
remove these conflicts while services are running? Would I need to do this on each of the
IPA 3.x servers?
>
> Looking at one of the conflicts on one of the IPA 3.0:
> $ ldapsearch -D "cn=directory manager" -w secret -b
"dc=domain,dc=tld" "cn=domain"
> # extended LDIF
> #
> # LDAPv3
> # base <dc=domain,dc=tld> with scope subtree
> # filter: cn=domain
> # requesting: ALL
> #
>
> # domain, topology, ipa, etc, domain.us
> dn: cn=domain,cn=topology,cn=ipa,cn=etc,dc=domain,dc=tld
> cn: domain
> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
> uccessfulauth krblastfailedauth krbloginfailedcount
> objectClass: top
> objectClass: iparepltopoconf
> ipaReplTopoConfRoot: dc=domain,dc=tld
> nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
> ternalModifyTimestamp
>
> # domain + e8d2f70e-512111e7-9205b5bf-43202000, topology, ipa, etc, domain.us
> dn: cn=domain+nsuniqueid=e8d2f70e-512111e7-9205b5bf-43202000,cn=topology,cn=ip
> a,cn=etc,dc=domain,dc=tld
> nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
> ternalModifyTimestamp
> ipaReplTopoConfRoot: dc=domain,dc=tld
> objectClass: top
> objectClass: iparepltopoconf
> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
> uccessfulauth krblastfailedauth krbloginfailedcount
> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
> cn: domain
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 3
> # numEntries: 2
>
> Would I need to remove the "dn:
cn=domain+nsuniqueid=e8d2f70e-512111e7-9205b5bf-43202000" entry in this case? And do
that removal on each server?
>
> Thank you and any help is appreciated!
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
12:15 p.m.
Which path would be better? Upgrading sssd on the older machines or attempting to delete
the ldap entries? Both eventually? Does having the namingConflict entries pose a threat
to the system stability?
1:21 a.m.
On Thu, Jun 15, 2017 at 05:15:41PM -0000, john.bowman--- via FreeIPA-users wrote:
Which path would be better? Upgrading sssd on the older machines or
attempting to delete the ldap entries?
I think you want to fix the server side, upgrading sssd is just a quick
kludge to let you access the systems.
2515
days inactive
2517
days old
freeipa-users@lists.fedorahosted.org
6 comments
3 participants
participants (3)
-
Jakub Hrozek -
John Bowman -
john.bowman＠zayo.com