4:54 a.m.
On 12/06/17 18:29, Mark Reynolds wrote:
On 06/12/2017 07:32 AM, Nick Campion via FreeIPA-users wrote:
>
> Thanks Mark,
>
> So this example is a user password change using kinit, the password
> has been changed on freeipa02 but not then replicated to the others.
> This happens for other records, but I don't have examples of these at
> the moment.
>
> As far as I'm aware, there is no fractal replication set up.
>
IPA uses fractional replication, and it's possible these attributes
are ignored/skipped. To confirm you can run this search on freeipa02:
ldapsearch -D "cn=directory manager" -W -b cn=config -xLLL
objectclass=nsds5ReplicationAgreement
Freeipa02:
dn:
cn=meTofreeipa01.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
tree,cn=config
cn:
meTofreeipa01.mgmt.example.com
description: me to
freeipa01.mgmt.example.com
nsDS5ReplicaBindMethod:
SASL/GSSAPI
nsDS5ReplicaHost:
freeipa01.mgmt.example.com
nsDS5ReplicaPort:
389
nsDS5ReplicaRoot:
dc=ipa,dc=example,dc=com
nsDS5ReplicaTransportInfo:
LDAP
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth
krbloginfailedcount
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
krblastsuccessfulauth krblastfailedauth
krbloginfailedcount
nsds50ruv: {replicageneration}
57867ff5000000040000
nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389}
57867ffe000000040000
593693b7001100040000
nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389}
57867ffa000000030000 5930e345000200030000
nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389}
59355988000000050000 59369317000300050000
nsds5ReplicaEnabled: on
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp
internalModifiersName internalModifyTimestamp
nsds5replicaTimeout: 120
nsruvReplicaLastModified: {replica 4
ldap://freeipa01.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 3
ldap://freeipa02.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 5
ldap://freeipa03.mgmt.example.com:389} 00000000
objectClass: nsds5replicationagreement
objectClass: top
objectClass: ipaReplTopoManagedAgreement
ipaReplTopoManagedAgreementState: managed agreement - controlled by
topology plugin
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090432Z
nsds5replicaLastUpdateEnd: 20170613090432Z
nsds5replicaChangesSentSinceStartup:: MzoxNTkvMjM5ODI0NyA1OjI0LzAg
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z
dn:
cn=meTofreeipa03.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexample2Cdc\3Dcom,cn=mapping
tree,cn=config
cn: meTofreeipa03.mgmt.example.com
objectClass: nsds5replicationagreement
objectClass: top
objectClass: ipaReplTopoManagedAgreement
nsDS5ReplicaTransportInfo: LDAP
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp
internalModifiersName internalModifyTimestamp
nsDS5ReplicaRoot: dc=ipa,dc=example,dc=com
nsDS5ReplicaHost: freeipa03.mgmt.example.com
nsds5replicaTimeout: 120
nsDS5ReplicaPort: 389
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth
krbloginfailedcount
description: me to freeipa03.mgmt.example.com
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
krblastsuccessfulauth krblastfailedauth krbloginfailedcount
ipaReplTopoManagedAgreementState: managed agreement - controlled by
topology plugin
nsds5ReplicaEnabled: on
nsds50ruv: {replicageneration} 57867ff5000000040000
nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389}
59355988000000050000 5936937b000200050000
nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389}
57867ffa000000030000 5930e345000200030000
nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389}
57867ffe000000040000 593693b7000c00040000
nsruvReplicaLastModified: {replica 5
ldap://freeipa03.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 3
ldap://freeipa02.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 4
ldap://freeipa01.mgmt.example.com:389} 00000000
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090432Z
nsds5replicaLastUpdateEnd: 20170613090432Z
nsds5replicaChangesSentSinceStartup:: MzoxMzkvMTkxMDI0NSA0OjkwNS8wIA==
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z
dn:
cn=cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat,cn=replica,cn=o\3Dipaca,cn=mapping
tree,cn=config
cn: cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat
description: cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat
nsDS5ReplicaBindDN: cn=Replication Manager
masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaBindMethod: Simple
nsDS5ReplicaCredentials: Redacted
nsDS5ReplicaHost: freeipa01.mgmt.example.com
nsDS5ReplicaPort: 389
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaTransportInfo: TLS
nsds50ruv: {replicageneration} 57868040000000600000
nsds50ruv: {replica 96 ldap://freeipa01.mgmt.example.com:389}
57868041000000600000 593692b2000000600000
nsds50ruv: {replica 97 ldap://freeipa02.mgmt.example.com:389}
57868050000000610000 59355a39000400610000
nsruvReplicaLastModified: {replica 96
ldap://freeipa01.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 97
ldap://freeipa02.mgmt.example.com:389} 00000000
objectClass: top
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
ipaReplTopoManagedAgreementState: managed agreement - controlled by
topology plugin
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090226Z
nsds5replicaLastUpdateEnd: 20170613090226Z
nsds5replicaChangesSentSinceStartup:
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z
Then please share these entries so we can see how they are
configured. Perhaps do this on freeipa01 as well for comparison.
Freeipa01:
dn:
cn=freeipa01.mgmt.example.com-to-freeipa03.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexamle\2Cdc\3Dcom,cn=mapping
tree,cn=config
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
objectClass: top
cn: freeipa01.mgmt.example.com-to-freeipa03.mgmt.example.com
nsDS5ReplicaHost: freeipa03.mgmt.example.com
nsDS5ReplicaPort: 389
nsds5replicaTimeout: 300
nsDS5ReplicaRoot: dc=ipa,dc=example,dc=com
description: freeipa01.mgmt.example.com to freeipa03.mgmt.example.com
ipaReplTopoManagedAgreementState: managed agreement - generated by
topology plugin
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth
krbloginfailedcount
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp
internalModifiersName internalModifyTimestamp
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds50ruv: {replicageneration} 57867ff5000000040000
nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389}
59355988000000050000 593b4a2f000300050000
nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389}
57867ffa000000030000 5937cccd000300030000
nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389}
57867ffe000000040000 593b4b2f000700040000
nsruvReplicaLastModified: {replica 5
ldap://freeipa03.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 3
ldap://freeipa02.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 4
ldap://freeipa01.mgmt.example.com:389} 00000000
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090421Z
nsds5replicaLastUpdateEnd: 20170613090421Z
nsds5replicaChangesSentSinceStartup:: NDoxMTM0MS8yNTkyNzgxMyA=
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z
dn:
cn=meTofreeipa02.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
tree,cn=config
cn: meTofreeipa02.mgmt.example.com
description: me to freeipa02.mgmt.example.com
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicaHost: freeipa02.mgmt.example.com
nsDS5ReplicaPort: 389
nsDS5ReplicaRoot: dc=ipa,dc=example,dc=com
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth
krbloginfailedcount
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds50ruv: {replicageneration} 57867ff5000000040000
nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389}
57867ffa000000030000 5937ccd3000a00030000
nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389}
57867ffe000000040000 593b4b2f000700040000
nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389}
59355988000000050000 593b49d8000400050000
nsds5ReplicaEnabled: on
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp
internalModifiersName internalModifyTimestamp
nsds5replicaTimeout: 120
nsruvReplicaLastModified: {replica 3
ldap://freeipa02.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 4
ldap://freeipa01.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 5
ldap://freeipa03.mgmt.example.com:389} 00000000
objectClass: nsds5replicationagreement
objectClass: top
objectClass: ipaReplTopoManagedAgreement
ipaReplTopoManagedAgreementState: managed agreement - controlled by
topology plugin
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090421Z
nsds5replicaLastUpdateEnd: 20170613090421Z
nsds5replicaChangesSentSinceStartup::
NDoxMDkyNy8yNTYzNjIwNiA1OjM3OC8wIDA6MTQvMCA=
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z
dn:
cn=masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat,cn=replica,cn=o\3Dipaca,cn=mapping
tree,cn=config
cn: masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat
description: masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat
nsDS5ReplicaBindDN: cn=Replication Manager
cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaBindMethod: Simple
nsDS5ReplicaCredentials: Redacted
nsDS5ReplicaHost: freeipa02.mgmt.example.com
nsDS5ReplicaPort: 389
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaTransportInfo: TLS
nsds50ruv: {replicageneration} 57868040000000600000
nsds50ruv: {replica 97 ldap://freeipa02.mgmt.example.com:389}
57868050000000610000 59355a39000400610000
nsds50ruv: {replica 96 ldap://freeipa01.mgmt.example.com:389}
57868047000000600000 593b488f000000600000
nsruvReplicaLastModified: {replica 97
ldap://freeipa02.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 96
ldap://freeipa01.mgmt.example.com:389} 00000000
objectClass: top
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
ipaReplTopoManagedAgreementState: managed agreement - controlled by
topology plugin
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090225Z
nsds5replicaLastUpdateEnd: 20170613090226Z
nsds5replicaChangesSentSinceStartup:: OTY6MzM3LzAg
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z
>
> Freeipa01:
>
> # dynamic-kepler, users, accounts, ipa.example.com
> dn: uid=dynamic-kepler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com
> uid: dynamic-kepler
> krbLastPwdChange: 20170608170011Z
> krbPasswordExpiration: 20170608170011Z
>
> Freeipa02:
>
> # dynamic-kepler, users, accounts, ipa.example.com
> dn: uid=dynamic-kepler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com
> uid: dynamic-kepler
> krbLastPwdChange: 20170608170021Z
> krbPasswordExpiration: 20170906170021Z
>
> Freeipa03:
>
> # dynamic-kepler, users, accounts, ipa.example.com
> dn: uid=dynamic-kepler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com
> uid: dynamic-kepler
> krbLastPwdChange: 20170608170011Z
> krbPasswordExpiration: 20170608170011Z
>
> Errors on Freeipa02:
>
> [08/Jun/2017:01:46:50.635529447 +0000] replica_generate_next_csn:
> opcsn=5938ac8b000500030000 <= basecsn=5938ac8b000500040000, adjusted
> opcsn=5938ac8b000600030000
> [08/Jun/2017:12:16:46.497249649 +0000] replica_generate_next_csn:
> opcsn=5939402f000500030000 <= basecsn=5939402f000800040000, adjusted
> opcsn=5939402f000900030000
> [08/Jun/2017:23:38:48.197750001 +0000] replica_generate_next_csn:
> opcsn=5939e009000100030000 <= basecsn=5939e009000f00040000, adjusted
> opcsn=5939e009001000030000
>
> The other nodes have no errors from this data.
>
> Access logs:
>
> Freeipa01:
>
> [08/Jun/2017:01:46:50.635529447 +0000] replica_generate_next_csn:
> opcsn=5938ac8b000500030000 <= basecsn=5938ac8b000500040000, adjusted
> opcsn=5938ac8b000600030000
> [08/Jun/2017:12:16:46.497249649 +0000] replica_generate_next_csn:
> opcsn=5939402f000500030000 <= basecsn=5939402f000800040000, adjusted
> opcsn=5939402f000900030000
> [08/Jun/2017:23:38:48.197750001 +0000] replica_generate_next_csn:
> opcsn=5939e009000100030000 <= basecsn=5939e009000f00040000, adjusted
> opcsn=5939e009001000030000
>
This is from an error log :-)
>
> Freeipa02:
>
> Shows no logs "to" the other 2 nodes.
>
Well it would only show incoming connections, not outgoing.
>
> Freeipa03:
>
> [08/Jun/2017:17:10:06.343697044 +0000] conn=9237 fd=70 slot=70
> connection from 192.168.0.12 to 192.168.0.13
> [08/Jun/2017:19:54:05.025713675 +0000] conn=9665 fd=70 slot=70
> connection from 192.168.0.12 to 192.168.0.13
>
> Freeipa02 replication logging:
>
> [09/Jun/2017:11:24:58.827281135 +0000] NSMMReplicationPlugin -
> csnplCommitALL: processing data csn 593964af000900030000
>
> Repeats 800 - 900 time per second with a different csn.
>
It looks like its replicating to other replicas, but some updates are
skipped. This again could be fractional replication "working".
If you look through freeipa01's access log what operation is this csn
from: 5937cccd000f00030000 ? Could this be one of the password
updates that is not replicated? This update is not sent to the other
replicas that's why I'm asking.
I cant find that csn anywhere but the error
log on freeipa02. Both
servers are logging csn's around the same time, just not this one.
Cheers
Nick
Thanks,
Mark
>
> On 08/06/17 15:45, Mark Reynolds wrote:
>>
>>
>> On 06/07/2017 10:58 AM, Nick Campion via FreeIPA-users wrote:
>>>
>>> Hi all,
>>>
>>>
>>>
>>> We have a 3 master setup that is failing to replicate changes from
>>> a particular node to the other IPA instances. The replication
>>> status says it's all fine, however the record hasn't been changed
>>> on the other servers. We've seen this on user password changes,
>>> adding hosts and services. The only thing we've found that seems to
>>> fix this temporarily is to re-initialize from the master with the
>>> changed record. A force-sync doesn't pick up the changed record.
>>>
>> What is the change you making, what attribute are you updating?
>> Could it be possible that its being excluded by fractional
>> replication? Or is it all changes?
>>
>> Any errors in the logs on the nodes(good and bad):
>> /var/log/dirsrv/slapd-INSTANCE/errors
>>
>> Do you see replication sessions starting between the bad node and
>> good ones? Are they talking? Check the access log (
>> /var/log/dirsrv/slapd-INSTANCE/access) on a good node and look for
>> "connection from <BAD NODE IP address>"
>>
>> Next would be to enable replication logging on the bad node and
>> reproduce the problem (then disable repl logging right away), then
>> send us the logs to look at. See
>>
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10...
>>
>> Regards,
>> Mark
>>
>>> Not sure what logs would be helpful to diagnose what is happening
>>> in this setup.
>>>
>>> # ipa-replica-manage -v list `hostname`
>>> freeipa03.mgmt.example.com: replica
>>> last init status: None
>>> last init ended: 1970-01-01 00:00:00+00:00
>>> last update status: Error (0) Replica acquired successfully:
>>> Incremental update succeeded
>>> last update ended: 2017-06-07 14:43:53+00:00
>>> freeipa02.mgmt.example.com: replica
>>> last init status: None
>>> last init ended: 1970-01-01 00:00:00+00:00
>>> last update status: Error (0) Replica acquired successfully:
>>> Incremental update succeeded
>>> last update ended: 2017-06-07 14:43:53+00:00
>>>
>>> # ldapsearch -W -x -D "cn=directory manager" -b
>>> "cn=users,cn=accounts,dc=ipa,dc=example,dc=com"
>>> "nsds5ReplConflict=*" \* nsds5ReplConflict
>>> Enter LDAP Password:
>>> # extended LDIF
>>> #
>>> # LDAPv3
>>> # base <cn=users,cn=accounts,dc=ipa,dc=example,dc=com> with scope
>>> subtree
>>> # filter: nsds5ReplConflict=*
>>> # requesting: * nsds5ReplConflict
>>> #
>>>
>>> # search result
>>> search: 2
>>> result: 0 Success
>>>
>>> # numResponses: 1
>>>
>>> Any help in what else can be checked or what logs would be helpful
>>> would be appreciated.
>>>
>>> Thanks
>>>
>>> Nick
>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>
>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org