We have a mutli-master configuration between two servers, ca-master1, and rep1.  It was discovered that there were some replication failures with some records.  We were instructed to clear these failed replication events by doing the following.


# ldapdelete -x -h localhost -D 'cn=Directory Manager' -W  'krbprincipalname=HTTP/ca-master1.ipa.xxx.org@IPA.XXX.ORG+nsuniqueid=024ed801-290c11eb-a80f9961-57f7bd5e,cn=services,cn=accounts,dc=ipa,dc=xxx,dc=org'
# ldapdelete -x -h localhost -D 'cn=Directory Manager' -W  'cn=ca-master1.ipa.xxx.org+nsuniqueid=f400bc09-290b11eb-a80f9961-57f7bd5e,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=xxx,dc=org'
# ldapdelete -x -h localhost -D 'cn=Directory Manager' -W  'cn=KDC+nsuniqueid=f400bc0e-290b11eb-a80f9961-57f7bd5e,cn=ca-master1.ipa.xxx.org,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=xxx,dc=org'
# ldapdelete -x -h localhost -D 'cn=Directory Manager' -W  'cn=KPASSWD+nsuniqueid=f400bc0f-290b11eb-a80f9961-57f7bd5e,cn=ca-master1.ipa.xxx.org,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=xxx,dc=org'
# ldapdelete -x -h localhost -D 'cn=Directory Manager' -W  'cn=HTTP+nsuniqueid=024ed802-290c11eb-a80f9961-57f7bd5e,cn=ca-master1.ipa.xxx.org,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=xxx,dc=org'
# ldapdelete -x -h localhost -D 'cn=Directory Manager' -W  'cn=OTPD+nsuniqueid=024ed803-290c11eb-a80f9961-57f7bd5e,cn=ca-master1.ipa.xxx.org,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=xxx,dc=org'
# ldapdelete -x -h localhost -D 'cn=Directory Manager' -W  'cn=KEYS+nsuniqueid=024ed804-290c11eb-a80f9961-57f7bd5e,cn=ca-master1.ipa.xxx.org,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=xxx,dc=org'
# ldapdelete -x -h localhost -D 'cn=Directory Manager' -W  'fqdn=oitidpnpdev02.xxx.org+nsuniqueid=10cf0001-a93e11eb-87aeb044-5694b0fb,cn=computers,cn=accounts,dc=ipa,dc=xxx,dc=org'

Unfortunately, right after performing the above actions, we noticed:


ca-master1:

[19/May/2021:15:27:59.825229655 -0500] - WARN - str2entry_dupcheck - Duplicate value for attribute type nisNetgroupTriple detected in entry cn=ir-nfs,cn=ng,cn=compat,dc=ipa,dc=xxx,dc=org. Extra value ignored.
[19/May/2021:15:27:59.861813312 -0500] - WARN - str2entry_dupcheck - Duplicate value for attribute type nisNetgroupTriple detected in entry cn=acc-hosts,cn=ng,cn=compat,dc=ipa,dc=xxx,dc=org. Extra value ignored.
[19/May/2021:15:27:59.899343450 -0500] - WARN - str2entry_dupcheck - Duplicate value for attribute type nisNetgroupTriple detected in entry cn=irunix,cn=ng,cn=compat,dc=ipa,dc=xxx,dc=org. Extra value ignored.
[19/May/2021:15:27:59.936539800 -0500] - WARN - str2entry_dupcheck - Duplicate value for attribute type nisNetgroupTriple detected in entry cn=nfs-hosts,cn=ng,cn=compat,dc=ipa,dc=xxx,dc=org. Extra value ignored.
[19/May/2021:15:27:59.983973594 -0500] - WARN - str2entry_dupcheck - Duplicate value for attribute type nisNetgroupTriple detected in entry cn=ir-nfs,cn=ng,cn=compat,dc=ipa,dc=xxx,dc=org. Extra value ignored.
[19/May/2021:15:28:00.020404656 -0500] - WARN - str2entry_dupcheck - Duplicate value for attribute type nisNetgroupTriple detected in entry cn=acc-hosts,cn=ng,cn=compat,dc=ipa,dc=xxx,dc=org. Extra value ignored.
[19/May/2021:18:32:35.566302057 -0500] - ERR - cos-plugin - cos_cache_entry_is_cos_related - Modified entry is NULL--updating cache just in case
[19/May/2021:18:32:35.570114071 -0500] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=ipa,dc=xxx,dc=org--no CoS Templates found, which should be added before the CoS Definition.
[19/May/2021:18:34:32.674161356 -0500] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=ca-master1.ipa.xxx.org-to-rep1.ipa.xxx.org" (rep1:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later.
[19/May/2021:18:34:35.681799169 -0500] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=ca-master1.ipa.xxx.org-to-rep1.ipa.xxx.org" (rep1:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later.
[19/May/2021:18:34:41.689490330 -0500] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=ca-master1.ipa.xxx.org-to-rep1.ipa.xxx.org" (rep1:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later.
[19/May/2021:18:34:53.711905379 -0500] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=ca-master1.ipa.xxx.org-to-rep1.ipa.xxx.org" (rep1:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later.
[19/May/2021:18:35:17.719796394 -0500] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=ca-master1.ipa.xxx.org-to-rep1.ipa.xxx.org" (rep1:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later.


rep1:

[19/May/2021:15:28:39.324345375 -0500] - WARN - str2entry_dupcheck - Duplicate value for attribute type nisNetgroupTriple detected in entry cn=iamshibnonprod,cn=ng,cn=compat,dc=ipa,dc=xxx,dc=org. Extra value ignored.
[19/May/2021:15:28:39.331127354 -0500] - WARN - str2entry_dupcheck - Duplicate value for attribute type nisNetgroupTriple detected in entry cn=iamshibnonprod,cn=ng,cn=compat,dc=ipa,dc=xxx,dc=org. Extra value ignored.
[19/May/2021:18:32:35.294020328 -0500] - ERR - ipa-topology-plugin - ipa_topo_util_modify: failed to modify entry (cn=replica,cn=dc\3Dipa\2Cdc\3Dxxx\2Cdc\3Dorg,cn=mapping tree,cn=config): error 16
[19/May/2021:18:32:35.310416977 -0500] - ERR - ipa-topology-plugin - ipa_topo_agmt_del: cn=rep1.ipa.xxx.org-to-ca-master1.ipa.xxx.org
[19/May/2021:18:32:36.592724176 -0500] - ERR - ipa-topology-plugin - ipa_topo_agmt_del: cn=rep1.ipa.xxx.org-to-ca-master1.ipa.xxx.org
[19/May/2021:18:32:36.705929392 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Initiating CleanAllRUV Task...
[19/May/2021:18:32:36.706688135 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Retrieving maxcsn...
[19/May/2021:18:32:36.707404944 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Found maxcsn (60a574f60003000b0000)
[19/May/2021:18:32:36.717663920 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Initiating CleanAllRUV Task...
[19/May/2021:18:32:36.718617493 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Retrieving maxcsn...
[19/May/2021:18:32:36.721014950 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Found maxcsn (60a59d050000000c0000)
[19/May/2021:18:32:36.739433329 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Cleaning rid (12)...
[19/May/2021:18:32:36.742689695 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Waiting to process all the updates from the deleted replica...
[19/May/2021:18:32:36.744353200 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Waiting for all the replicas to be online...
[19/May/2021:18:32:36.746343965 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Waiting for all the replicas to receive all the deleted replica updates...
[19/May/2021:18:32:36.748234565 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Sending cleanAllRUV task to all the replicas...
[19/May/2021:18:32:36.750036702 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Cleaning local ruv's...
[19/May/2021:18:32:37.720908584 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Cleaning rid (11)...
[19/May/2021:18:32:37.723446365 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Waiting to process all the updates from the deleted replica...
[19/May/2021:18:32:37.725861245 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Waiting for all the replicas to be online...
[19/May/2021:18:32:37.728834914 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Waiting for all the replicas to receive all the deleted replica updates...
[19/May/2021:18:32:37.731854881 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Sending cleanAllRUV task to all the replicas...
[19/May/2021:18:32:37.733648087 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Cleaning local ruv's...
[19/May/2021:18:32:37.766056624 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Waiting for all the replicas to be cleaned...
[19/May/2021:18:32:37.776216157 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Waiting for all the replicas to finish cleaning...
[19/May/2021:18:32:37.778075680 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Original task deletes Keep alive entry (12).
[19/May/2021:18:32:37.830528541 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): No Keep-Alive entry to remove (cn=repl keep alive 12,o=ipaca)
[19/May/2021:18:32:37.833448028 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 12): Successfully cleaned rid(12)
[19/May/2021:18:32:38.746020775 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Waiting for all the replicas to be cleaned...
[19/May/2021:18:32:38.756663129 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Waiting for all the replicas to finish cleaning...
[19/May/2021:18:32:38.758723096 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Original task deletes Keep alive entry (11).
[19/May/2021:18:32:38.766944696 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Removed Keep-Alive entry (cn=repl keep alive 11,dc=ipa,dc=xxx,dc=org)
[19/May/2021:18:32:38.768635297 -0500] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 11): Successfully cleaned rid(11)
[19/May/2021:18:34:32.668761455 -0500] - ERR - NSMMReplicationPlugin - multimaster_extop_StartNSDS50ReplicationRequest - conn=6307648 op=5 replica="o=ipaca": Unable to acquire replica: error: permission denied
[19/May/2021:18:34:35.676187613 -0500] - ERR - NSMMReplicationPlugin - multimaster_extop_StartNSDS50ReplicationRequest - conn=6307648 op=6 replica="o=ipaca": Unable to acquire replica: error: permission denied
[19/May/2021:18:34:41.684145521 -0500] - ERR - NSMMReplicationPlugin - multimaster_extop_StartNSDS50ReplicationRequest - conn=6307648 op=7 replica="o=ipaca": Unable to acquire replica: error: permission denied
[19/May/2021:18:34:53.699403689 -0500] - ERR - NSMMReplicationPlugin - multimaster_extop_StartNSDS50ReplicationRequest - conn=6307648 op=8 replica="o=ipaca": Unable to acquire replica: error: permission denied


Now these two servers cannot sync anything, and are becoming disjoint.

Any tips on how to rectify this?

I'm getting the sinking feeling that I'll need to do the following:


Is there perhaps a simpler trick to fix things?

Amos