I am on a long journey upgrading from a mixture of different versions of freeipa to the lastest and i am almost there, currently on 4.10. After removing some older replicas yesterday, a new replica had a fault and we had to restart dirsrv. When it came back up we saw we have ghost replicas across the whole estate 6x replicas in two locations. Issues are reported via `cipa`
Digging into the tombstone records, i find there are not the usual tombstone replica records but more like the below.
My question is, how do i remove these redundant records? As I believe this is what `cipa` is counting to report the errors. (marked with <<<<<<<<<<<<<<)
The usual things do not work:
# ipa-replica-manage -p $pass clean-ruv 15 Replica ID 15 not found
$ cat remove_ruv.ldif dn: cn=clean 15,cn=cleanallruv,cn=tasks,cn=config changetype: add objectclass: top objectclass: extensibleObject replica-base-dn: dc=ad,dc=companyx,dc=fm replica-id: 15 cn: clean 15
ldapmodify -Y GSSAPI -f remove_ruv.ldif
Thanks in advance. Nick
$ ldapsearch -D "cn=Directory Manager" -w $pass -Q -o ldif-wrap=no -LLL -b "dc=ad,dc=companyx,dc=fm" '(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))' dn: cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindDNGroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=ad,dc=companyx,dc=fm nsDS5ReplicaBindDnGroupCheckInterval: 60 nsDS5ReplicaId: 56 nsDS5ReplicaName: a6b5640c-ad3911ed-a50980fb-6203228c nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm nsDS5ReplicaType: 3 nsState:: OAAAAAAAAABSPEJkAAAAAAAAAAAAAAAA7AAAAAAAAAAGAAAAAAAAAA== nsds5ReplicaBackoffMax: 300 nsds5ReplicaLegacyConsumer: off nsds5ReplicaReleaseTimeout: 60 objectClass: top objectClass: nsds5replica objectClass: extensibleobject nsds50ruv: {replicageneration} 5d9e2076000000040000 nsds50ruv: {replica 56 ldap://ipa006.ad.companyx.fm:389} 63ece66f000000380000 64423d4e000000380000 nsds50ruv: {replica 46 ldap://ipa005.ad.companyx.fm:389} 63dbcc200001002e0000 64423cf6000f002e0000 nsds50ruv: {replica 21 ldap://etcd0dc.ad.companyx.fm:389} 5f2d4bc1000000150000 64319ec3276200150000 nsds50ruv: {replica 48 ldap://ipa007.ad.companyx.fm:389} 63ea4e54000100300000 6442397e000a00300000 nsds50ruv: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 643d03280001003a0000 644232c00001003a0000 nsds50ruv: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 643d19680001003c0000 644232660004003c0000 nsds50ruv: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 643d491e0001003e0000 644233340000003e0000 nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm; ipa006.ad.companyx.fm-to-ipa003dc.ad.companyx.fm;ipa003dc.ad.companyx.fm ;389;62;644238c9000000380000 nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm; ipa006.ad.companyx.fm-to-ipa007.ad.companyx.fm;ipa007.ad.companyx.fm ;389;48;644238c9000000380000 nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm; ipa006.ad.companyx.fm-to-etcd2dc.ad.companyx.fm;etcd2dc.ad.companyx.fm;389;unavailable;64423cf6000f002e0000 <<<<<<<<<<<<<< nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm; ipa006.ad.companyx.fm-to-ipa005.ad.companyx.fm;ipa005.ad.companyx.fm ;389;46;644238c9000000380000 nsruvReplicaLastModified: {replica 56 ldap://ipa006.ad.companyx.fm:389} 64423c62 nsruvReplicaLastModified: {replica 46 ldap://ipa005.ad.companyx.fm:389} 64423c0c nsruvReplicaLastModified: {replica 21 ldap://etcd0dc.ad.companyx.fm:389} 00000000 nsruvReplicaLastModified: {replica 48 ldap://ipa007.ad.companyx.fm:389} 64423894 nsruvReplicaLastModified: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 644231da nsruvReplicaLastModified: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 6442317a nsruvReplicaLastModified: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 64423249 nsruvReplicaLastModified: {replica 12} 644180f7 <<<<<<<<<<<<<< nsruvReplicaLastModified: {replica 15} 644180f7 <<<<<<<<<<<<<< nsruvReplicaLastModified: {replica 25} 644180f7 <<<<<<<<<<<<<< nsruvReplicaLastModified: {replica 23} 644180f7 <<<<<<<<<<<<<< nsruvReplicaLastModified: {replica 40} 644180f7 <<<<<<<<<<<<<< nsds5ReplicaChangeCount: 734077 nsds5replicareapactive: 0
I think i have a handle on this now.
There are a number of issues that i am now aware of.
1. old replication agreement to oldbox1 on newbox6
2. corrupt RUVs, giving the impression of Ghost Replicas.
For #1 i normally can delete these fine with a ldap command. BUT! running this crashes the dirsrv service.
ldapdelete -D "cn=Directory Manager" -w $pwd -p 389 -h localhost -x "cn=newbox6.ad.dice.fm-to-oldbox1.ad.dice.fm,cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping tree,cn=config"
Apr 21 11:35:48 newbox6.ad.dice.fm systemd[1]: Starting 389 Directory Server AD-DICE-FM.... Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv@AD-DICE-FM.service: Failed with result 'signal'. Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv@AD-DICE-FM.service: Main process exited, code=killed, status=6/ABRT Apr 21 11:35:11 newbox6.ad.dice.fm ns-slapd[2934110]: ns-slapd: ldap/servers/plugins/sync/sync_persist.c:234: sync_update_persist_op: Assertion `prim_op' failed.
For #2 , this issue of not being able to remove the replication agreement, stops the removal of the corrupt RUVs.
As you can see i have tried to kick off some RUV removals but they are failing as not all replicas are online. (but they dont exist as #1)
$ ipa-replica-manage list-clean-ruv -p $pass ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission denied: '/var/log/ipa/cli.log' CLEANALLRUV tasks RID 12: Not all replicas online, retrying in 40 seconds... RID restarted-2658134: Not all replicas online, retrying in 320 seconds... RID restarted-2658136: Not all replicas online, retrying in 320 seconds...
No abort CLEANALLRUV tasks running
So, more questions really, Why is the ldapdelete crashing the service? How do i fix it? thanks, Nick
Nicholas Cross via FreeIPA-users wrote:
I think i have a handle on this now.
There are a number of issues that i am now aware of.
old replication agreement to oldbox1 on newbox6
corrupt RUVs, giving the impression of Ghost Replicas.
For #1 i normally can delete these fine with a ldap command. BUT! running this crashes the dirsrv service.
ldapdelete -D "cn=Directory Manager" -w $pwd -p 389 -h localhost -x "cn=newbox6.ad.dice.fm-to-oldbox1.ad.dice.fm,cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping tree,cn=config"
Apr 21 11:35:48 newbox6.ad.dice.fm systemd[1]: Starting 389 Directory Server AD-DICE-FM.... Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv@AD-DICE-FM.service: Failed with result 'signal'. Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv@AD-DICE-FM.service: Main process exited, code=killed, status=6/ABRT Apr 21 11:35:11 newbox6.ad.dice.fm ns-slapd[2934110]: ns-slapd: ldap/servers/plugins/sync/sync_persist.c:234: sync_update_persist_op: Assertion `prim_op' failed.
For #2 , this issue of not being able to remove the replication agreement, stops the removal of the corrupt RUVs.
As you can see i have tried to kick off some RUV removals but they are failing as not all replicas are online. (but they dont exist as #1)
$ ipa-replica-manage list-clean-ruv -p $pass ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission denied: '/var/log/ipa/cli.log' CLEANALLRUV tasks RID 12: Not all replicas online, retrying in 40 seconds... RID restarted-2658134: Not all replicas online, retrying in 320 seconds... RID restarted-2658136: Not all replicas online, retrying in 320 seconds...
No abort CLEANALLRUV tasks running
So, more questions really, Why is the ldapdelete crashing the service?
Can you install the debuginfo packages and get a stack trace? I'm sure the 389-ds developers would like to take a look at it. It may already have been addressed. You haven't provided any version or distro information.
How do i fix it?
If you shut down dirsrv then you can edit /etc/dirsrv/slapd-REALM/dse.ldif and manually remove the replication agreement.
But you have to shut down dirsrv first.
rob
Thanks for the tips. I'll run the trace on Monday.
Also i'll edit the ldif Monday too.
Distro: alma9
# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64
Shutdown dirsrv backed up dse.ldif edited the ldif restarted dirsrv
replication agreement came back!
Is it being sync'ed from some where else? another file?
Nicholas Cross via FreeIPA-users wrote:
Shutdown dirsrv backed up dse.ldif edited the ldif restarted dirsrv
replication agreement came back!
Is it being sync'ed from some where else? another file?
It sounds like dirsrv was still running if values in cn=config were restored. It writes the file on exit. This subtree is not replicated.
rob
Tested this again making sure that dirsrv is not running and the replica record is back.
I am obviously doing something wrong. My steps are below. I appreciate your time on this.
# # check dirsrv is currently running # [root@ipa006 ~]# ps aux | grep dirsrv dirsrv 3221639 31.4 5.4 2418488 883856 ? Ssl Apr24 322:04 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-AD-companyx-FM -i /run/dirsrv/slapd-AD-companyx-FM.pid root 3281205 0.0 0.0 6412 2204 pts/2 S+ 09:11 0:00 grep --color=auto dirsrv
# # shutdown dirsrv # [root@ipa006 ~]# time systemctl stop dirsrv@AD-companyx-FM.service
real 10m0.130s user 0m0.009s sys 0m0.012s
# # check dirsrv is not running 1 # [root@ipa006 ~]# ps aux | grep dirsrv root 3282962 0.0 0.0 6412 2244 pts/2 S+ 09:47 0:00 grep --color=auto dirsrv
# # check dirsrv is not running 2 # [root@ipa006 slapd-AD-companyx-FM]# ipactl status Directory Service: STOPPED krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING 1 service(s) are not running
# # go to right folder # [root@ipa006 ~]# cd /etc/dirsrv/slapd-AD-companyx-FM/
# # make a backup just incase # [root@ipa006 slapd-AD-companyx-FM]# cp dse.ldif dse.ldif.nickx-25apr23
# # edit ldif # [root@ipa006 slapd-AD-companyx-FM]# vi dse.ldif
# # remove this record. Hoping its the right thing to do. # dn: cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi ce\2Cdc\3Dfm,cn=mapping tree,cn=config objectClass: nsds5replicationagreement objectClass: ipaReplTopoManagedAgreement objectClass: top cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm nsDS5ReplicaPort: 389 nsds5replicaTimeout: 300 nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm ipaReplTopoManagedAgreementState: managed agreement - generated by topology pl ugin nsDS5ReplicaTransportInfo: LDAP nsDS5ReplicaBindMethod: SASL/GSSAPI nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in ternalModifyTimestamp nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts uccessfulauth krblastfailedauth krbloginfailedcount creatorsName: cn=IPA Topology Configuration,cn=plugins,cn=config modifiersName: cn=IPA Topology Configuration,cn=plugins,cn=config createTimestamp: 20230425095140Z modifyTimestamp: 20230425095140Z
# # check no records exist in dse.ldif # [root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif [root@ipa006 slapd-AD-companyx-FM]#
[root@ipa006 slapd-AD-companyx-FM]# time systemctl start dirsrv@AD-companyx-FM.service
real 0m12.343s user 0m0.006s sys 0m0.007s
# # Look in logs # Apr 25 09:51:51 ipa006.ad.companyx.fm ns-slapd[3283119]: [25/Apr/2023:09:51:51.484197325 +0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm" (bad_serverdc:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
# # check dse.ldif again - find entry is back ! # [root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif dn: cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm
# # scratch head and ponder life, the universe and everything #
Nicholas Cross via FreeIPA-users wrote:
Tested this again making sure that dirsrv is not running and the replica record is back.
I am obviously doing something wrong. My steps are below. I appreciate your time on this.
# # check dirsrv is currently running # [root@ipa006 ~]# ps aux | grep dirsrv dirsrv 3221639 31.4 5.4 2418488 883856 ? Ssl Apr24 322:04 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-AD-companyx-FM -i /run/dirsrv/slapd-AD-companyx-FM.pid root 3281205 0.0 0.0 6412 2204 pts/2 S+ 09:11 0:00 grep --color=auto dirsrv
# # shutdown dirsrv # [root@ipa006 ~]# time systemctl stop dirsrv@AD-companyx-FM.service
real 10m0.130s user 0m0.009s sys 0m0.012s
# # check dirsrv is not running 1 # [root@ipa006 ~]# ps aux | grep dirsrv root 3282962 0.0 0.0 6412 2244 pts/2 S+ 09:47 0:00 grep --color=auto dirsrv
# # check dirsrv is not running 2 # [root@ipa006 slapd-AD-companyx-FM]# ipactl status Directory Service: STOPPED krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING 1 service(s) are not running
# # go to right folder # [root@ipa006 ~]# cd /etc/dirsrv/slapd-AD-companyx-FM/
# # make a backup just incase # [root@ipa006 slapd-AD-companyx-FM]# cp dse.ldif dse.ldif.nickx-25apr23
# # edit ldif # [root@ipa006 slapd-AD-companyx-FM]# vi dse.ldif
# # remove this record. Hoping its the right thing to do. # dn: cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi ce\2Cdc\3Dfm,cn=mapping tree,cn=config objectClass: nsds5replicationagreement objectClass: ipaReplTopoManagedAgreement objectClass: top cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm nsDS5ReplicaPort: 389 nsds5replicaTimeout: 300 nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm ipaReplTopoManagedAgreementState: managed agreement - generated by topology pl ugin nsDS5ReplicaTransportInfo: LDAP nsDS5ReplicaBindMethod: SASL/GSSAPI nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in ternalModifyTimestamp nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts uccessfulauth krblastfailedauth krbloginfailedcount creatorsName: cn=IPA Topology Configuration,cn=plugins,cn=config modifiersName: cn=IPA Topology Configuration,cn=plugins,cn=config createTimestamp: 20230425095140Z modifyTimestamp: 20230425095140Z
# # check no records exist in dse.ldif # [root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif [root@ipa006 slapd-AD-companyx-FM]#
[root@ipa006 slapd-AD-companyx-FM]# time systemctl start dirsrv@AD-companyx-FM.service
real 0m12.343s user 0m0.006s sys 0m0.007s
# # Look in logs # Apr 25 09:51:51 ipa006.ad.companyx.fm ns-slapd[3283119]: [25/Apr/2023:09:51:51.484197325 +0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm" (bad_serverdc:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
# # check dse.ldif again - find entry is back ! # [root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif dn: cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm
# # scratch head and ponder life, the universe and everything #
I have a guess what is happening. The topology is also stored within IPA itself. I wonder if this is self-healing.
I assume you've ensure that the bad server is not anywhere else inside of IPA? ipa server-del bad_server? Does its host entry exist?
ipa-replica-manage list -v `hostname` on the server the agreement exists?
rob
Ah got it! Wonderful.
The trick as to run the topologysegement-del on the same server it was on.
It seems i am moving forward with this now - thanks.
# # To remove the topology segment, which removed the replica agreement #
# # Show the bad replication agreement #
# ipa-replica-manage list -v `hostname` Directory Manager password:
bad_server.ad.companyx.fm: replica last update status: Error (-1) Problem connecting to replica - LDAP error: Can't contact LDAP server (connection error) last update ended: 1970-01-01 00:00:00+00:00 ipa003dc.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2023-04-26 06:43:07+00:00 ipa005.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2023-04-26 06:43:14+00:00 ipa007.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2023-04-26 06:43:02+00:00
# # find the segment (domain or ca) # $ ipa topologysegment-find domain | grep etcd Segment name: ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm Right node: bad_server.ad.companyx.fm
# # delete that segment # $ ipa topologysegment-del domain ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm --------------------------------------------------------- Deleted segment "ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm" ---------------------------------------------------------
# # check it has gone - tada! # $ ipa-replica-manage list -v `hostname` ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission denied: '/var/log/ipa/cli.log' ipa003dc.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update started last update ended: 1970-01-01 00:00:00+00:00 ipa005.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update started last update ended: 1970-01-01 00:00:00+00:00 ipa007.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2023-04-26 06:46:08+00:00
# # Next up, removing the "LDAP Conflicts" but - "Removal of Segment disconnects topology.Deletion not allowed." #
$ ldapdelete cn=bad_server.ad.companyx.fm-to-ipa006.ad.companyx.fm+nsuniqueid=34b26c01-ceee11ed-9d1d82de-03f3a8a3,cn=ca,cn=topology,cn=ipa,cn=etc,dc=ad,dc=companyx,dc=fm SASL/GSSAPI authentication started SASL username: nicholas.cross@AD.companyx.FM SASL SSF: 256 SASL data security layer installed. ldap_delete: Server is unwilling to perform (53) additional info: Removal of Segment disconnects topology.Deletion not allowed. # # I think this is the solution: https://access.redhat.com/solutions/5507711 # # Question1: during running the above RedHat solution, does this only disable the topology replication? and leaves all other dirsrv components running? #
# # After that - finally remove the Ghost Replicas - which was the original question. #
$ ldapsearch -D "cn=Directory Manager" -w $pass -Q -o ldif-wrap=no -LLL -b "dc=ad,dc=companyx,dc=fm" '(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))' dn: cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindDNGroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=ad,dc=companyx,dc=fm nsDS5ReplicaBindDnGroupCheckInterval: 60 nsDS5ReplicaId: 56 nsDS5ReplicaName: a6b5640c-ad3911ed-a50980fb-6203228c nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm nsDS5ReplicaType: 3 nsState:: OAAAAAAAAABf0EhkAAAAAAAAAAAAAAAA7AAAAAAAAAAFAAAAAAAAAA== nsds5ReplicaBackoffMax: 300 nsds5ReplicaLegacyConsumer: off nsds5ReplicaReleaseTimeout: 60 objectClass: top objectClass: nsds5replica objectClass: extensibleobject nsds5ReplicaCleanRUV: 15:no:0:dc=ad,dc=companyx,dc=fm nsds5ReplicaCleanRUV: 24:no:0:dc=ad,dc=companyx,dc=fm nsds50ruv: {replicageneration} 5d9e2076000000040000 nsds50ruv: {replica 56 ldap://ipa006.ad.companyx.fm:389} 63ece66f000000380000 6448d15d000400380000 nsds50ruv: {replica 46 ldap://ipa005.ad.companyx.fm:389} 63dbcc200001002e0000 6448d115000e002e0000 nsds50ruv: {replica 48 ldap://ipa007.ad.companyx.fm:389} 63ea4e54000100300000 6448d115000700300000 nsds50ruv: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 643d03280001003a0000 6448ca410000003a0000 nsds50ruv: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 643d19680001003c0000 6448c9e40009003c0000 nsds50ruv: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 643d491e0001003e0000 6448cab40000003e0000 nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa003dc.ad.companyx.fm;ipa003dc.ad.companyx.fm;389;62;6448cf8e000800380000 nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa005.ad.companyx.fm;ipa005.ad.companyx.fm;389;46;6448cf8e000800380000 nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa007.ad.companyx.fm;ipa007.ad.companyx.fm;389;48;6448cf8e000800380000 nsruvReplicaLastModified: {replica 56 ldap://ipa006.ad.companyx.fm:389} 6448d071 nsruvReplicaLastModified: {replica 46 ldap://ipa005.ad.companyx.fm:389} 6448d02b nsruvReplicaLastModified: {replica 48 ldap://ipa007.ad.companyx.fm:389} 6448d02b nsruvReplicaLastModified: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 6448c956 nsruvReplicaLastModified: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 6448c8fb nsruvReplicaLastModified: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 6448c9c9 nsruvReplicaLastModified: {replica 25} 00000000 nsruvReplicaLastModified: {replica 23} 00000000 nsruvReplicaLastModified: {replica 40} 00000000 nsruvReplicaLastModified: {replica 12} 00000000 nsruvReplicaLastModified: {replica 21} 00000000 nsds5ReplicaChangeCount: 790081 nsds5replicareapactive: 0
# # Question2: How to remove these? from the above #
nsruvReplicaLastModified: {replica 25} 00000000 nsruvReplicaLastModified: {replica 23} 00000000 nsruvReplicaLastModified: {replica 40} 00000000 nsruvReplicaLastModified: {replica 12} 00000000 nsruvReplicaLastModified: {replica 21} 00000000
# this sort of thing doesn't seem to work.
dn: cn=clean 12,cn=cleanallruv,cn=tasks,cn=config changetype: add objectclass: top objectclass: extensibleObject replica-base-dn: dc=ad,dc=companyx,dc=fm replica-id: 12 cn: clean 12
Many thanks.
Nicholas Cross via FreeIPA-users wrote:
Ah got it! Wonderful.
The trick as to run the topologysegement-del on the same server it was on.
It seems i am moving forward with this now - thanks.
# # To remove the topology segment, which removed the replica agreement #
# # Show the bad replication agreement #
# ipa-replica-manage list -v `hostname` Directory Manager password:
bad_server.ad.companyx.fm: replica last update status: Error (-1) Problem connecting to replica - LDAP error: Can't contact LDAP server (connection error) last update ended: 1970-01-01 00:00:00+00:00 ipa003dc.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2023-04-26 06:43:07+00:00 ipa005.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2023-04-26 06:43:14+00:00 ipa007.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2023-04-26 06:43:02+00:00
# # find the segment (domain or ca) # $ ipa topologysegment-find domain | grep etcd Segment name: ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm Right node: bad_server.ad.companyx.fm
# # delete that segment # $ ipa topologysegment-del domain ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm
Deleted segment "ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm"
# # check it has gone - tada! # $ ipa-replica-manage list -v `hostname` ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission denied: '/var/log/ipa/cli.log' ipa003dc.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update started last update ended: 1970-01-01 00:00:00+00:00 ipa005.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update started last update ended: 1970-01-01 00:00:00+00:00 ipa007.ad.companyx.fm: replica last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2023-04-26 06:46:08+00:00
# # Next up, removing the "LDAP Conflicts" but - "Removal of Segment disconnects topology.Deletion not allowed." #
$ ldapdelete cn=bad_server.ad.companyx.fm-to-ipa006.ad.companyx.fm+nsuniqueid=34b26c01-ceee11ed-9d1d82de-03f3a8a3,cn=ca,cn=topology,cn=ipa,cn=etc,dc=ad,dc=companyx,dc=fm SASL/GSSAPI authentication started SASL username: nicholas.cross@AD.companyx.FM SASL SSF: 256 SASL data security layer installed. ldap_delete: Server is unwilling to perform (53) additional info: Removal of Segment disconnects topology.Deletion not allowed.
# # I think this is the solution: https://access.redhat.com/solutions/5507711 # # Question1: during running the above RedHat solution, does this only disable the topology replication? and leaves all other dirsrv components running? #
# # After that - finally remove the Ghost Replicas - which was the original question. #
$ ldapsearch -D "cn=Directory Manager" -w $pass -Q -o ldif-wrap=no -LLL -b "dc=ad,dc=companyx,dc=fm" '(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))' dn: cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindDNGroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=ad,dc=companyx,dc=fm nsDS5ReplicaBindDnGroupCheckInterval: 60 nsDS5ReplicaId: 56 nsDS5ReplicaName: a6b5640c-ad3911ed-a50980fb-6203228c nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm nsDS5ReplicaType: 3 nsState:: OAAAAAAAAABf0EhkAAAAAAAAAAAAAAAA7AAAAAAAAAAFAAAAAAAAAA== nsds5ReplicaBackoffMax: 300 nsds5ReplicaLegacyConsumer: off nsds5ReplicaReleaseTimeout: 60 objectClass: top objectClass: nsds5replica objectClass: extensibleobject nsds5ReplicaCleanRUV: 15:no:0:dc=ad,dc=companyx,dc=fm nsds5ReplicaCleanRUV: 24:no:0:dc=ad,dc=companyx,dc=fm nsds50ruv: {replicageneration} 5d9e2076000000040000 nsds50ruv: {replica 56 ldap://ipa006.ad.companyx.fm:389} 63ece66f000000380000 6448d15d000400380000 nsds50ruv: {replica 46 ldap://ipa005.ad.companyx.fm:389} 63dbcc200001002e0000 6448d115000e002e0000 nsds50ruv: {replica 48 ldap://ipa007.ad.companyx.fm:389} 63ea4e54000100300000 6448d115000700300000 nsds50ruv: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 643d03280001003a0000 6448ca410000003a0000 nsds50ruv: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 643d19680001003c0000 6448c9e40009003c0000 nsds50ruv: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 643d491e0001003e0000 6448cab40000003e0000 nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa003dc.ad.companyx.fm;ipa003dc.ad.companyx.fm;389;62;6448cf8e000800380000 nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa005.ad.companyx.fm;ipa005.ad.companyx.fm;389;46;6448cf8e000800380000 nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa007.ad.companyx.fm;ipa007.ad.companyx.fm;389;48;6448cf8e000800380000 nsruvReplicaLastModified: {replica 56 ldap://ipa006.ad.companyx.fm:389} 6448d071 nsruvReplicaLastModified: {replica 46 ldap://ipa005.ad.companyx.fm:389} 6448d02b nsruvReplicaLastModified: {replica 48 ldap://ipa007.ad.companyx.fm:389} 6448d02b nsruvReplicaLastModified: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 6448c956 nsruvReplicaLastModified: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 6448c8fb nsruvReplicaLastModified: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 6448c9c9 nsruvReplicaLastModified: {replica 25} 00000000 nsruvReplicaLastModified: {replica 23} 00000000 nsruvReplicaLastModified: {replica 40} 00000000 nsruvReplicaLastModified: {replica 12} 00000000 nsruvReplicaLastModified: {replica 21} 00000000 nsds5ReplicaChangeCount: 790081 nsds5replicareapactive: 0
# # Question2: How to remove these? from the above #
nsruvReplicaLastModified: {replica 25} 00000000 nsruvReplicaLastModified: {replica 23} 00000000 nsruvReplicaLastModified: {replica 40} 00000000 nsruvReplicaLastModified: {replica 12} 00000000 nsruvReplicaLastModified: {replica 21} 00000000
# this sort of thing doesn't seem to work.
dn: cn=clean 12,cn=cleanallruv,cn=tasks,cn=config changetype: add objectclass: top objectclass: extensibleObject replica-base-dn: dc=ad,dc=companyx,dc=fm replica-id: 12 cn: clean 12
You can try ipa-replica-manage clean-ruv <value> to try to remove specific values.
rob
Just an update on this.
Came back from the long weekend and 50% of our servers (3) were not responding, the dirsrv was crashing everytime it had an update from the CA master (we could not figure out why). If we closed the firewall between replica and CA master the servers stayed up.
After a few days of trying various things to resurrect the down servers we rebuilt the whole cluster based off the master CA server. None of the original servers are now present.
After another long weekend we seem (so far) to have a stable cluster. Ignoring the usual replication conflicts we get with heavy server creation/deletion due to AWS spot instances.
The only out standing item now is the records that make "cipa" think we have "ghost replicas"
nsruvReplicaLastModified: {replica 25} 00000000 nsruvReplicaLastModified: {replica 23} 00000000 nsruvReplicaLastModified: {replica 40} 00000000 nsruvReplicaLastModified: {replica 21} 00000000
There are no RUVs to match these replicas (21,23, 25, 40). So it looks like these key/value pairs are the only things left.
Any ideas on how to remove them?
Many thanks.
Howdy folks,
We also have a similar issue. Some servers in our IPA topology show ghost replicas and if comes down to an entry like the following for an old replica which no longer exists
$ ldapsearch -xLLL -D "cn=directory manager" -W -b dc=DICOMP,dc=NET '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' Enter LDAP Password: dn: cn=replica,cn=dc\3Ddicomp\2Cdc\3Dnet,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindDNGroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=dicomp,dc=net nsDS5ReplicaBindDnGroupCheckInterval: 60 nsDS5ReplicaId: 11 nsDS5ReplicaName: 13387f82-373b11eb-a1r2gff0-4sda870 nsDS5ReplicaRoot: dc=dicomp,dc=net nsDS5ReplicaType: 3 nsState:: CwAAAAAAAABzzalmAAAAAAAAAAAAAAAAUpEAAAAAAAALAAAAAAAAAA== nsds5ReplicaBackoffMax: 300 nsds5ReplicaLegacyConsumer: off nsds5ReplicaReleaseTimeout: 60 objectClass: top objectClass: nsds5replica objectClass: extensibleobject nsds50ruv: {replicageneration} 5fc9ab2e000000040000 nsds50ruv: {replica 11 ldap://camper26.dicomp.net:389} 5fcbf1fa0000000b0000 66aa5 edc0000000b0000 nsds50ruv: {replica 3 ldap://camper21.dicomp.net:389} 5fc9ab34000000030000 66aa53c e000100030000 nsds50ruv: {replica 5 ldap://camper23.dicomp.net:389} 5fc9b44b000000050000 66aa58 d0000000050000 nsds50ruv: {replica 10 ldap://camper24.dicomp.net:389} 5fc9c7650000000a0000 66aa5 3d10004000a0000 nsds50ruv: {replica 33 ldap://ipa.dicomp.net:389} 626998ac000100210000 66aa5af1 000100210000 nsds50ruv: {replica 45 ldap://az1-iparepl-01.dicomp.net:389} 629644dc0001002d00 00 66aa58960000002d0000 nsds50ruv: {replica 46 ldap://au1-compca-01.dicomp.net:389} 6297aca50002002e0000 66aa59130003002e0000 nsds50ruv: {replica 48 ldap://nz1-freeipa-backup.dicomp.net:389} 62c8635e000200 300000 66aa4991000800300000 nsds50ruv: {replica 56 ldap://in1-iparepl-01.dicomp.net:389} 667aa1b90001003800 00 66aa553d000000380000 nsds50ruv: {replica 57 ldap://camper27.dicomp.net:389} 667bac3f000100390000 66aa5 547000000390000 nsds50ruv: {replica 60 ldap://camper25.dicomp.net:389} 667cf5c50000003c0000 66aa5a e00000003c0000 nsds50ruv: {replica 63 ldap://camper22.dicomp.net:389} 667d3ec50001003f0000 66aa 5d720000003f0000 nsds50ruv: {replica 64 ldap://nz1-compca-01.dicomp.net:389} 668e3565000100400000 66aa5d7e000000400000 nsds5agmtmaxcsn: dc=dicomp,dc=net;camper26.dicomp.net-to-camper27.dicomp.net;camper27.dicomp.net;389;57;66aa55c00000000b0000 nsds5agmtmaxcsn: dc=dicomp,dc=net;camper26.dicomp.net-to-in1-iparepl-01.dicomp.net; in1-iparepl-01.dicomp.net;389;56;66aa55c00000000b0000 nsruvReplicaLastModified: {replica 11 ldap://camper26.dicomp.net:389} 66a9cd8a nsruvReplicaLastModified: {replica 3 ldap://camper21.dicomp.net:389} 66a9c27f nsruvReplicaLastModified: {replica 5 ldap://camper23.dicomp.net:389} 66a9c780 nsruvReplicaLastModified: {replica 10 ldap://camper24.dicomp.net:389} 66a9c281 nsruvReplicaLastModified: {replica 33 ldap://ipa.dicomp.net:389} 66a9c9a4 nsruvReplicaLastModified: {replica 45 ldap://az1-iparepl-01.dicomp.net:389} 66a 9c745 nsruvReplicaLastModified: {replica 46 ldap://au1-compca-01.dicomp.net:389} 66a9c 7c5 nsruvReplicaLastModified: {replica 48 ldap://nz1-freeipa-backup.dicomp.net:389} 66a9c306 nsruvReplicaLastModified: {replica 56 ldap://in1-iparepl-01.dicomp.net:389} 66a 9c3eb nsruvReplicaLastModified: {replica 57 ldap://camper27.dicomp.net:389} 66a9c3f5 nsruvReplicaLastModified: {replica 60 ldap://camper25.dicomp.net:389} 66a9c990 nsruvReplicaLastModified: {replica 63 ldap://camper22.dicomp.net:389} 66a9cc21 nsruvReplicaLastModified: {replica 64 ldap://nz1-compca-01.dicomp.net:389} 66a9c c63 nsruvReplicaLastModified: {replica 52} 66a9cd67 nsds5ReplicaChangeCount: 117369 nsds5replicareapactive: 0
This one nsruvReplicaLastModified: {replica 52} 66a9cd67
does not have an associated nsds50ruv associated with it so removal via other tool does not work.
Trying to remove them via an LDAP modify too fails with an error additional info: Deletion of nsruvReplicaLastModified attribute is not allowed
Any help on gettng these records to vanish is very much appreciated as its causing cipa to believe there are ghost replicas. Looking at the cipa code tells me that its looking for entries for replica without an associated LDAP url to count towards ghost replicas.
Thanks !
Harikumar Krishnan via FreeIPA-users wrote:
Howdy folks,
We also have a similar issue. Some servers in our IPA topology show ghost replicas and if comes down to an entry like the following for an old replica which no longer exists
$ ldapsearch -xLLL -D "cn=directory manager" -W -b dc=DICOMP,dc=NET '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' Enter LDAP Password: dn: cn=replica,cn=dc\3Ddicomp\2Cdc\3Dnet,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindDNGroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=dicomp,dc=net nsDS5ReplicaBindDnGroupCheckInterval: 60 nsDS5ReplicaId: 11 nsDS5ReplicaName: 13387f82-373b11eb-a1r2gff0-4sda870 nsDS5ReplicaRoot: dc=dicomp,dc=net nsDS5ReplicaType: 3 nsState:: CwAAAAAAAABzzalmAAAAAAAAAAAAAAAAUpEAAAAAAAALAAAAAAAAAA== nsds5ReplicaBackoffMax: 300 nsds5ReplicaLegacyConsumer: off nsds5ReplicaReleaseTimeout: 60 objectClass: top objectClass: nsds5replica objectClass: extensibleobject nsds50ruv: {replicageneration} 5fc9ab2e000000040000 nsds50ruv: {replica 11 ldap://camper26.dicomp.net:389} 5fcbf1fa0000000b0000 66aa5 edc0000000b0000 nsds50ruv: {replica 3 ldap://camper21.dicomp.net:389} 5fc9ab34000000030000 66aa53c e000100030000 nsds50ruv: {replica 5 ldap://camper23.dicomp.net:389} 5fc9b44b000000050000 66aa58 d0000000050000 nsds50ruv: {replica 10 ldap://camper24.dicomp.net:389} 5fc9c7650000000a0000 66aa5 3d10004000a0000 nsds50ruv: {replica 33 ldap://ipa.dicomp.net:389} 626998ac000100210000 66aa5af1 000100210000 nsds50ruv: {replica 45 ldap://az1-iparepl-01.dicomp.net:389} 629644dc0001002d00 00 66aa58960000002d0000 nsds50ruv: {replica 46 ldap://au1-compca-01.dicomp.net:389} 6297aca50002002e0000 66aa59130003002e0000 nsds50ruv: {replica 48 ldap://nz1-freeipa-backup.dicomp.net:389} 62c8635e000200 300000 66aa4991000800300000 nsds50ruv: {replica 56 ldap://in1-iparepl-01.dicomp.net:389} 667aa1b90001003800 00 66aa553d000000380000 nsds50ruv: {replica 57 ldap://camper27.dicomp.net:389} 667bac3f000100390000 66aa5 547000000390000 nsds50ruv: {replica 60 ldap://camper25.dicomp.net:389} 667cf5c50000003c0000 66aa5a e00000003c0000 nsds50ruv: {replica 63 ldap://camper22.dicomp.net:389} 667d3ec50001003f0000 66aa 5d720000003f0000 nsds50ruv: {replica 64 ldap://nz1-compca-01.dicomp.net:389} 668e3565000100400000 66aa5d7e000000400000 nsds5agmtmaxcsn: dc=dicomp,dc=net;camper26.dicomp.net-to-camper27.dicomp.net;camper27.dicomp.net;389;57;66aa55c00000000b0000 nsds5agmtmaxcsn: dc=dicomp,dc=net;camper26.dicomp.net-to-in1-iparepl-01.dicomp.net; in1-iparepl-01.dicomp.net;389;56;66aa55c00000000b0000 nsruvReplicaLastModified: {replica 11 ldap://camper26.dicomp.net:389} 66a9cd8a nsruvReplicaLastModified: {replica 3 ldap://camper21.dicomp.net:389} 66a9c27f nsruvReplicaLastModified: {replica 5 ldap://camper23.dicomp.net:389} 66a9c780 nsruvReplicaLastModified: {replica 10 ldap://camper24.dicomp.net:389} 66a9c281 nsruvReplicaLastModified: {replica 33 ldap://ipa.dicomp.net:389} 66a9c9a4 nsruvReplicaLastModified: {replica 45 ldap://az1-iparepl-01.dicomp.net:389} 66a 9c745 nsruvReplicaLastModified: {replica 46 ldap://au1-compca-01.dicomp.net:389} 66a9c 7c5 nsruvReplicaLastModified: {replica 48 ldap://nz1-freeipa-backup.dicomp.net:389} 66a9c306 nsruvReplicaLastModified: {replica 56 ldap://in1-iparepl-01.dicomp.net:389} 66a 9c3eb nsruvReplicaLastModified: {replica 57 ldap://camper27.dicomp.net:389} 66a9c3f5 nsruvReplicaLastModified: {replica 60 ldap://camper25.dicomp.net:389} 66a9c990 nsruvReplicaLastModified: {replica 63 ldap://camper22.dicomp.net:389} 66a9cc21 nsruvReplicaLastModified: {replica 64 ldap://nz1-compca-01.dicomp.net:389} 66a9c c63 nsruvReplicaLastModified: {replica 52} 66a9cd67 nsds5ReplicaChangeCount: 117369 nsds5replicareapactive: 0
This one nsruvReplicaLastModified: {replica 52} 66a9cd67
does not have an associated nsds50ruv associated with it so removal via other tool does not work.
Trying to remove them via an LDAP modify too fails with an error additional info: Deletion of nsruvReplicaLastModified attribute is not allowed
Any help on gettng these records to vanish is very much appreciated as its causing cipa to believe there are ghost replicas. Looking at the cipa code tells me that its looking for entries for replica without an associated LDAP url to count towards ghost replicas.
You didn't say what you tried and how it failed. Either cleanruv or cleanallruv should do the trick.
rob
Hi rob,
Thanks a lot for replying back.
Things I tried
# clean-ruv via ipa-replica-manage
$ ipa-replica-manage clean-ruv 52 -f
Directory Manager password:
Replica ID 52 not found $
# clean-ruv job via ldapmodify ldif
$ cat cleanruv.ldif dn: cn=replica,cn=dc\3Ddicomp\2Cdc\3Dnet,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV52 $ $ ldapmodify -H ldap://$(hostname) -D "cn=Directory Manager" -W -f cleanruv.ldif Enter LDAP Password: modifying entry "cn=replica,cn=dc\3Ddicomp\2Cdc\3Dnet,cn=mapping tree,cn=config" $ Although this task says its modifying entry, the atrribute remains as such.
# manual ldap modify to delete the attribute
$ cat clean-ghost-repl.ldif
dn: cn=replica,cn=dc\3Ddicomp\2Cdc\3Dnet,cn=mapping tree,cn=config
changetype: modify
delete: nsruvReplicaLastModified
nsruvReplicaLastModified: {replica 52} 66a9cd67 $ $ ldapmodify -H ldap://$(hostname) -D "cn=Directory Manager" -W -f clean-ghost-repl.ldif
Enter LDAP Password:
modifying entry "cn=replica,cn=dc\3Ddicomp\2Cdc\3Dnet,cn=mapping tree,cn=config"
ldap_modify: Server is unwilling to perform (53)
additional info: Deletion of nsruvReplicaLastModified attribute is not allowed $
I have not tried to induce a cleanallruv task via an ldif, I thought cleanallruv is a globally replicated task similar to cleanruv just running on the local replica ? Does it still make sense to try a cleanallruv via ldapmodify as mentioned in https://www.port389.org/docs/389ds/howto/howto-cleanruv.html ?
Howdy folks,
We also have a similar issue. Some servers in our IPA topology show ghost replicas and if comes down to an entry like the following for an old replica which no longer exists
$ ldapsearch -xLLL -D "cn=directory manager" -W -b dc=DICOMP,dc=NET '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' Enter LDAP Password: dn: cn=replica,cn=dc\3Ddicomp\2Cdc\3Dnet,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindDNGroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=dicomp,dc=net nsDS5ReplicaBindDnGroupCheckInterval: 60 nsDS5ReplicaId: 11 nsDS5ReplicaName: 13387f82-373b11eb-a1r2gff0-4sda870 nsDS5ReplicaRoot: dc=dicomp,dc=net nsDS5ReplicaType: 3 nsState:: CwAAAAAAAABzzalmAAAAAAAAAAAAAAAAUpEAAAAAAAALAAAAAAAAAA== nsds5ReplicaBackoffMax: 300 nsds5ReplicaLegacyConsumer: off nsds5ReplicaReleaseTimeout: 60 objectClass: top objectClass: nsds5replica objectClass: extensibleobject nsds50ruv: {replicageneration} 5fc9ab2e000000040000 nsds50ruv: {replica 11 ldap://camper26.dicomp.net:389} 5fcbf1fa0000000b0000 66aa5 edc0000000b0000 nsds50ruv: {replica 3 ldap://camper21.dicomp.net:389} 5fc9ab34000000030000 66aa53c e000100030000 nsds50ruv: {replica 5 ldap://camper23.dicomp.net:389} 5fc9b44b000000050000 66aa58 d0000000050000 nsds50ruv: {replica 10 ldap://camper24.dicomp.net:389} 5fc9c7650000000a0000 66aa5 3d10004000a0000 nsds50ruv: {replica 33 ldap://ipa.dicomp.net:389} 626998ac000100210000 66aa5af1 000100210000 nsds50ruv: {replica 45 ldap://az1-iparepl-01.dicomp.net:389} 629644dc0001002d00 00 66aa58960000002d0000 nsds50ruv: {replica 46 ldap://au1-compca-01.dicomp.net:389} 6297aca50002002e0000 66aa59130003002e0000 nsds50ruv: {replica 48 ldap://nz1-freeipa-backup.dicomp.net:389} 62c8635e000200 300000 66aa4991000800300000 nsds50ruv: {replica 56 ldap://in1-iparepl-01.dicomp.net:389} 667aa1b90001003800 00 66aa553d000000380000 nsds50ruv: {replica 57 ldap://camper27.dicomp.net:389} 667bac3f000100390000 66aa5 547000000390000 nsds50ruv: {replica 60 ldap://camper25.dicomp.net:389} 667cf5c50000003c0000 66aa5a e00000003c0000 nsds50ruv: {replica 63 ldap://camper22.dicomp.net:389} 667d3ec50001003f0000 66aa 5d720000003f0000 nsds50ruv: {replica 64 ldap://nz1-compca-01.dicomp.net:389} 668e3565000100400000 66aa5d7e000000400000 nsds5agmtmaxcsn: dc=dicomp,dc=net;camper26.dicomp.net-to-camper27.dicomp.net;camper27.dicomp.net;389;57;66aa55c00000000b0000 nsds5agmtmaxcsn: dc=dicomp,dc=net;camper26.dicomp.net-to-in1-iparepl-01.dicomp.net; in1-iparepl-01.dicomp.net;389;56;66aa55c00000000b0000 nsruvReplicaLastModified: {replica 11 ldap://camper26.dicomp.net:389} 66a9cd8a nsruvReplicaLastModified: {replica 3 ldap://camper21.dicomp.net:389} 66a9c27f nsruvReplicaLastModified: {replica 5 ldap://camper23.dicomp.net:389} 66a9c780 nsruvReplicaLastModified: {replica 10 ldap://camper24.dicomp.net:389} 66a9c281 nsruvReplicaLastModified: {replica 33 ldap://ipa.dicomp.net:389} 66a9c9a4 nsruvReplicaLastModified: {replica 45 ldap://az1-iparepl-01.dicomp.net:389} 66a 9c745 nsruvReplicaLastModified: {replica 46 ldap://au1-compca-01.dicomp.net:389} 66a9c 7c5 nsruvReplicaLastModified: {replica 48 ldap://nz1-freeipa-backup.dicomp.net:389} 66a9c306 nsruvReplicaLastModified: {replica 56 ldap://in1-iparepl-01.dicomp.net:389} 66a 9c3eb nsruvReplicaLastModified: {replica 57 ldap://camper27.dicomp.net:389} 66a9c3f5 nsruvReplicaLastModified: {replica 60 ldap://camper25.dicomp.net:389} 66a9c990 nsruvReplicaLastModified: {replica 63 ldap://camper22.dicomp.net:389} 66a9cc21 nsruvReplicaLastModified: {replica 64 ldap://nz1-compca-01.dicomp.net:389} 66a9c c63 nsruvReplicaLastModified: {replica 52} 66a9cd67 nsds5ReplicaChangeCount: 117369 nsds5replicareapactive: 0
This one nsruvReplicaLastModified: {replica 52} 66a9cd67
does not have an associated nsds50ruv associated with it so removal via other tool does not work.
Trying to remove them via an LDAP modify too fails with an error additional info: Deletion of nsruvReplicaLastModified attribute is not allowed
Any help on gettng these records to vanish is very much appreciated as its causing cipa to believe there are ghost replicas. Looking at the cipa code tells me that its looking for entries for replica without an associated LDAP url to count towards ghost replicas.
Thanks !
freeipa-users@lists.fedorahosted.org
-
Harikumar Krishnan -
Nicholas Cross -
Rob Crittenden