4:42 a.m.
I am on a long journey upgrading from a mixture of different versions of
freeipa to the lastest and i am almost there, currently on 4.10.
After removing some older replicas yesterday, a new replica had a fault and
we had to restart dirsrv.
When it came back up we saw we have ghost replicas across the whole estate
6x replicas in two locations.
Issues are reported via `cipa`
Digging into the tombstone records, i find there are not the usual
tombstone replica records but more like the below.
My question is, how do i remove these redundant records? As I believe this
is what `cipa` is counting to report the errors.
(marked with <<<<<<<<<<<<<<)
The usual things do not work:
# ipa-replica-manage -p $pass clean-ruv 15
Replica ID 15 not found
$ cat remove_ruv.ldif
dn: cn=clean 15,cn=cleanallruv,cn=tasks,cn=config
changetype: add
objectclass: top
objectclass: extensibleObject
replica-base-dn: dc=ad,dc=companyx,dc=fm
replica-id: 15
cn: clean 15
ldapmodify -Y GSSAPI -f remove_ruv.ldif
Thanks in advance.
Nick
$ ldapsearch -D "cn=Directory Manager" -w $pass -Q -o ldif-wrap=no -LLL -b
"dc=ad,dc=companyx,dc=fm"
'(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))'
dn: cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping
tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=replication manager,cn=config
nsDS5ReplicaBindDNGroup: cn=replication
managers,cn=sysaccounts,cn=etc,dc=ad,dc=companyx,dc=fm
nsDS5ReplicaBindDnGroupCheckInterval: 60
nsDS5ReplicaId: 56
nsDS5ReplicaName: a6b5640c-ad3911ed-a50980fb-6203228c
nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm
nsDS5ReplicaType: 3
nsState:: OAAAAAAAAABSPEJkAAAAAAAAAAAAAAAA7AAAAAAAAAAGAAAAAAAAAA==
nsds5ReplicaBackoffMax: 300
nsds5ReplicaLegacyConsumer: off
nsds5ReplicaReleaseTimeout: 60
objectClass: top
objectClass: nsds5replica
objectClass: extensibleobject
nsds50ruv: {replicageneration} 5d9e2076000000040000
nsds50ruv: {replica 56 ldap://ipa006.ad.companyx.fm:389}
63ece66f000000380000 64423d4e000000380000
nsds50ruv: {replica 46 ldap://ipa005.ad.companyx.fm:389}
63dbcc200001002e0000 64423cf6000f002e0000
nsds50ruv: {replica 21 ldap://etcd0dc.ad.companyx.fm:389}
5f2d4bc1000000150000 64319ec3276200150000
nsds50ruv: {replica 48 ldap://ipa007.ad.companyx.fm:389}
63ea4e54000100300000 6442397e000a00300000
nsds50ruv: {replica 58 ldap://ipa001dc.ad.companyx.fm:389}
643d03280001003a0000 644232c00001003a0000
nsds50ruv: {replica 60 ldap://ipa002dc.ad.companyx.fm:389}
643d19680001003c0000 644232660004003c0000
nsds50ruv: {replica 62 ldap://ipa003dc.ad.companyx.fm:389}
643d491e0001003e0000 644233340000003e0000
nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;
ipa006.ad.companyx.fm-to-ipa003dc.ad.companyx.fm;ipa003dc.ad.companyx.fm
;389;62;644238c9000000380000
nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;
ipa006.ad.companyx.fm-to-ipa007.ad.companyx.fm;ipa007.ad.companyx.fm
;389;48;644238c9000000380000
nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;
ipa006.ad.companyx.fm-to-etcd2dc.ad.companyx.fm;etcd2dc.ad.companyx.fm;389;unavailable;64423cf6000f002e0000
<<<<<<<<<<<<<<
nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;
ipa006.ad.companyx.fm-to-ipa005.ad.companyx.fm;ipa005.ad.companyx.fm
;389;46;644238c9000000380000
nsruvReplicaLastModified: {replica 56 ldap://ipa006.ad.companyx.fm:389}
64423c62
nsruvReplicaLastModified: {replica 46 ldap://ipa005.ad.companyx.fm:389}
64423c0c
nsruvReplicaLastModified: {replica 21 ldap://etcd0dc.ad.companyx.fm:389}
00000000
nsruvReplicaLastModified: {replica 48 ldap://ipa007.ad.companyx.fm:389}
64423894
nsruvReplicaLastModified: {replica 58 ldap://ipa001dc.ad.companyx.fm:389}
644231da
nsruvReplicaLastModified: {replica 60 ldap://ipa002dc.ad.companyx.fm:389}
6442317a
nsruvReplicaLastModified: {replica 62 ldap://ipa003dc.ad.companyx.fm:389}
64423249
nsruvReplicaLastModified: {replica 12} 644180f7
<<<<<<<<<<<<<<
nsruvReplicaLastModified: {replica 15} 644180f7
<<<<<<<<<<<<<<
nsruvReplicaLastModified: {replica 25} 644180f7
<<<<<<<<<<<<<<
nsruvReplicaLastModified: {replica 23} 644180f7
<<<<<<<<<<<<<<
nsruvReplicaLastModified: {replica 40} 644180f7
<<<<<<<<<<<<<<
nsds5ReplicaChangeCount: 734077
nsds5replicareapactive: 0
7:04 a.m.
I think i have a handle on this now.
There are a number of issues that i am now aware of.
1. old replication agreement to oldbox1 on newbox6
2. corrupt RUVs, giving the impression of Ghost Replicas.
For #1 i normally can delete these fine with a ldap command. BUT! running this crashes
the dirsrv service.
ldapdelete -D "cn=Directory Manager" -w $pwd -p 389 -h localhost -x
"cn=newbox6.ad.dice.fm-to-oldbox1.ad.dice.fm,cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping
tree,cn=config"
Apr 21 11:35:48 newbox6.ad.dice.fm systemd[1]: Starting 389 Directory Server
AD-DICE-FM....
Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv(a)AD-DICE-FM.service: Failed with
result 'signal'.
Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv(a)AD-DICE-FM.service: Main process
exited, code=killed, status=6/ABRT
Apr 21 11:35:11 newbox6.ad.dice.fm ns-slapd[2934110]: ns-slapd:
ldap/servers/plugins/sync/sync_persist.c:234: sync_update_persist_op: Assertion
`prim_op' failed.
For #2 , this issue of not being able to remove the replication agreement, stops the
removal of the corrupt RUVs.
As you can see i have tried to kick off some RUV removals but they are failing as not all
replicas are online. (but they dont exist as #1)
$ ipa-replica-manage list-clean-ruv -p $pass
ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission
denied: '/var/log/ipa/cli.log'
CLEANALLRUV tasks
RID 12: Not all replicas online, retrying in 40 seconds...
RID restarted-2658134: Not all replicas online, retrying in 320 seconds...
RID restarted-2658136: Not all replicas online, retrying in 320 seconds...
No abort CLEANALLRUV tasks running
So, more questions really,
Why is the ldapdelete crashing the service?
How do i fix it?
thanks,
Nick
10:51 a.m.
Nicholas Cross via FreeIPA-users wrote:
I think i have a handle on this now.
There are a number of issues that i am now aware of.
1. old replication agreement to oldbox1 on newbox6
2. corrupt RUVs, giving the impression of Ghost Replicas.
For #1 i normally can delete these fine with a ldap command. BUT! running this crashes
the dirsrv service.
ldapdelete -D "cn=Directory Manager" -w $pwd -p 389 -h localhost -x
"cn=newbox6.ad.dice.fm-to-oldbox1.ad.dice.fm,cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping
tree,cn=config"
Apr 21 11:35:48 newbox6.ad.dice.fm systemd[1]: Starting 389 Directory Server
AD-DICE-FM....
Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv(a)AD-DICE-FM.service: Failed with
result 'signal'.
Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv(a)AD-DICE-FM.service: Main process
exited, code=killed, status=6/ABRT
Apr 21 11:35:11 newbox6.ad.dice.fm ns-slapd[2934110]: ns-slapd:
ldap/servers/plugins/sync/sync_persist.c:234: sync_update_persist_op: Assertion
`prim_op' failed.
For #2 , this issue of not being able to remove the replication agreement, stops the
removal of the corrupt RUVs.
As you can see i have tried to kick off some RUV removals but they are failing as not all
replicas are online. (but they dont exist as #1)
$ ipa-replica-manage list-clean-ruv -p $pass
ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission
denied: '/var/log/ipa/cli.log'
CLEANALLRUV tasks
RID 12: Not all replicas online, retrying in 40 seconds...
RID restarted-2658134: Not all replicas online, retrying in 320 seconds...
RID restarted-2658136: Not all replicas online, retrying in 320 seconds...
No abort CLEANALLRUV tasks running
So, more questions really,
Why is the ldapdelete crashing the service?
Can you install the debuginfo packages and get a stack trace? I'm sure
the 389-ds developers would like to take a look at it. It may already
have been addressed. You haven't provided any version or distro information.
How do i fix it?
If you shut down dirsrv then you can edit
/etc/dirsrv/slapd-REALM/dse.ldif and manually remove the replication
agreement.
But you have to shut down dirsrv first.
rob
1:21 p.m.
Thanks for the tips. I'll run the trace on Monday.
Also i'll edit the ldif Monday too.
Distro: alma9
# rpm -qa | grep ipa | sort
almalinux-logos-ipa-90.5.1-1.1.el9.noarch
ipa-client-4.10.0-8.el9_1.x86_64
ipa-client-common-4.10.0-8.el9_1.noarch
ipa-common-4.10.0-8.el9_1.noarch
ipa-healthcheck-0.9-9.el9.noarch
ipa-healthcheck-core-0.9-9.el9.noarch
ipa-selinux-4.9.8-7.el9_0.noarch
ipa-server-4.10.0-8.el9_1.x86_64
ipa-server-common-4.10.0-8.el9_1.noarch
ipa-server-dns-4.10.0-8.el9_1.noarch
libipa_hbac-2.7.3-4.el9_1.3.x86_64
python3-ipaclient-4.10.0-8.el9_1.noarch
python3-ipalib-4.10.0-8.el9_1.noarch
python3-ipaserver-4.10.0-8.el9_1.noarch
python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64
sssd-ipa-2.7.3-4.el9_1.3.x86_64
4:56 a.m.
Shutdown dirsrv
backed up dse.ldif
edited the ldif
restarted dirsrv
replication agreement came back!
Is it being sync'ed from some where else? another file?
2:53 p.m.
Nicholas Cross via FreeIPA-users wrote:
Shutdown dirsrv
backed up dse.ldif
edited the ldif
restarted dirsrv
replication agreement came back!
Is it being sync'ed from some where else? another file?
It sounds like dirsrv was still running if values in cn=config were
restored. It writes the file on exit. This subtree is not replicated.
rob
10:10 a.m.
Tested this again making sure that dirsrv is not running and the replica record is back.
I am obviously doing something wrong. My steps are below. I appreciate your time on
this.
#
# check dirsrv is currently running
#
[root@ipa006 ~]# ps aux | grep dirsrv
dirsrv 3221639 31.4 5.4 2418488 883856 ? Ssl Apr24 322:04 /usr/sbin/ns-slapd -D
/etc/dirsrv/slapd-AD-companyx-FM -i /run/dirsrv/slapd-AD-companyx-FM.pid
root 3281205 0.0 0.0 6412 2204 pts/2 S+ 09:11 0:00 grep --color=auto
dirsrv
#
# shutdown dirsrv
#
[root@ipa006 ~]# time systemctl stop dirsrv(a)AD-companyx-FM.service
real 10m0.130s
user 0m0.009s
sys 0m0.012s
#
# check dirsrv is not running 1
#
[root@ipa006 ~]# ps aux | grep dirsrv
root 3282962 0.0 0.0 6412 2244 pts/2 S+ 09:47 0:00 grep --color=auto
dirsrv
#
# check dirsrv is not running 2
#
[root@ipa006 slapd-AD-companyx-FM]# ipactl status
Directory Service: STOPPED
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
1 service(s) are not running
#
# go to right folder
#
[root@ipa006 ~]# cd /etc/dirsrv/slapd-AD-companyx-FM/
#
# make a backup just incase
#
[root@ipa006 slapd-AD-companyx-FM]# cp dse.ldif dse.ldif.nickx-25apr23
#
# edit ldif
#
[root@ipa006 slapd-AD-companyx-FM]# vi dse.ldif
#
# remove this record. Hoping its the right thing to do.
#
dn:
cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi
ce\2Cdc\3Dfm,cn=mapping tree,cn=config
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
objectClass: top
cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm
nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm
nsDS5ReplicaPort: 389
nsds5replicaTimeout: 300
nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm
description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm
ipaReplTopoManagedAgreementState: managed agreement - generated by topology pl
ugin
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
ternalModifyTimestamp
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
uccessfulauth krblastfailedauth krbloginfailedcount
creatorsName: cn=IPA Topology Configuration,cn=plugins,cn=config
modifiersName: cn=IPA Topology Configuration,cn=plugins,cn=config
createTimestamp: 20230425095140Z
modifyTimestamp: 20230425095140Z
#
# check no records exist in dse.ldif
#
[root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif
[root@ipa006 slapd-AD-companyx-FM]#
[root@ipa006 slapd-AD-companyx-FM]# time systemctl start dirsrv(a)AD-companyx-FM.service
real 0m12.343s
user 0m0.006s
sys 0m0.007s
#
# Look in logs
#
Apr 25 09:51:51 ipa006.ad.companyx.fm ns-slapd[3283119]: [25/Apr/2023:09:51:51.484197325
+0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp -
agmt="cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm"
(bad_serverdc:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't
contact LDAP server) ()
#
# check dse.ldif again - find entry is back !
#
[root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif
dn:
cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi
cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm
nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm
description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm
#
# scratch head and ponder life, the universe and everything
#
4:26 p.m.
Nicholas Cross via FreeIPA-users wrote:
Tested this again making sure that dirsrv is not running and the
replica record is back.
I am obviously doing something wrong. My steps are below. I appreciate your time on
this.
#
# check dirsrv is currently running
#
[root@ipa006 ~]# ps aux | grep dirsrv
dirsrv 3221639 31.4 5.4 2418488 883856 ? Ssl Apr24 322:04 /usr/sbin/ns-slapd -D
/etc/dirsrv/slapd-AD-companyx-FM -i /run/dirsrv/slapd-AD-companyx-FM.pid
root 3281205 0.0 0.0 6412 2204 pts/2 S+ 09:11 0:00 grep --color=auto
dirsrv
#
# shutdown dirsrv
#
[root@ipa006 ~]# time systemctl stop dirsrv(a)AD-companyx-FM.service
real 10m0.130s
user 0m0.009s
sys 0m0.012s
#
# check dirsrv is not running 1
#
[root@ipa006 ~]# ps aux | grep dirsrv
root 3282962 0.0 0.0 6412 2244 pts/2 S+ 09:47 0:00 grep --color=auto
dirsrv
#
# check dirsrv is not running 2
#
[root@ipa006 slapd-AD-companyx-FM]# ipactl status
Directory Service: STOPPED
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
1 service(s) are not running
#
# go to right folder
#
[root@ipa006 ~]# cd /etc/dirsrv/slapd-AD-companyx-FM/
#
# make a backup just incase
#
[root@ipa006 slapd-AD-companyx-FM]# cp dse.ldif dse.ldif.nickx-25apr23
#
# edit ldif
#
[root@ipa006 slapd-AD-companyx-FM]# vi dse.ldif
#
# remove this record. Hoping its the right thing to do.
#
dn:
cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi
ce\2Cdc\3Dfm,cn=mapping tree,cn=config
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
objectClass: top
cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm
nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm
nsDS5ReplicaPort: 389
nsds5replicaTimeout: 300
nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm
description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm
ipaReplTopoManagedAgreementState: managed agreement - generated by topology pl
ugin
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
ternalModifyTimestamp
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
uccessfulauth krblastfailedauth krbloginfailedcount
creatorsName: cn=IPA Topology Configuration,cn=plugins,cn=config
modifiersName: cn=IPA Topology Configuration,cn=plugins,cn=config
createTimestamp: 20230425095140Z
modifyTimestamp: 20230425095140Z
#
# check no records exist in dse.ldif
#
[root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif
[root@ipa006 slapd-AD-companyx-FM]#
[root@ipa006 slapd-AD-companyx-FM]# time systemctl start dirsrv(a)AD-companyx-FM.service
real 0m12.343s
user 0m0.006s
sys 0m0.007s
#
# Look in logs
#
Apr 25 09:51:51 ipa006.ad.companyx.fm ns-slapd[3283119]: [25/Apr/2023:09:51:51.484197325
+0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp -
agmt="cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm"
(bad_serverdc:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't
contact LDAP server) ()
#
# check dse.ldif again - find entry is back !
#
[root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif
dn:
cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi
cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm
nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm
description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm
#
# scratch head and ponder life, the universe and everything
#
I have a guess what is happening. The topology is also stored within IPA
itself. I wonder if this is self-healing.
I assume you've ensure that the bad server is not anywhere else inside
of IPA? ipa server-del bad_server? Does its host entry exist?
ipa-replica-manage list -v `hostname` on the server the agreement exists?
rob
6 a.m.
Ah got it! Wonderful.
The trick as to run the topologysegement-del on the same server it was on.
It seems i am moving forward with this now - thanks.
#
# To remove the topology segment, which removed the replica agreement
#
#
# Show the bad replication agreement
#
# ipa-replica-manage list -v `hostname`
Directory Manager password:
bad_server.ad.companyx.fm: replica
last update status: Error (-1) Problem connecting to replica - LDAP error: Can't
contact LDAP server (connection error)
last update ended: 1970-01-01 00:00:00+00:00
ipa003dc.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2023-04-26 06:43:07+00:00
ipa005.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2023-04-26 06:43:14+00:00
ipa007.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2023-04-26 06:43:02+00:00
#
# find the segment (domain or ca)
#
$ ipa topologysegment-find domain | grep etcd
Segment name: ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm
Right node: bad_server.ad.companyx.fm
#
# delete that segment
#
$ ipa topologysegment-del domain ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm
---------------------------------------------------------
Deleted segment "ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm"
---------------------------------------------------------
#
# check it has gone - tada!
#
$ ipa-replica-manage list -v `hostname`
ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission
denied: '/var/log/ipa/cli.log'
ipa003dc.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update started
last update ended: 1970-01-01 00:00:00+00:00
ipa005.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update started
last update ended: 1970-01-01 00:00:00+00:00
ipa007.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2023-04-26 06:46:08+00:00
#
# Next up, removing the "LDAP Conflicts" but - "Removal of Segment
disconnects topology.Deletion not allowed."
#
$ ldapdelete
cn=bad_server.ad.companyx.fm-to-ipa006.ad.companyx.fm+nsuniqueid=34b26c01-ceee11ed-9d1d82de-03f3a8a3,cn=ca,cn=topology,cn=ipa,cn=etc,dc=ad,dc=companyx,dc=fm
SASL/GSSAPI authentication started
SASL username: nicholas.cross(a)AD.companyx.FM
SASL SSF: 256
SASL data security layer installed.
ldap_delete: Server is unwilling to perform (53)
additional info: Removal of Segment disconnects topology.Deletion not allowed.
#
# I think this is the solution: https://access.redhat.com/solutions/5507711
#
# Question1: during running the above RedHat solution, does this only disable the topology
replication? and leaves all other dirsrv components running?
#
#
# After that - finally remove the Ghost Replicas - which was the original question.
#
$ ldapsearch -D "cn=Directory Manager" -w $pass -Q -o ldif-wrap=no -LLL -b
"dc=ad,dc=companyx,dc=fm"
'(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))'
dn: cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=replication manager,cn=config
nsDS5ReplicaBindDNGroup: cn=replication
managers,cn=sysaccounts,cn=etc,dc=ad,dc=companyx,dc=fm
nsDS5ReplicaBindDnGroupCheckInterval: 60
nsDS5ReplicaId: 56
nsDS5ReplicaName: a6b5640c-ad3911ed-a50980fb-6203228c
nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm
nsDS5ReplicaType: 3
nsState:: OAAAAAAAAABf0EhkAAAAAAAAAAAAAAAA7AAAAAAAAAAFAAAAAAAAAA==
nsds5ReplicaBackoffMax: 300
nsds5ReplicaLegacyConsumer: off
nsds5ReplicaReleaseTimeout: 60
objectClass: top
objectClass: nsds5replica
objectClass: extensibleobject
nsds5ReplicaCleanRUV: 15:no:0:dc=ad,dc=companyx,dc=fm
nsds5ReplicaCleanRUV: 24:no:0:dc=ad,dc=companyx,dc=fm
nsds50ruv: {replicageneration} 5d9e2076000000040000
nsds50ruv: {replica 56 ldap://ipa006.ad.companyx.fm:389} 63ece66f000000380000
6448d15d000400380000
nsds50ruv: {replica 46 ldap://ipa005.ad.companyx.fm:389} 63dbcc200001002e0000
6448d115000e002e0000
nsds50ruv: {replica 48 ldap://ipa007.ad.companyx.fm:389} 63ea4e54000100300000
6448d115000700300000
nsds50ruv: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 643d03280001003a0000
6448ca410000003a0000
nsds50ruv: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 643d19680001003c0000
6448c9e40009003c0000
nsds50ruv: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 643d491e0001003e0000
6448cab40000003e0000
nsds5agmtmaxcsn:
dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa003dc.ad.companyx.fm;ipa003dc.ad.companyx.fm;389;62;6448cf8e000800380000
nsds5agmtmaxcsn:
dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa005.ad.companyx.fm;ipa005.ad.companyx.fm;389;46;6448cf8e000800380000
nsds5agmtmaxcsn:
dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa007.ad.companyx.fm;ipa007.ad.companyx.fm;389;48;6448cf8e000800380000
nsruvReplicaLastModified: {replica 56 ldap://ipa006.ad.companyx.fm:389} 6448d071
nsruvReplicaLastModified: {replica 46 ldap://ipa005.ad.companyx.fm:389} 6448d02b
nsruvReplicaLastModified: {replica 48 ldap://ipa007.ad.companyx.fm:389} 6448d02b
nsruvReplicaLastModified: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 6448c956
nsruvReplicaLastModified: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 6448c8fb
nsruvReplicaLastModified: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 6448c9c9
nsruvReplicaLastModified: {replica 25} 00000000
nsruvReplicaLastModified: {replica 23} 00000000
nsruvReplicaLastModified: {replica 40} 00000000
nsruvReplicaLastModified: {replica 12} 00000000
nsruvReplicaLastModified: {replica 21} 00000000
nsds5ReplicaChangeCount: 790081
nsds5replicareapactive: 0
#
# Question2: How to remove these? from the above
#
nsruvReplicaLastModified: {replica 25} 00000000
nsruvReplicaLastModified: {replica 23} 00000000
nsruvReplicaLastModified: {replica 40} 00000000
nsruvReplicaLastModified: {replica 12} 00000000
nsruvReplicaLastModified: {replica 21} 00000000
# this sort of thing doesn't seem to work.
dn: cn=clean 12,cn=cleanallruv,cn=tasks,cn=config
changetype: add
objectclass: top
objectclass: extensibleObject
replica-base-dn: dc=ad,dc=companyx,dc=fm
replica-id: 12
cn: clean 12
Many thanks.
9:59 a.m.
Nicholas Cross via FreeIPA-users wrote:
Ah got it! Wonderful.
The trick as to run the topologysegement-del on the same server it was on.
It seems i am moving forward with this now - thanks.
#
# To remove the topology segment, which removed the replica agreement
#
#
# Show the bad replication agreement
#
# ipa-replica-manage list -v `hostname`
Directory Manager password:
bad_server.ad.companyx.fm: replica
last update status: Error (-1) Problem connecting to replica - LDAP error: Can't
contact LDAP server (connection error)
last update ended: 1970-01-01 00:00:00+00:00
ipa003dc.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2023-04-26 06:43:07+00:00
ipa005.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2023-04-26 06:43:14+00:00
ipa007.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2023-04-26 06:43:02+00:00
#
# find the segment (domain or ca)
#
$ ipa topologysegment-find domain | grep etcd
Segment name: ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm
Right node: bad_server.ad.companyx.fm
#
# delete that segment
#
$ ipa topologysegment-del domain ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm
---------------------------------------------------------
Deleted segment "ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm"
---------------------------------------------------------
#
# check it has gone - tada!
#
$ ipa-replica-manage list -v `hostname`
ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission
denied: '/var/log/ipa/cli.log'
ipa003dc.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update
started
last update ended: 1970-01-01 00:00:00+00:00
ipa005.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update
started
last update ended: 1970-01-01 00:00:00+00:00
ipa007.ad.companyx.fm: replica
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2023-04-26 06:46:08+00:00
#
# Next up, removing the "LDAP Conflicts" but - "Removal of Segment
disconnects topology.Deletion not allowed."
#
$ ldapdelete
cn=bad_server.ad.companyx.fm-to-ipa006.ad.companyx.fm+nsuniqueid=34b26c01-ceee11ed-9d1d82de-03f3a8a3,cn=ca,cn=topology,cn=ipa,cn=etc,dc=ad,dc=companyx,dc=fm
SASL/GSSAPI authentication started
SASL username: nicholas.cross(a)AD.companyx.FM
SASL SSF: 256
SASL data security layer installed.
ldap_delete: Server is unwilling to perform (53)
additional info: Removal of Segment disconnects topology.Deletion not allowed.
#
# I think this is the solution: https://access.redhat.com/solutions/5507711
#
# Question1: during running the above RedHat solution, does this only disable the
topology replication? and leaves all other dirsrv components running?
#
#
# After that - finally remove the Ghost Replicas - which was the original question.
#
$ ldapsearch -D "cn=Directory Manager" -w $pass -Q -o ldif-wrap=no -LLL -b
"dc=ad,dc=companyx,dc=fm"
'(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))'
dn: cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=replication manager,cn=config
nsDS5ReplicaBindDNGroup: cn=replication
managers,cn=sysaccounts,cn=etc,dc=ad,dc=companyx,dc=fm
nsDS5ReplicaBindDnGroupCheckInterval: 60
nsDS5ReplicaId: 56
nsDS5ReplicaName: a6b5640c-ad3911ed-a50980fb-6203228c
nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm
nsDS5ReplicaType: 3
nsState:: OAAAAAAAAABf0EhkAAAAAAAAAAAAAAAA7AAAAAAAAAAFAAAAAAAAAA==
nsds5ReplicaBackoffMax: 300
nsds5ReplicaLegacyConsumer: off
nsds5ReplicaReleaseTimeout: 60
objectClass: top
objectClass: nsds5replica
objectClass: extensibleobject
nsds5ReplicaCleanRUV: 15:no:0:dc=ad,dc=companyx,dc=fm
nsds5ReplicaCleanRUV: 24:no:0:dc=ad,dc=companyx,dc=fm
nsds50ruv: {replicageneration} 5d9e2076000000040000
nsds50ruv: {replica 56 ldap://ipa006.ad.companyx.fm:389} 63ece66f000000380000
6448d15d000400380000
nsds50ruv: {replica 46 ldap://ipa005.ad.companyx.fm:389} 63dbcc200001002e0000
6448d115000e002e0000
nsds50ruv: {replica 48 ldap://ipa007.ad.companyx.fm:389} 63ea4e54000100300000
6448d115000700300000
nsds50ruv: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 643d03280001003a0000
6448ca410000003a0000
nsds50ruv: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 643d19680001003c0000
6448c9e40009003c0000
nsds50ruv: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 643d491e0001003e0000
6448cab40000003e0000
nsds5agmtmaxcsn:
dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa003dc.ad.companyx.fm;ipa003dc.ad.companyx.fm;389;62;6448cf8e000800380000
nsds5agmtmaxcsn:
dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa005.ad.companyx.fm;ipa005.ad.companyx.fm;389;46;6448cf8e000800380000
nsds5agmtmaxcsn:
dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa007.ad.companyx.fm;ipa007.ad.companyx.fm;389;48;6448cf8e000800380000
nsruvReplicaLastModified: {replica 56 ldap://ipa006.ad.companyx.fm:389} 6448d071
nsruvReplicaLastModified: {replica 46 ldap://ipa005.ad.companyx.fm:389} 6448d02b
nsruvReplicaLastModified: {replica 48 ldap://ipa007.ad.companyx.fm:389} 6448d02b
nsruvReplicaLastModified: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 6448c956
nsruvReplicaLastModified: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 6448c8fb
nsruvReplicaLastModified: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 6448c9c9
nsruvReplicaLastModified: {replica 25} 00000000
nsruvReplicaLastModified: {replica 23} 00000000
nsruvReplicaLastModified: {replica 40} 00000000
nsruvReplicaLastModified: {replica 12} 00000000
nsruvReplicaLastModified: {replica 21} 00000000
nsds5ReplicaChangeCount: 790081
nsds5replicareapactive: 0
#
# Question2: How to remove these? from the above
#
nsruvReplicaLastModified: {replica 25} 00000000
nsruvReplicaLastModified: {replica 23} 00000000
nsruvReplicaLastModified: {replica 40} 00000000
nsruvReplicaLastModified: {replica 12} 00000000
nsruvReplicaLastModified: {replica 21} 00000000
# this sort of thing doesn't seem to work.
dn: cn=clean 12,cn=cleanallruv,cn=tasks,cn=config
changetype: add
objectclass: top
objectclass: extensibleObject
replica-base-dn: dc=ad,dc=companyx,dc=fm
replica-id: 12
cn: clean 12
You can try ipa-replica-manage clean-ruv <value> to try to remove
specific values.
rob
5:43 a.m.
Just an update on this.
Came back from the long weekend and 50% of our servers (3) were not responding, the dirsrv
was crashing everytime it had an update from the CA master (we could not figure out why).
If we closed the firewall between replica and CA master the servers stayed up.
After a few days of trying various things to resurrect the down servers we rebuilt the
whole cluster based off the master CA server. None of the original servers are now
present.
After another long weekend we seem (so far) to have a stable cluster. Ignoring the usual
replication conflicts we get with heavy server creation/deletion due to AWS spot
instances.
The only out standing item now is the records that make "cipa" think we have
"ghost replicas"
nsruvReplicaLastModified: {replica 25} 00000000
nsruvReplicaLastModified: {replica 23} 00000000
nsruvReplicaLastModified: {replica 40} 00000000
nsruvReplicaLastModified: {replica 21} 00000000
There are no RUVs to match these replicas (21,23, 25, 40).
So it looks like these key/value pairs are the only things left.
Any ideas on how to remove them?
Many thanks.
351
days inactive
369
days old
freeipa-users@lists.fedorahosted.org
10 comments
2 participants
participants (2)
-
Nicholas Cross -
Rob Crittenden