Broken replicas and CleanRUV question - 389-users - Fedora mailing-lists

31 May 2017

      Hi all,
long time ago we have started with 389-DS and due to lack of experience 
I have installed and used admin server (which is abandoned later, 
because it is too complicated and requires someone at keyboard).
As consequence of that, we have started to replicate netscapeRoot 
space... During time, we have upgraded s/w from initial 
389-ds-1.2.1-1.el5 (started from FDS repository, moved to EPEL one 
later) to today's 389-ds-base-1.3.5.14-1.el6.x86_64 (this one is 
compiled from source and that was introduced before we have migrated 
boxes from RHEL5 to RHEL6 - actually CentOS OS).
During various phases of upgrades, netscapeRoot replicas went out of 
sync (we did not spotted that, because of bug in monitoring script - 
that is another issue).
Our setup includes MultiMaster ReadWrite replication (ldap1 <--> ldap2) 
and one ReadOnly (ldap3, consumes from both suppliers in MMR).
Right now, this:
$ for ldap in ldap1 ldap2; do
   ldapsearch -x -H ldaps://${ldap}.MyDomain.com -b "cn=mapping 
tree,cn=config" -D "cn=Directory Manager" -w ${DMPASS} -o ldif-wrap=no 
objectClass=nsDS5ReplicationAgreement |\
   awk -vLDAP=${ldap} '/^dn/ {printf("#===== %s =====#\n%s\n", LDAP, 
$0); next}; /^nsDS5ReplicaHost:/ {printf("%s\n", $0); next;}; 
/^nsds5replicaLastUpdateStatus:/ {printf("%s\n", $0); next;}'
done
returns (I have excluded working MyDomain replicas output):
$ #===== ldap1 =====#
dn: cn=2eLDAPmmr,cn=replica,cn=o\3Dnetscaperoot,cn=mapping tree,cn=config
nsDS5ReplicaHost: ldap2.MyDomain.com
nsds5replicaLastUpdateStatus: Error (0) No replication sessions started 
since server startup
#===== ldap1 =====#
dn: cn=2eLDAPror,cn=replica,cn=o\3Dnetscaperoot,cn=mapping tree,cn=config
nsDS5ReplicaHost: ldap3.MyDomain.com
nsds5replicaLastUpdateStatus: Error (0) No replication sessions started 
since server startup
#===== ldap2 =====#
dn: cn=2eLDAPmmr,cn=replica,cn=o\3Dnetscaperoot,cn=mapping tree,cn=config
nsDS5ReplicaHost: ldap1.MyDomain.com
nsds5replicaLastUpdateStatus: Error (0) No replication sessions started 
since server startup
#===== ldap2 =====#
dn: cn=2eLDAPror,cn=replica,cn=o\3Dnetscaperoot,cn=mapping tree,cn=config
nsDS5ReplicaHost: ldap3.MyDomain.com
nsds5replicaLastUpdateStatus: Error (0) No replication sessions started 
since server startup
I have tried various tricks to recover that replication, but w/o luck...
When I check (for example ldap1) with this:
$ ldapsearch -xLLLo ldif-wrap=no -H ldaps://ldap1.MyDomain.com -D 
'cn=directory manager' -w ${DMPASS} -b o=netscapeRoot 
'(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'
I get as result:
dn: cn=replica,cn=o\3Dnetscaperoot,cn=mapping tree,cn=config
objectClass: nsDS5Replica
objectClass: top
nsDS5ReplicaRoot: o=netscaperoot
nsDS5ReplicaType: 3
nsDS5Flags: 1
nsDS5ReplicaId: 11
nsds5ReplicaPurgeDelay: 604800
nsDS5ReplicaBindDN: cn=replication manager,cn=config
nsDS5ReplicaReferral: ldap://ldap2.MyDomain.com:636/o%3dnetscaperoot
cn: replica
nsState:: CwAAAAAAAACRKiRZAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAA==
nsDS5ReplicaName: dc964102-1dd111b2-8970c75e-63880000
nsds50ruv: {replicageneration} 4dcb9f790000000b0000
nsds50ruv: {replica 11 ldap://ldap1.MyDomain.com:0}
nsds50ruv: {replica 21 ldap://ldap2.MyDomain.com:0} 4dda4a3a000000150000 
4fd5f742000300150000
nsds5agmtmaxcsn: o=netscaperoot;2eLDAPror;ldap3.MyDomain.com;636;unavailable
nsruvReplicaLastModified: {replica 11 ldap://ldap1.MyDomain.com:0} 00000000
nsruvReplicaLastModified: {replica 21 ldap://ldap2.MyDomain.com:0} 00000000
nsds5ReplicaChangeCount: 1
nsds5replicareapactive: 0
Tried to CleanRUV (ldif applied with ldapmodify command to all suppliers 
and consumers):
$ cat /tmp/ldap.cleanRUV-tasks-for-netscapeRoot-replica.11.ldif
dn: cn=replica,cn=o\3Dnetscaperoot,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV11
At some moment, ldap1 replied:
"ldap_modify: Server is unwilling to perform (53)"
which explains nothing, because that error means:
"Indicates that the LDAP server cannot process the request because of 
server-defined restrictions. This error is returned for the following 
reasons: The add entry request violates the server's structure 
rules...OR...The modify attribute request specifies attributes that 
users cannot modify...OR...Password restrictions prevent the 
action...OR...Connection restrictions prevent the action. "
Right now, CleanRUV task is stuck... and replication is still broken... 
Similar situation is present on ldap2, with RUV 21 (if not worse):
dn: cn=replica,cn=o\3Dnetscaperoot,cn=mapping tree,cn=config
objectClass: nsDS5Replica
objectClass: top
nsDS5ReplicaRoot: o=netscaperoot
nsDS5ReplicaType: 3
nsDS5Flags: 1
nsDS5ReplicaId: 21
nsds5ReplicaPurgeDelay: 604800
nsDS5ReplicaBindDN: cn=replication manager,cn=config
nsDS5ReplicaReferral: ldap://ldap1.MyDomain.com:636/o%3dnetscaperoot
cn: replica
nsState:: FQAAAAAAAADeiyVZAAAAAAAAAAAAAAAAAQAAAAAAAAABAAAAAAAAAA==
nsDS5ReplicaName: cb016902-1dd111b2-821cbcea-f7780000
nsds50ruv: {replicageneration} 4dcb9f790000000b0000
nsds50ruv: {replica 21 ldap://ldap2.MyDomain.com:0} 4dda4a3a000000150000 
4fd5f742000300150000
nsds5agmtmaxcsn: o=netscaperoot;2eLDAPmmr;ldap1.MyDomain.com;636;unavailable
nsds5agmtmaxcsn: o=netscaperoot;2eLDAPror;ldap3.MyDomain.com;636;unavailable
nsruvReplicaLastModified: {replica 21 ldap://ldap2.MyDomain.com:0} 00000000
nsds5ReplicaChangeCount: 1
nsds5replicareapactive: 0
# What would be proper way to get out from this situation?
# Do I have to execute CleanAllRUV task and start replication from 
scratch or there is better way?
BTW, loglevel is set to 8192, so from ldap1 logs:
$ sudo grep cleanruv_task: /var/log/dirsrv/slapd-ldap?/errors
[31/May/2017:09:11:39 +0200] NSMMReplicationPlugin - cleanruv_task: 
cleaning rid (11)...
we see that task is "started" and never finished
Any advice or documentation (which is more up-2-date) than:
* 
http://directory.fedoraproject.org/docs/389ds/howto/howto-cleanruv.html#clea...
* 
https://access.redhat.com/documentation/en-us/red_hat_directory_server/9.0/h...
* 
http://directory.fedoraproject.org/docs/389ds/FAQ/troubleshoot-cleanallruv.h... 
(CleanRUV FAQ troubleshooting is missing at all)
is welcome.
With best regards.
Predrag Zečević
-- 
Predrag Zečević
Technical Support Analyst
2e Systems GmbH

Telephone: +49 6196 9505 815, Facsimile: +49 6196 9505 894
Mobile:    +49  174 3109 288,     Skype: predrag.zecevic
E-mail:    predrag.zecevic@2e-systems.com

Headquarter:          2e Systems GmbH, Königsteiner Str. 87,
                       65812 Bad Soden am Taunus, Germany
Company registration: Amtsgericht Königstein (Germany), HRB 7303
Managing director:    Phil Douglas

http://www.2e-systems.com/ - Making your business fly!