Hi,
when extracting the relevant data, we see:

[root@ipa14 ~]
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 6
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
~~~~~~~~
[root@ipa15 ~]
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 16
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
~~~~~~~~
[root@ipa34 ~]
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 12
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
~~~~~~~~
[root@ipa35 ~]
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 8
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000


this indicates that all replicas are in sync for replicaid 8 and 12, but for rid 16, ipa 34 and ipa 35 have no data and for rid 6 they have older data.
I cannot say what has happened, but I think you need reinit 34 and 35 from either 14 or 15

On 04/13/2018 11:13 AM, Sandor Juhasz wrote:
here are the results:

~~~~~~~~

[root@ipa14 ~]# ldapsearch -H ldap://ipa14.bpo.cxn -o ldif-wrap=no  -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <cn=config> with scope subtree
# filter: objectclass=nsds5replica
# requesting: nsds5replicaid nsds50ruv 
#

# replica, dc\3Dcxn, mapping tree, config
dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
nsds5replicaid: 4
nsds50ruv: {replicageneration} 58987d9e000000040000
nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad07160000000040000
nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000
nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad0711c003a000b0000
nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 6
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2
[root@ipa14 ~]#

~~~~~~~~

[root@ipa15 ~]# ldapsearch -H ldap://ipa15 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <cn=config> with scope subtree
# filter: objectclass=nsds5replica
# requesting: nsds5replicaid nsds50ruv 
#

# replica, dc\3Dcxn, mapping tree, config
dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
nsds5replicaid: 15
nsds50ruv: {replicageneration} 58987d9e000000040000
nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad071c20000000f0000
nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000
nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000
nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad071d20021000b0000

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 16
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2
[root@ipa15 ~]#

~~~~~~~~

[root@ipa34 ~]# ldapsearch -H ldap://ipa34 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <cn=config> with scope subtree
# filter: objectclass=nsds5replica
# requesting: nsds5replicaid nsds50ruv 
#

# replica, dc\3Dcxn, mapping tree, config
dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
nsds5replicaid: 11
nsds50ruv: {replicageneration} 58987d9e000000040000
nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad072120003000b0000
nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000
nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000
nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 12
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2
[root@ipa34 ~]#

~~~~~~~~

[root@ipa35 ~]# ldapsearch -H ldap://ipa35 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <cn=config> with scope subtree
# filter: objectclass=nsds5replica
# requesting: nsds5replicaid nsds50ruv 
#

# replica, dc\3Dcxn, mapping tree, config
dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
nsds5replicaid: 7
nsds50ruv: {replicageneration} 58987d9e000000040000
nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad07248001800070000
nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000
nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad072490010000b0000
nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 8
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2
[root@ipa35 ~]#

~~~~~~~~

--
Sándor Juhász
System Administrator
ChemAxon Ltd.
Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
Cell: +36704258964

On Fri, Apr 13, 2018 at 10:51 AM, Ludwig Krispenz via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:

On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote:
Hello,

we are using freeipa in a 4way multi master replication setup. 
Servers ipa14,ipa15 and ipa34,ipa35 on
CentOS Linux release 7.3.1611 (Core) with version
ipa-server-common-4.4.0-14.el7.centos.7.noarch.

We have an issue where one of the servers log a missing CSN. It happens even after 
ipa replication reinitialized.
We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, but we see it in those files listed on ipa15 and the error is reported there.
Please see attached file with logs.
the missing csn is from Nov,13,2017 - so it is not unlikely it was trimmed. But in some RUV there seems to be a reference to it, and replication uses to position it in the changelog.



How can we fix this?
we first should get a full picture of the replicaids and RUVs on all servers, could you do on all servers the following search:
ldapsearch  .... -o ldif-wrap=no  -D "cn=directory manager" .... -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv

That should help in deciding what to do.

There is also on option to kick an agreement to ingnore a missing change:

do the following change on the failing replication agreement, but it would be better to have the data first:

ldapmodify ....
dn: <agmt>
replace: nsds5ReplicaIgnoreMissingChange
nsds5ReplicaIgnoreMissingChange: once


--
Sándor Juhász
System Administrator
ChemAxon Ltd.
Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
Cell: +36704258964


_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org

-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, 
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander

_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org



-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, 
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander