On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote: Hello, we are using freeipa in a 4way multi master replication setup. Servers ipa14,ipa15 and ipa34,ipa35 on CentOS Linux release 7.3.1611 (Core) with version ipa-server-common-4.4.0-14.el7.centos.7.noarch. We have an issue where one of the servers log a missing CSN. It happens even after ipa replication reinitialized. We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, but we see it in those files listed on ipa15 and the error is reported there. Please see attached file with logs. the missing csn is from Nov,13,2017 - so it is not unlikely it was trimmed. But in some RUV there seems to be a reference to it, and replication uses to position it in the changelog. How can we fix this? we first should get a full picture of the replicaids and RUVs on all servers, could you do on all servers the following search: ldapsearch .... -o ldif-wrap=no -D "cn=directory manager" .... -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv That should help in deciding what to do. There is also on option to kick an agreement to ingnore a missing change: do the following change on the failing replication agreement, but it would be better to have the data first: ldapmodify .... dn: <agmt> replace: nsds5ReplicaIgnoreMissingChange nsds5ReplicaIgnoreMissingChange: once -- *Sándor Juhász* System Administrator *ChemAxon* *Ltd*. Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 Cell: +36704258964 _______________________________________________ FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander _______________________________________________ FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org

Hi, when extracting the relevant data, we see: [root@ipa14 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 6 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 ~~~~~~~~ [root@ipa15 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 16 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 ~~~~~~~~ [root@ipa34 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 12 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 ~~~~~~~~ [root@ipa35 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 8 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 this indicates that all replicas are in sync for replicaid 8 and 12, but for rid 16, ipa 34 and ipa 35 have no data and for rid 6 they have older data. I cannot say what has happened, but I think you need reinit 34 and 35 from either 14 or 15 On 04/13/2018 11:13 AM, Sandor Juhasz wrote: here are the results: ~~~~~~~~ [root@ipa14 ~]# ldapsearch -H ldap://ipa14.bpo.cxn -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv # # replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 4 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad07160000000040000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad0711c003a000b0000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000 # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 6 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2 [root@ipa14 ~]# ~~~~~~~~ [root@ipa15 ~]# ldapsearch -H ldap://ipa15 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv # # replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 15 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad071c20000000f0000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad071d20021000b0000 # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 16 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2 [root@ipa15 ~]# ~~~~~~~~ [root@ipa34 ~]# ldapsearch -H ldap://ipa34 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv # # replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 11 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad072120003000b0000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000 # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 12 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2 [root@ipa34 ~]# ~~~~~~~~ [root@ipa35 ~]# ldapsearch -H ldap://ipa35 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv # # replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 7 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad07248001800070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad072490010000b0000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000 # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 8 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2 [root@ipa35 ~]# ~~~~~~~~ -- *Sándor Juhász* System Administrator *ChemAxon* *Ltd*. Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 Cell: +36704258964 On Fri, Apr 13, 2018 at 10:51 AM, Ludwig Krispenz via FreeIPA-users < freeipa-users(a)lists.fedorahosted.org&gt; wrote: > > On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote: > > Hello, > > we are using freeipa in a 4way multi master replication setup. > Servers ipa14,ipa15 and ipa34,ipa35 on > CentOS Linux release 7.3.1611 (Core) with version > ipa-server-common-4.4.0-14.el7.centos.7.noarch. > > We have an issue where one of the servers log a missing CSN. It happens > even after > ipa replication reinitialized. > We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, but > we see it in those files listed on ipa15 and the error is reported there. > Please see attached file with logs. > > the missing csn is from Nov,13,2017 - so it is not unlikely it was > trimmed. But in some RUV there seems to be a reference to it, and > replication uses to position it in the changelog. > > > > How can we fix this? > > we first should get a full picture of the replicaids and RUVs on all > servers, could you do on all servers the following search: > ldapsearch .... -o ldif-wrap=no -D "cn=directory manager" .... -b > cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv > > That should help in deciding what to do. > > There is also on option to kick an agreement to ingnore a missing change: > > do the following change on the failing replication agreement, but it > would be better to have the data first: > > ldapmodify .... > dn: <agmt> > replace: nsds5ReplicaIgnoreMissingChange > nsds5ReplicaIgnoreMissingChange: once > > > -- > *Sándor Juhász* > System Administrator > *ChemAxon* *Ltd*. > Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 > Cell: +36704258964 > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > > -- > Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, > Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to > freeipa-users-leave(a)lists.fedorahosted.org > > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander