Rich,

I have an setup like:

A <-----> B /\ \ / /\ | \ / | | / | | / \ | | / \ | // \ /\ D <-----> C

At first, I do set the agreements up for the Ring A to B to C to B to A. This works. Then, I try to set the cross agreements from A to C and B to D up. This is where I run into this issue.

Let's have a look how I do those cross agreements. First I add an agreement on A for C. This is fine. Then, I do the same on C (for A) and I get the messages NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation On C and on A I get: [10/Aug/2010:17:12:37 -0400] - somehow, there are still 16 entries in the entry cache. :/ [10/Aug/2010:17:12:38 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:17:12:38 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0

Hope, this helps.

Thanks, -Reinhard -----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Reinhard Nappert Sent: Tuesday, August 10, 2010 2:42 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Rich, on the consumer, I see the following messages:

NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 12:41 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

Hi,

I have seen the following message in the errors log file, when I set MMR agreements up:

[10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=base: 1 [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - multimaster_be_state_change: replica o=base is going offline; disabling replication [10/Aug/2010:11:46:46 -0400] - somehow, there are still 20 entries in the entrycache. :/ [10/Aug/2010:11:46:46 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:11:46:48 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 [10/Aug/2010:11:46:52 -0400] - Fedora-Directory/1.1.2 B2009.090.1643 starting up [10/Aug/2010:11:46:52 -0400] - Detected Disorderly Shutdown last time DirectoryServer was running, recovering database.

After I re-initialize the database from the supplier (setting attribute nsds5BeginReplicaRefresh to start of the agreement object), the database gets correctly imported.

Any idea, what is going on?

No, not sure. But if you can develop a reproducible test case, that would be helpful.

Thanks, -Reinhard

---

--

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

At first I create (besides the changelog and replica entry with nsDS5ReplicaType=3, nsDS5Flags=1 and an unique nsDS5ReplicaId) the shadowing agreement with nsDS5ReplicaHost=<hostname of remote server>, nsDS5ReplicaTransportInfo=LDAP, nsDS5ReplicaPort=389, nsDS5ReplicaBindDN=<replManager-DN>, nsDS5ReplicaBindMethod=SIMPLE, nsDS5ReplicaCredentials=<replManager-Password> on both sides, let's say A and D (A first and then D). Then, I do initiate the replication by setting nsds5BeginReplicaRefresh to start on A.

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 5:57 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

Rich,

I have an setup like:

A <-----> B

/\ \ / /\ | \ / | | / | | / \ | | / \ | // \ /\ D <-----> C

At first, I do set the agreements up for the Ring A to B to C to B to A. This works. Then, I try to set the cross agreements from A to C and B to D up. This is where I run into this issue.

Let's have a look how I do those cross agreements. First I add an agreement on A for C. This is fine. Then, I do the same on C (for A) and I get the messages NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation On C and on A I get: [10/Aug/2010:17:12:37 -0400] - somehow, there are still 16 entries in the entry cache. :/ [10/Aug/2010:17:12:38 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:17:12:38 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0

Hope, this helps.

How do you do the replica init?

Thanks, -Reinhard -----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Reinhard Nappert Sent: Tuesday, August 10, 2010 2:42 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Rich, on the consumer, I see the following messages:

NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 12:41 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Hi,

I have seen the following message in the errors log file, when I set MMR agreements up:

[10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=base: 1 [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - multimaster_be_state_change: replica o=base is going offline; disabling replication [10/Aug/2010:11:46:46 -0400] - somehow, there are still 20 entries in the entrycache. :/ [10/Aug/2010:11:46:46 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:11:46:48 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 [10/Aug/2010:11:46:52 -0400] - Fedora-Directory/1.1.2 B2009.090.1643 starting up [10/Aug/2010:11:46:52 -0400] - Detected Disorderly Shutdown last time DirectoryServer was running, recovering database.

After I re-initialize the database from the supplier (setting attribute nsds5BeginReplicaRefresh to start of the agreement object), the database gets correctly imported.

Any idea, what is going on?

No, not sure. But if you can develop a reproducible test case, that would be helpful.

...

Thanks, -Reinhard

---

--

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

To explain it a bit easier, I define two "methods": 1. createAgreement(<remote ldap>): <-- creates locally replication agreement for remote ldap nsDS5ReplicaType=3 nsDS5Flags=1 nsDS5ReplicaId=<unique id> nsDS5ReplicaHost=<hostname of remote ldap> nsDS5ReplicaTransportInfo=LDAP nsDS5ReplicaPort=389 nsDS5ReplicaBindDN=<replManager-DN> nsDS5ReplicaBindMethod=SIMPLE nsDS5ReplicaCredentials=<replManager-Password>

2. initReplication(<local ldap>, <remote ldap>): <-- modifies the existing remote replication agreement for the local ldap nsds5BeginReplicaRefresh=start

So, the order is the following: 1. On A: createAgreement(B) 2. On B: createAgreement(A) 3. On B: initReplication(B, A) 4. On B: createAgreement(C) 5. On C: createAgreement(B) 6. On C: initReplication(C, B) 7. On C: createAgreement(D) 8. On D: createAgreement(C) 9. On D: initReplication(D, C) 10. On D: createAgreement(A) 11. On A: createAgreement(D) 12. On A: initReplication(A, D)

Now, I have the ring A<-->B<-->C<-->D<-->A. All of this works fine! Then, I want to create the cross-references from A to C and B to D 13. On A: createAgreement(C) 14. On C: createAgreement(A) 15. On C: initReplication(C, A)

After step 15, I run into this issue. The same thing happens, when I set B and D up. 16. On B: createAgreement(D) 17. On D: createAgreement(B) 18. On D: initReplication(D, B)

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:09 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

At first I create (besides the changelog and replica entry with nsDS5ReplicaType=3, nsDS5Flags=1 and an unique nsDS5ReplicaId) the shadowing agreement with nsDS5ReplicaHost=<hostname of remote server>, nsDS5ReplicaTransportInfo=LDAP, nsDS5ReplicaPort=389, nsDS5ReplicaBindDN=<replManager-DN>, nsDS5ReplicaBindMethod=SIMPLE, nsDS5ReplicaCredentials=<replManager-Password> on both sides, let's say A and D (A first and then D). Then, I do initiate the replication by setting nsds5BeginReplicaRefresh to start on A.

And you do that for A -> B, A -> C? How do you initialize D?

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 5:57 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Rich,

I have an setup like:

A <-----> B

/\ \ / /\ | \ / | | / | | / \ | | / \ | // \ /\ D <-----> C

At first, I do set the agreements up for the Ring A to B to C to B to A. This works. Then, I try to set the cross agreements from A to C and B to D up. This is where I run into this issue.

Let's have a look how I do those cross agreements. First I add an agreement on A for C. This is fine. Then, I do the same on C (for A) and I get the messages NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation On C and on A I get: [10/Aug/2010:17:12:37 -0400] - somehow, there are still 16 entries in the entry cache. :/ [10/Aug/2010:17:12:38 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:17:12:38 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0

Hope, this helps.

How do you do the replica init?

...

Thanks, -Reinhard -----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Reinhard Nappert Sent: Tuesday, August 10, 2010 2:42 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Rich, on the consumer, I see the following messages:

NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 12:41 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Hi,

I have seen the following message in the errors log file, when I set MMR agreements up:

[10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=base: 1 [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - multimaster_be_state_change: replica o=base is going offline; disabling replication [10/Aug/2010:11:46:46 -0400] - somehow, there are still 20 entries in the entrycache. :/ [10/Aug/2010:11:46:46 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:11:46:48 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 [10/Aug/2010:11:46:52 -0400] - Fedora-Directory/1.1.2 B2009.090.1643 starting up [10/Aug/2010:11:46:52 -0400] - Detected Disorderly Shutdown last time DirectoryServer was running, recovering database.

After I re-initialize the database from the supplier (setting attribute nsds5BeginReplicaRefresh to start of the agreement object), the database gets correctly imported.

Any idea, what is going on?

No, not sure. But if you can develop a reproducible test case, that would be helpful.

...

Thanks, -Reinhard

---

--

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

Rick,

Are you saying that, once I have replicated the data from A to B and from B to C and from C to D, I don't replicate it from D to A? If so, can you explain why? Anyway, this step works!

So, 15 and 18 are up-to-date at this stage. Since the entire setup is done kind of automatically, the setting of nsds5BeginReplicaRefresh to start is always done, if the corresponding agreement exists on the remote box. Is there a way to find out when I have to set nsds5BeginReplicaRefresh to start?

In any case, this does not explain that I fix the issue by resetting nsds5BeginReplicaRefresh to start, once I run into this issue.

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:37 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

To explain it a bit easier, I define two "methods":

1. createAgreement(<remote ldap>): <-- creates locally replication agreement for remote ldap nsDS5ReplicaType=3 nsDS5Flags=1 nsDS5ReplicaId=<unique id> nsDS5ReplicaHost=<hostname of remote ldap> nsDS5ReplicaTransportInfo=LDAP nsDS5ReplicaPort=389 nsDS5ReplicaBindDN=<replManager-DN> nsDS5ReplicaBindMethod=SIMPLE nsDS5ReplicaCredentials=<replManager-Password>

2. initReplication(<local ldap>, <remote ldap>): <-- modifies the existing remote replication agreement for the local ldap nsds5BeginReplicaRefresh=start

So, the order is the following:

1. On A: createAgreement(B)
2. On B: createAgreement(A)
3. On B: initReplication(B, A)
4. On B: createAgreement(C)
5. On C: createAgreement(B)
6. On C: initReplication(C, B)
7. On C: createAgreement(D)
8. On D: createAgreement(C)
9. On D: initReplication(D, C)
10. On D: createAgreement(A)
11. On A: createAgreement(D)
12. On A: initReplication(A, D)

12 is a problem - you don't initialize the master (A) you started from

Now, I have the ring A<-->B<-->C<-->D<-->A. All of this works fine! Then, I want to create the cross-references from A to C and B to D 13. On A: createAgreement(C) 14. On C: createAgreement(A) 15. On C: initReplication(C, A)

15 is a problem - C has already been initialized

After step 15, I run into this issue. The same thing happens, when I set B and D up. 16. On B: createAgreement(D) 17. On D: createAgreement(B) 18. On D: initReplication(D, B)

18 is a problem - D has already been initialized

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:09 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

At first I create (besides the changelog and replica entry with nsDS5ReplicaType=3, nsDS5Flags=1 and an unique nsDS5ReplicaId) the shadowing agreement with nsDS5ReplicaHost=<hostname of remote server>, nsDS5ReplicaTransportInfo=LDAP, nsDS5ReplicaPort=389, nsDS5ReplicaBindDN=<replManager-DN>, nsDS5ReplicaBindMethod=SIMPLE, nsDS5ReplicaCredentials=<replManager-Password> on both sides, let's say A and D (A first and then D). Then, I do initiate the replication by setting nsds5BeginReplicaRefresh to start on A.

And you do that for A -> B, A -> C? How do you initialize D?

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 5:57 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Rich,

I have an setup like:

A <-----> B

/\ \ / /\ | \ / | | / | | / \ | | / \ | // \ /\ D <-----> C

At first, I do set the agreements up for the Ring A to B to C to B to A. This works. Then, I try to set the cross agreements from A to C and B to D up. This is where I run into this issue.

Let's have a look how I do those cross agreements. First I add an agreement on A for C. This is fine. Then, I do the same on C (for A) and I get the messages NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation On C and on A I get: [10/Aug/2010:17:12:37 -0400] - somehow, there are still 16 entries in the entry cache. :/ [10/Aug/2010:17:12:38 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:17:12:38 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0

Hope, this helps.

How do you do the replica init?

...

Thanks, -Reinhard -----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Reinhard Nappert Sent: Tuesday, August 10, 2010 2:42 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Rich, on the consumer, I see the following messages:

NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 12:41 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Hi,

I have seen the following message in the errors log file, when I set MMR agreements up:

[10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=base: 1 [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - multimaster_be_state_change: replica o=base is going offline; disabling replication [10/Aug/2010:11:46:46 -0400] - somehow, there are still 20 entries in the entrycache. :/ [10/Aug/2010:11:46:46 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:11:46:48 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 [10/Aug/2010:11:46:52 -0400] - Fedora-Directory/1.1.2 B2009.090.1643 starting up [10/Aug/2010:11:46:52 -0400] - Detected Disorderly Shutdown last time DirectoryServer was running, recovering database.

After I re-initialize the database from the supplier (setting attribute nsds5BeginReplicaRefresh to start of the agreement object), the database gets correctly imported.

Any idea, what is going on?

No, not sure. But if you can develop a reproducible test case, that would be helpful.

...

Thanks, -Reinhard

---

--

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

So, Is there a way to find out if a server was used for the initialization of other servers?

I am still not convinced that this is the cause, because when I add another server as a consumer (E) to A and I do a initReplication(E, A) I run into the same issue.

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 11:12 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

Rick,

Are you saying that, once I have replicated the data from A to B and from B to C and from C to D, I don't replicate it from D to A? If so, can you explain why? Anyway, this step works!

If you replace the word "replicated" with "initialized", then yes, you don't initialize from D to A. Although it may work, I think it may introduce subtle errors, such as the ones you see.

So, 15 and 18 are up-to-date at this stage. Since the entire setup is done kind of automatically, the setting of nsds5BeginReplicaRefresh to start is always done, if the corresponding agreement exists on the remote box. Is there a way to find out when I have to set nsds5BeginReplicaRefresh to start?

In any case, this does not explain that I fix the issue by resetting nsds5BeginReplicaRefresh to start, once I run into this issue.

I'm not exactly sure why you are seeing the errors you are seeing, nor why you can fix the issue with start refresh. I do know that you should not re-initialize a server that has been used to initialize other servers.

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:37 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

To explain it a bit easier, I define two "methods":

1. createAgreement(<remote ldap>): <-- creates locally replication agreement for remote ldap nsDS5ReplicaType=3 nsDS5Flags=1 nsDS5ReplicaId=<unique id> nsDS5ReplicaHost=<hostname of remote ldap> nsDS5ReplicaTransportInfo=LDAP nsDS5ReplicaPort=389 nsDS5ReplicaBindDN=<replManager-DN> nsDS5ReplicaBindMethod=SIMPLE nsDS5ReplicaCredentials=<replManager-Password>

2. initReplication(<local ldap>, <remote ldap>): <-- modifies the existing remote replication agreement for the local ldap nsds5BeginReplicaRefresh=start

So, the order is the following:

1. On A: createAgreement(B)
2. On B: createAgreement(A)
3. On B: initReplication(B, A)
4. On B: createAgreement(C)
5. On C: createAgreement(B)
6. On C: initReplication(C, B)
7. On C: createAgreement(D)
8. On D: createAgreement(C)
9. On D: initReplication(D, C)
10. On D: createAgreement(A)
11. On A: createAgreement(D)
12. On A: initReplication(A, D)

12 is a problem - you don't initialize the master (A) you started from

...

Now, I have the ring A<-->B<-->C<-->D<-->A. All of this works fine! Then, I want to create the cross-references from A to C and B to D 13. On A: createAgreement(C) 14. On C: createAgreement(A) 15. On C: initReplication(C, A)

15 is a problem - C has already been initialized

...

After step 15, I run into this issue. The same thing happens, when I set B and D up. 16. On B: createAgreement(D) 17. On D: createAgreement(B) 18. On D: initReplication(D, B)

18 is a problem - D has already been initialized

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:09 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

At first I create (besides the changelog and replica entry with nsDS5ReplicaType=3, nsDS5Flags=1 and an unique nsDS5ReplicaId) the shadowing agreement with nsDS5ReplicaHost=<hostname of remote server>, nsDS5ReplicaTransportInfo=LDAP, nsDS5ReplicaPort=389, nsDS5ReplicaBindDN=<replManager-DN>, nsDS5ReplicaBindMethod=SIMPLE, nsDS5ReplicaCredentials=<replManager-Password> on both sides, let's say A and D (A first and then D). Then, I do initiate the replication by setting nsds5BeginReplicaRefresh to start on A.

And you do that for A -> B, A -> C? How do you initialize D?

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 5:57 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Rich,

I have an setup like:

A <-----> B

/\ \ / /\ | \ / | | / | | / \ | | / \ | // \ /\ D <-----> C

At first, I do set the agreements up for the Ring A to B to C to B to A. This works. Then, I try to set the cross agreements from A to C and B to D up. This is where I run into this issue.

Let's have a look how I do those cross agreements. First I add an agreement on A for C. This is fine. Then, I do the same on C (for A) and I get the messages NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation On C and on A I get: [10/Aug/2010:17:12:37 -0400] - somehow, there are still 16 entries in the entry cache. :/ [10/Aug/2010:17:12:38 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:17:12:38 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0

Hope, this helps.

How do you do the replica init?

...

Thanks, -Reinhard -----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Reinhard Nappert Sent: Tuesday, August 10, 2010 2:42 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Rich, on the consumer, I see the following messages:

NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 12:41 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Hi,

I have seen the following message in the errors log file, when I set MMR agreements up:

[10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=base: 1 [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - multimaster_be_state_change: replica o=base is going offline; disabling replication [10/Aug/2010:11:46:46 -0400] - somehow, there are still 20 entries in the entrycache. :/ [10/Aug/2010:11:46:46 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:11:46:48 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 [10/Aug/2010:11:46:52 -0400] - Fedora-Directory/1.1.2 B2009.090.1643 starting up [10/Aug/2010:11:46:52 -0400] - Detected Disorderly Shutdown last time DirectoryServer was running, recovering database.

After I re-initialize the database from the supplier (setting attribute nsds5BeginReplicaRefresh to start of the agreement object), the database gets correctly imported.

Any idea, what is going on?

No, not sure. But if you can develop a reproducible test case, that would be helpful.

...

Thanks, -Reinhard

---

--

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

Thanks Rich.

When does the server delete the RUV entry? After I set the server back to "standalone", by removing the changelog entry, all agreements and the replica entry, I still see the RUV entry: ldapsearch -D <Directory Manager> -w <password> -b dc=your,dc=suffix -x -LLL "(&(nsuniqueid=*)(objectclass=nsTombstone))" nsds50ruv dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=your,dc=suffix nsds50ruv: {replicageneration} 4c61bf2e000000010000 nsds50ruv: {replica 7 ldap://yale:389} nsds50ruv: {replica 6 ldap://mustrum:389} nsds50ruv: {replica 1 ldap://louise:389} 4c62a60c000000010000 4c62a60c000000010000 nsds50ruv: {replica 4 ldap://nix:389} 4c61c172000000040000 4c62c59c000000040000 nsds50ruv: {replica 3 ldap://yale:389} 4c62a5c5000000030000 4c62a5f1000000030000 nsds50ruv: {replica 2 ldap://mustrum:389} nsds50ruv: {replica 8 ldap://nix:389} 4c62d029000000080000 4c62efcc000000080000 nsds50ruv: {replica 5 ldap://louise:389} 4c62d1b9000000050000 4c62d1b9000000050000

I would expect that this would have been deleted by the server. If not, this appraoch does not help.

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 12:32 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

So, Is there a way to find out if a server was used for the initialization of other servers?

You can query the RUV entry in the server: ldapsearch -s one -b "dc=your,dc=suffix" "(objectclass=nsTombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)" The generation is a CSN. The first 8 bytes are the timestamp. The next 4 bytes are the sequence number. The next 4 bytes are the replica ID of the original master. If there is no RUV, or the generation is missing, the server has either not been configured for replication, or has not been initialized.

I am still not convinced that this is the cause, because when I add another server as a consumer (E) to A and I do a initReplication(E, A) I run into the same issue.

If you initReplication(A, D), then initReplication(E, A) you may run into the issue.

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 11:12 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Rick,

Are you saying that, once I have replicated the data from A to B and from B to C and from C to D, I don't replicate it from D to A? If so, can you explain why? Anyway, this step works!

If you replace the word "replicated" with "initialized", then yes, you don't initialize from D to A. Although it may work, I think it may introduce subtle errors, such as the ones you see.

...

So, 15 and 18 are up-to-date at this stage. Since the entire setup is done kind of automatically, the setting of nsds5BeginReplicaRefresh to start is always done, if the corresponding agreement exists on the remote box. Is there a way to find out when I have to set nsds5BeginReplicaRefresh to start?

In any case, this does not explain that I fix the issue by resetting nsds5BeginReplicaRefresh to start, once I run into this issue.

I'm not exactly sure why you are seeing the errors you are seeing, nor why you can fix the issue with start refresh. I do know that you should not re-initialize a server that has been used to initialize other servers.

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:37 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

To explain it a bit easier, I define two "methods":

1. createAgreement(<remote ldap>): <-- creates locally replication agreement for remote ldap nsDS5ReplicaType=3 nsDS5Flags=1 nsDS5ReplicaId=<unique id> nsDS5ReplicaHost=<hostname of remote ldap> nsDS5ReplicaTransportInfo=LDAP nsDS5ReplicaPort=389 nsDS5ReplicaBindDN=<replManager-DN> nsDS5ReplicaBindMethod=SIMPLE nsDS5ReplicaCredentials=<replManager-Password>

2. initReplication(<local ldap>, <remote ldap>): <-- modifies the existing remote replication agreement for the local ldap nsds5BeginReplicaRefresh=start

So, the order is the following:

1. On A: createAgreement(B)
2. On B: createAgreement(A)
3. On B: initReplication(B, A)
4. On B: createAgreement(C)
5. On C: createAgreement(B)
6. On C: initReplication(C, B)
7. On C: createAgreement(D)
8. On D: createAgreement(C)
9. On D: initReplication(D, C)
10. On D: createAgreement(A)
11. On A: createAgreement(D)
12. On A: initReplication(A, D)

12 is a problem - you don't initialize the master (A) you started from

...

Now, I have the ring A<-->B<-->C<-->D<-->A. All of this works fine! Then, I want to create the cross-references from A to C and B to D 13. On A: createAgreement(C) 14. On C: createAgreement(A) 15. On C: initReplication(C, A)

15 is a problem - C has already been initialized

...

After step 15, I run into this issue. The same thing happens, when I set B and D up. 16. On B: createAgreement(D) 17. On D: createAgreement(B) 18. On D: initReplication(D, B)

18 is a problem - D has already been initialized

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:09 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

At first I create (besides the changelog and replica entry with nsDS5ReplicaType=3, nsDS5Flags=1 and an unique nsDS5ReplicaId) the shadowing agreement with nsDS5ReplicaHost=<hostname of remote server>, nsDS5ReplicaTransportInfo=LDAP, nsDS5ReplicaPort=389, nsDS5ReplicaBindDN=<replManager-DN>, nsDS5ReplicaBindMethod=SIMPLE, nsDS5ReplicaCredentials=<replManager-Password> on both sides, let's say A and D (A first and then D). Then, I do initiate the replication by setting nsds5BeginReplicaRefresh to start on A.

And you do that for A -> B, A -> C? How do you initialize D?

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 5:57 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Rich,

I have an setup like:

A <-----> B

/\ \ / /\ | \ / | | / | | / \ | | / \ | // \ /\ D <-----> C

At first, I do set the agreements up for the Ring A to B to C to B to A. This works. Then, I try to set the cross agreements from A to C and B to D up. This is where I run into this issue.

Let's have a look how I do those cross agreements. First I add an agreement on A for C. This is fine. Then, I do the same on C (for A) and I get the messages NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation On C and on A I get: [10/Aug/2010:17:12:37 -0400] - somehow, there are still 16 entries in the entry cache. :/ [10/Aug/2010:17:12:38 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:17:12:38 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0

Hope, this helps.

How do you do the replica init?

...

Thanks, -Reinhard -----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Reinhard Nappert Sent: Tuesday, August 10, 2010 2:42 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Rich, on the consumer, I see the following messages:

NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 12:41 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Hi,

I have seen the following message in the errors log file, when I set MMR agreements up:

[10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=base: 1 [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - multimaster_be_state_change: replica o=base is going offline; disabling replication [10/Aug/2010:11:46:46 -0400] - somehow, there are still 20 entries in the entrycache. :/ [10/Aug/2010:11:46:46 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:11:46:48 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 [10/Aug/2010:11:46:52 -0400] - Fedora-Directory/1.1.2 B2009.090.1643 starting up [10/Aug/2010:11:46:52 -0400] - Detected Disorderly Shutdown last time DirectoryServer was running, recovering database.

After I re-initialize the database from the supplier (setting attribute nsds5BeginReplicaRefresh to start of the agreement object), the database gets correctly imported.

Any idea, what is going on?

No, not sure. But if you can develop a reproducible test case, that would be helpful.

...

Thanks, -Reinhard

---

--

--

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

Actually I tried this.

First, I just deleted the attributes nsds50ruv, which was fine, from an ldap operational point of view, but when I wanted to set replication up again, the server complained with an operation error (nds50ruv attribute missing). Then I though I just delete the entire entry (nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=your,dc=suffix). This crashes the ldap server!

Why does it not help, if I don't get rid of it? Since the info stays in there, even after replication was disabled, I can not use this entry in order to determine whether the server was already initialized, when I enable replication for a second time.

I think, I found a solution to this by using some objects from my internal framework.

However, the ldap server should not crash, when I try to delete this entry

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 4:44 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

Thanks Rich.

When does the server delete the RUV entry? After I set the server back to "standalone", by removing the changelog entry, all agreements and the replica entry, I still see the RUV entry: ldapsearch -D <Directory Manager> -w <password> -b dc=your,dc=suffix -x -LLL "(&(nsuniqueid=*)(objectclass=nsTombstone))" nsds50ruv dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=your,dc=suffix nsds50ruv: {replicageneration} 4c61bf2e000000010000 nsds50ruv: {replica 7 ldap://yale:389} nsds50ruv: {replica 6 ldap://mustrum:389} nsds50ruv: {replica 1 ldap://louise:389} 4c62a60c000000010000 4c62a60c000000010000 nsds50ruv: {replica 4 ldap://nix:389} 4c61c172000000040000 4c62c59c000000040000 nsds50ruv: {replica 3 ldap://yale:389} 4c62a5c5000000030000 4c62a5f1000000030000 nsds50ruv: {replica 2 ldap://mustrum:389} nsds50ruv: {replica 8 ldap://nix:389} 4c62d029000000080000 4c62efcc000000080000 nsds50ruv: {replica 5 ldap://louise:389} 4c62d1b9000000050000 4c62d1b9000000050000

I would expect that this would have been deleted by the server.

No, but you should be able to manually delete it. It doesn't do anything if you're using replication.

If not, this appraoch does not help.

Why?

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 12:32 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

So, Is there a way to find out if a server was used for the initialization of other servers?

You can query the RUV entry in the server: ldapsearch -s one -b "dc=your,dc=suffix" "(objectclass=nsTombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)" The generation is a CSN. The first 8 bytes are the timestamp. The next 4 bytes are the sequence number. The next 4 bytes are the replica ID of the original master. If there is no RUV, or the generation is missing, the server has either not been configured for replication, or has not been initialized.

...

I am still not convinced that this is the cause, because when I add another server as a consumer (E) to A and I do a initReplication(E, A) I run into the same issue.

If you initReplication(A, D), then initReplication(E, A) you may run into the issue.

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 11:12 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Rick,

Are you saying that, once I have replicated the data from A to B and from B to C and from C to D, I don't replicate it from D to A? If so, can you explain why? Anyway, this step works!

If you replace the word "replicated" with "initialized", then yes, you don't initialize from D to A. Although it may work, I think it may introduce subtle errors, such as the ones you see.

...

So, 15 and 18 are up-to-date at this stage. Since the entire setup is done kind of automatically, the setting of nsds5BeginReplicaRefresh to start is always done, if the corresponding agreement exists on the remote box. Is there a way to find out when I have to set nsds5BeginReplicaRefresh to start?

In any case, this does not explain that I fix the issue by resetting nsds5BeginReplicaRefresh to start, once I run into this issue.

I'm not exactly sure why you are seeing the errors you are seeing, nor why you can fix the issue with start refresh. I do know that you should not re-initialize a server that has been used to initialize other servers.

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:37 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

To explain it a bit easier, I define two "methods":

1. createAgreement(<remote ldap>): <-- creates locally replication agreement for remote ldap nsDS5ReplicaType=3 nsDS5Flags=1 nsDS5ReplicaId=<unique id> nsDS5ReplicaHost=<hostname of remote ldap> nsDS5ReplicaTransportInfo=LDAP nsDS5ReplicaPort=389 nsDS5ReplicaBindDN=<replManager-DN> nsDS5ReplicaBindMethod=SIMPLE nsDS5ReplicaCredentials=<replManager-Password>

2. initReplication(<local ldap>, <remote ldap>): <-- modifies the existing remote replication agreement for the local ldap nsds5BeginReplicaRefresh=start

So, the order is the following:

1. On A: createAgreement(B)
2. On B: createAgreement(A)
3. On B: initReplication(B, A)
4. On B: createAgreement(C)
5. On C: createAgreement(B)
6. On C: initReplication(C, B)
7. On C: createAgreement(D)
8. On D: createAgreement(C)
9. On D: initReplication(D, C)
10. On D: createAgreement(A)
11. On A: createAgreement(D)
12. On A: initReplication(A, D)

12 is a problem - you don't initialize the master (A) you started from

...

Now, I have the ring A<-->B<-->C<-->D<-->A. All of this works fine! Then, I want to create the cross-references from A to C and B to D 13. On A: createAgreement(C) 14. On C: createAgreement(A) 15. On C: initReplication(C, A)

15 is a problem - C has already been initialized

...

After step 15, I run into this issue. The same thing happens, when I set B and D up. 16. On B: createAgreement(D) 17. On D: createAgreement(B) 18. On D: initReplication(D, B)

18 is a problem - D has already been initialized

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:09 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

At first I create (besides the changelog and replica entry with nsDS5ReplicaType=3, nsDS5Flags=1 and an unique nsDS5ReplicaId) the shadowing agreement with nsDS5ReplicaHost=<hostname of remote server>, nsDS5ReplicaTransportInfo=LDAP, nsDS5ReplicaPort=389, nsDS5ReplicaBindDN=<replManager-DN>, nsDS5ReplicaBindMethod=SIMPLE, nsDS5ReplicaCredentials=<replManager-Password> on both sides, let's say A and D (A first and then D). Then, I do initiate the replication by setting nsds5BeginReplicaRefresh to start on A.

And you do that for A -> B, A -> C? How do you initialize D?

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 5:57 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Rich,

I have an setup like:

A <-----> B

/\ \ / /\ | \ / | | / | | / \ | | / \ | // \ /\ D <-----> C

At first, I do set the agreements up for the Ring A to B to C to B to A. This works. Then, I try to set the cross agreements from A to C and B to D up. This is where I run into this issue.

Let's have a look how I do those cross agreements. First I add an agreement on A for C. This is fine. Then, I do the same on C (for A) and I get the messages NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation On C and on A I get: [10/Aug/2010:17:12:37 -0400] - somehow, there are still 16 entries in the entry cache. :/ [10/Aug/2010:17:12:38 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:17:12:38 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0

Hope, this helps.

How do you do the replica init?

...

Thanks, -Reinhard -----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Reinhard Nappert Sent: Tuesday, August 10, 2010 2:42 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Rich, on the consumer, I see the following messages:

NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 12:41 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

> Hi, > > I have seen the following message in the errors log file, when I > set MMR agreements up: > > [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - > repl_set_mtn_referrals: could not set referrals for replica o=base: > 1 > [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - > multimaster_be_state_change: replica o=base is going offline; > disabling replication > [10/Aug/2010:11:46:46 -0400] - somehow, there are still 20 > entries in the entrycache. :/ > [10/Aug/2010:11:46:46 -0400] - WARNING: Import is running with > nsslapd-db-private-import-mem on; No other process is allowed to > access the database > [10/Aug/2010:11:46:48 -0400] - BAD CACHE ASSERTION at > ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 > [10/Aug/2010:11:46:52 -0400] - Fedora-Directory/1.1.2 > B2009.090.1643 starting up > [10/Aug/2010:11:46:52 -0400] - Detected Disorderly Shutdown last > time DirectoryServer was running, recovering database. > > After I re-initialize the database from the supplier (setting > attribute nsds5BeginReplicaRefresh to start of the agreement > object), the database gets correctly imported. > > Any idea, what is going on? > > > > > > No, not sure. But if you can develop a reproducible test case, that would be helpful.

> Thanks, > -Reinhard > > ---------------------------------------------------------------- > - > -- > - > - > - > -- > > -- > 389 users mailing list > 389-users@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/389-users > > > > >

>

389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

One more question: Can I read/modify the nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=your,dc=suffix object, without being logged in as "Directory Manager". If so, what kind of aci's need I to set?

Thanks, -Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 5:41 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

Actually I tried this.

First, I just deleted the attributes nsds50ruv, which was fine, from an ldap operational point of view, but when I wanted to set replication up again, the server complained with an operation error (nds50ruv attribute missing).

Right, you cannot just delete the attribute, you have to delete the entire entry.

Then I though I just delete the entire entry (nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=your,dc=suffix). This crashes the ldap server!

Why does it not help, if I don't get rid of it? Since the info stays in there, even after replication was disabled, I can not use this entry in order to determine whether the server was already initialized, when I enable replication for a second time.

I think, I found a solution to this by using some objects from my internal framework.

However, the ldap server should not crash, when I try to delete this entry

I think we fixed that crashing bug a while ago. Can you post a stack trace?

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 4:44 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Thanks Rich.

When does the server delete the RUV entry? After I set the server back to "standalone", by removing the changelog entry, all agreements and the replica entry, I still see the RUV entry: ldapsearch -D <Directory Manager> -w <password> -b dc=your,dc=suffix -x -LLL "(&(nsuniqueid=*)(objectclass=nsTombstone))" nsds50ruv dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=your,dc=suffix nsds50ruv: {replicageneration} 4c61bf2e000000010000 nsds50ruv: {replica 7 ldap://yale:389} nsds50ruv: {replica 6 ldap://mustrum:389} nsds50ruv: {replica 1 ldap://louise:389} 4c62a60c000000010000 4c62a60c000000010000 nsds50ruv: {replica 4 ldap://nix:389} 4c61c172000000040000 4c62c59c000000040000 nsds50ruv: {replica 3 ldap://yale:389} 4c62a5c5000000030000 4c62a5f1000000030000 nsds50ruv: {replica 2 ldap://mustrum:389} nsds50ruv: {replica 8 ldap://nix:389} 4c62d029000000080000 4c62efcc000000080000 nsds50ruv: {replica 5 ldap://louise:389} 4c62d1b9000000050000 4c62d1b9000000050000

I would expect that this would have been deleted by the server.

No, but you should be able to manually delete it. It doesn't do anything if you're using replication.

...

If not, this appraoch does not help.

Why?

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 12:32 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

So, Is there a way to find out if a server was used for the initialization of other servers?

You can query the RUV entry in the server: ldapsearch -s one -b "dc=your,dc=suffix" "(objectclass=nsTombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)" The generation is a CSN. The first 8 bytes are the timestamp. The next 4 bytes are the sequence number. The next 4 bytes are the replica ID of the original master. If there is no RUV, or the generation is missing, the server has either not been configured for replication, or has not been initialized.

...

I am still not convinced that this is the cause, because when I add another server as a consumer (E) to A and I do a initReplication(E, A) I run into the same issue.

If you initReplication(A, D), then initReplication(E, A) you may run into the issue.

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 11:12 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Rick,

Are you saying that, once I have replicated the data from A to B and from B to C and from C to D, I don't replicate it from D to A? If so, can you explain why? Anyway, this step works!

If you replace the word "replicated" with "initialized", then yes, you don't initialize from D to A. Although it may work, I think it may introduce subtle errors, such as the ones you see.

...

So, 15 and 18 are up-to-date at this stage. Since the entire setup is done kind of automatically, the setting of nsds5BeginReplicaRefresh to start is always done, if the corresponding agreement exists on the remote box. Is there a way to find out when I have to set nsds5BeginReplicaRefresh to start?

In any case, this does not explain that I fix the issue by resetting nsds5BeginReplicaRefresh to start, once I run into this issue.

I'm not exactly sure why you are seeing the errors you are seeing, nor why you can fix the issue with start refresh. I do know that you should not re-initialize a server that has been used to initialize other servers.

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:37 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

To explain it a bit easier, I define two "methods":

1. createAgreement(<remote ldap>): <-- creates locally replication agreement for remote ldap nsDS5ReplicaType=3 nsDS5Flags=1 nsDS5ReplicaId=<unique id> nsDS5ReplicaHost=<hostname of remote ldap> nsDS5ReplicaTransportInfo=LDAP nsDS5ReplicaPort=389 nsDS5ReplicaBindDN=<replManager-DN> nsDS5ReplicaBindMethod=SIMPLE nsDS5ReplicaCredentials=<replManager-Password>

2. initReplication(<local ldap>, <remote ldap>): <-- modifies the existing remote replication agreement for the local ldap nsds5BeginReplicaRefresh=start

So, the order is the following:

1. On A: createAgreement(B)
2. On B: createAgreement(A)
3. On B: initReplication(B, A)
4. On B: createAgreement(C)
5. On C: createAgreement(B)
6. On C: initReplication(C, B)
7. On C: createAgreement(D)
8. On D: createAgreement(C)
9. On D: initReplication(D, C)
10. On D: createAgreement(A)
11. On A: createAgreement(D)
12. On A: initReplication(A, D)

12 is a problem - you don't initialize the master (A) you started from

...

Now, I have the ring A<-->B<-->C<-->D<-->A. All of this works fine! Then, I want to create the cross-references from A to C and B to D 13. On A: createAgreement(C) 14. On C: createAgreement(A) 15. On C: initReplication(C, A)

15 is a problem - C has already been initialized

...

After step 15, I run into this issue. The same thing happens, when I set B and D up. 16. On B: createAgreement(D) 17. On D: createAgreement(B) 18. On D: initReplication(D, B)

18 is a problem - D has already been initialized

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:09 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

At first I create (besides the changelog and replica entry with nsDS5ReplicaType=3, nsDS5Flags=1 and an unique nsDS5ReplicaId) the shadowing agreement with nsDS5ReplicaHost=<hostname of remote server>, nsDS5ReplicaTransportInfo=LDAP, nsDS5ReplicaPort=389, nsDS5ReplicaBindDN=<replManager-DN>, nsDS5ReplicaBindMethod=SIMPLE, nsDS5ReplicaCredentials=<replManager-Password> on both sides, let's say A and D (A first and then D). Then, I do initiate the replication by setting nsds5BeginReplicaRefresh to start on A.

And you do that for A -> B, A -> C? How do you initialize D?

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 5:57 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

> Rich, > > I have an setup like: > > A <-----> B > /\ \ / /\ > | \ / | > | / | > | / \ | > | / \ | > // \ /\ > D <-----> C > > At first, I do set the agreements up for the Ring A to B to C to B to A. This works. Then, I try to set the cross agreements from A to C and B to D up. This is where I run into this issue. > > Let's have a look how I do those cross agreements. First I add > an agreement on A for C. This is fine. Then, I do the same on C > (for > A) and I get the messages NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation On C and on A I get: > [10/Aug/2010:17:12:37 -0400] - somehow, there are still 16 > entries in the entry cache. :/ > [10/Aug/2010:17:12:38 -0400] - WARNING: Import is running with > nsslapd-db-private-import-mem on; No other process is allowed to > access the database > [10/Aug/2010:17:12:38 -0400] - BAD CACHE ASSERTION at > ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 > > > Hope, this helps. > > > > > > > How do you do the replica init?

> Thanks, > -Reinhard > -----Original Message----- > From: 389-users-bounces@lists.fedoraproject.org > [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of > Reinhard Nappert > Sent: Tuesday, August 10, 2010 2:42 PM > To: General discussion list for the 389 Directory server project. > Subject: Re: [389-users] Multi-Master setup > > Rich, on the consumer, I see the following messages: > > NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): > Received error 89: NULL for total update operation > > -Reinhard > > -----Original Message----- > From: 389-users-bounces@lists.fedoraproject.org > [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of > Rich Megginson > Sent: Tuesday, August 10, 2010 12:41 PM > To: General discussion list for the 389 Directory server project. > Subject: Re: [389-users] Multi-Master setup > > Reinhard Nappert wrote: > > > > > > > >> Hi, >> >> I have seen the following message in the errors log file, when >> I set MMR agreements up: >> >> [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - >> repl_set_mtn_referrals: could not set referrals for replica o=base: >> 1 >> [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - >> multimaster_be_state_change: replica o=base is going offline; >> disabling replication >> [10/Aug/2010:11:46:46 -0400] - somehow, there are still 20 >> entries in the entrycache. :/ >> [10/Aug/2010:11:46:46 -0400] - WARNING: Import is running with >> nsslapd-db-private-import-mem on; No other process is allowed >> to access the database >> [10/Aug/2010:11:46:48 -0400] - BAD CACHE ASSERTION at >> ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 >> [10/Aug/2010:11:46:52 -0400] - Fedora-Directory/1.1.2 >> B2009.090.1643 starting up >> [10/Aug/2010:11:46:52 -0400] - Detected Disorderly Shutdown >> last time DirectoryServer was running, recovering database. >> >> After I re-initialize the database from the supplier (setting >> attribute nsds5BeginReplicaRefresh to start of the agreement >> object), the database gets correctly imported. >> >> Any idea, what is going on? >> >> >> >> >> >> >> > No, not sure. But if you can develop a reproducible test case, that would be helpful. > > > > > > > >> Thanks, >> -Reinhard >> >> --------------------------------------------------------------- >> - >> - >> -- >> - >> - >> - >> -- >> >> -- >> 389 users mailing list >> 389-users@lists.fedoraproject.org >> https://admin.fedoraproject.org/mailman/listinfo/389-users >> >> >> >> >> >> >> > -- > 389 users mailing list > 389-users@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/389-users > -- > 389 users mailing list > 389-users@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/389-users > -- > 389 users mailing list > 389-users@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/389-users > > > > > >

>

389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

Reinhard Nappert wrote:

Rich,

I did some additional tests regarding replicationIds.

Let's say, I just have two MM A <--> B. I start configuring the replica and agreement on A and assign id 1. Then I do the same for B with the id 2. Everything is fine. Then, I disable on both boxes the replication. Then, I start setting the same thing up, but I start with B and assign 1 as id. A gets 2 as id assigned. Now, the replication fails with the message: "Unable to acquire replica: error: duplicate replica ID detected"

I am pretty sure that it has to do with the RUV entry "nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=your,dc=suffix", because it still shows:

dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff, dc=your,dc=suffix objectClass: top objectClass: nsTombstone objectClass: extensibleobject nsds50ruv: {replicageneration} 4c6445e4000000010000 nsds50ruv: {replica 1 ldap://A:389} nsds50ruv: {replica 2 ldap://B:389} nsruvReplicaLastModified: {replica 1 ldap://A:389} 00000000 nsruvReplicaLastModified: {replica 2 ldap://B:389} 00000000

My replica configuration objects use the correct ids (1 for B) and (2 for A). All this said, I believe the server should internally delete the RUV entry, once the replica configuration object is deleted.

Ok. Please file a bug.

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Reinhard Nappert Sent: Thursday, August 12, 2010 2:56 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

One more question: Can I read/modify the nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=your,dc=suffix object, without being logged in as "Directory Manager". If so, what kind of aci's need I to set?

Thanks, -Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 5:41 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Actually I tried this.

First, I just deleted the attributes nsds50ruv, which was fine, from an ldap operational point of view, but when I wanted to set replication up again, the server complained with an operation error (nds50ruv attribute missing).

Right, you cannot just delete the attribute, you have to delete the entire entry.

...

Then I though I just delete the entire entry (nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=your,dc=suffix). This crashes the ldap server!

Why does it not help, if I don't get rid of it? Since the info stays in there, even after replication was disabled, I can not use this entry in order to determine whether the server was already initialized, when I enable replication for a second time.

I think, I found a solution to this by using some objects from my internal framework.

However, the ldap server should not crash, when I try to delete this entry

I think we fixed that crashing bug a while ago. Can you post a stack trace?

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 4:44 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Thanks Rich.

When does the server delete the RUV entry? After I set the server back to "standalone", by removing the changelog entry, all agreements and the replica entry, I still see the RUV entry: ldapsearch -D <Directory Manager> -w <password> -b dc=your,dc=suffix -x -LLL "(&(nsuniqueid=*)(objectclass=nsTombstone))" nsds50ruv dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=your,dc=suffix nsds50ruv: {replicageneration} 4c61bf2e000000010000 nsds50ruv: {replica 7 ldap://yale:389} nsds50ruv: {replica 6 ldap://mustrum:389} nsds50ruv: {replica 1 ldap://louise:389} 4c62a60c000000010000 4c62a60c000000010000 nsds50ruv: {replica 4 ldap://nix:389} 4c61c172000000040000 4c62c59c000000040000 nsds50ruv: {replica 3 ldap://yale:389} 4c62a5c5000000030000 4c62a5f1000000030000 nsds50ruv: {replica 2 ldap://mustrum:389} nsds50ruv: {replica 8 ldap://nix:389} 4c62d029000000080000 4c62efcc000000080000 nsds50ruv: {replica 5 ldap://louise:389} 4c62d1b9000000050000 4c62d1b9000000050000

I would expect that this would have been deleted by the server.

No, but you should be able to manually delete it. It doesn't do anything if you're using replication.

...

If not, this appraoch does not help.

Why?

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 12:32 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

So, Is there a way to find out if a server was used for the initialization of other servers?

You can query the RUV entry in the server: ldapsearch -s one -b "dc=your,dc=suffix" "(objectclass=nsTombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)" The generation is a CSN. The first 8 bytes are the timestamp. The next 4 bytes are the sequence number. The next 4 bytes are the replica ID of the original master. If there is no RUV, or the generation is missing, the server has either not been configured for replication, or has not been initialized.

...

I am still not convinced that this is the cause, because when I add another server as a consumer (E) to A and I do a initReplication(E, A) I run into the same issue.

If you initReplication(A, D), then initReplication(E, A) you may run into the issue.

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 11:12 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Rick,

Are you saying that, once I have replicated the data from A to B and from B to C and from C to D, I don't replicate it from D to A? If so, can you explain why? Anyway, this step works!

If you replace the word "replicated" with "initialized", then yes, you don't initialize from D to A. Although it may work, I think it may introduce subtle errors, such as the ones you see.

...

So, 15 and 18 are up-to-date at this stage. Since the entire setup is done kind of automatically, the setting of nsds5BeginReplicaRefresh to start is always done, if the corresponding agreement exists on the remote box. Is there a way to find out when I have to set nsds5BeginReplicaRefresh to start?

In any case, this does not explain that I fix the issue by resetting nsds5BeginReplicaRefresh to start, once I run into this issue.

I'm not exactly sure why you are seeing the errors you are seeing, nor why you can fix the issue with start refresh. I do know that you should not re-initialize a server that has been used to initialize other servers.

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:37 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

To explain it a bit easier, I define two "methods":

1. createAgreement(<remote ldap>): <-- creates locally replication agreement for remote ldap nsDS5ReplicaType=3 nsDS5Flags=1 nsDS5ReplicaId=<unique id> nsDS5ReplicaHost=<hostname of remote ldap> nsDS5ReplicaTransportInfo=LDAP nsDS5ReplicaPort=389 nsDS5ReplicaBindDN=<replManager-DN> nsDS5ReplicaBindMethod=SIMPLE nsDS5ReplicaCredentials=<replManager-Password>

2. initReplication(<local ldap>, <remote ldap>): <-- modifies the existing remote replication agreement for the local ldap nsds5BeginReplicaRefresh=start

So, the order is the following:

1. On A: createAgreement(B)
2. On B: createAgreement(A)
3. On B: initReplication(B, A)
4. On B: createAgreement(C)
5. On C: createAgreement(B)
6. On C: initReplication(C, B)
7. On C: createAgreement(D)
8. On D: createAgreement(C)
9. On D: initReplication(D, C)
10. On D: createAgreement(A)
11. On A: createAgreement(D)
12. On A: initReplication(A, D)

12 is a problem - you don't initialize the master (A) you started from

...

Now, I have the ring A<-->B<-->C<-->D<-->A. All of this works fine! Then, I want to create the cross-references from A to C and B to D 13. On A: createAgreement(C) 14. On C: createAgreement(A) 15. On C: initReplication(C, A)

15 is a problem - C has already been initialized

...

After step 15, I run into this issue. The same thing happens, when I set B and D up. 16. On B: createAgreement(D) 17. On D: createAgreement(B) 18. On D: initReplication(D, B)

18 is a problem - D has already been initialized

...

-Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Wednesday, August 11, 2010 10:09 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

> At first I create (besides the changelog and replica entry with nsDS5ReplicaType=3, nsDS5Flags=1 and an unique nsDS5ReplicaId) the shadowing agreement with nsDS5ReplicaHost=<hostname of remote server>, nsDS5ReplicaTransportInfo=LDAP, nsDS5ReplicaPort=389, nsDS5ReplicaBindDN=<replManager-DN>, nsDS5ReplicaBindMethod=SIMPLE, nsDS5ReplicaCredentials=<replManager-Password> on both sides, let's say A and D (A first and then D). > Then, I do initiate the replication by setting nsds5BeginReplicaRefresh to start on A. > > > > > > > And you do that for A -> B, A -> C? How do you initialize D?

> -Reinhard > > -----Original Message----- > From: 389-users-bounces@lists.fedoraproject.org > [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of > Rich Megginson > Sent: Tuesday, August 10, 2010 5:57 PM > To: General discussion list for the 389 Directory server project. > Subject: Re: [389-users] Multi-Master setup > > Reinhard Nappert wrote: > > > > > > > >> Rich, >> >> I have an setup like: >> >> A <-----> B >> /\ \ / /\ >> | \ / | >> | / | >> | / \ | >> | / \ | >> // \ /\ >> D <-----> C >> >> At first, I do set the agreements up for the Ring A to B to C to B to A. This works. Then, I try to set the cross agreements from A to C and B to D up. This is where I run into this issue. >> >> Let's have a look how I do those cross agreements. First I add >> an agreement on A for C. This is fine. Then, I do the same on C >> (for >> A) and I get the messages NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): Received error 89: NULL for total update operation On C and on A I get: >> [10/Aug/2010:17:12:37 -0400] - somehow, there are still 16 >> entries in the entry cache. :/ >> [10/Aug/2010:17:12:38 -0400] - WARNING: Import is running with >> nsslapd-db-private-import-mem on; No other process is allowed to >> access the database >> [10/Aug/2010:17:12:38 -0400] - BAD CACHE ASSERTION at >> ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 >> >> >> Hope, this helps. >> >> >> >> >> >> >> >> > How do you do the replica init? > > > > > > > >> Thanks, >> -Reinhard >> -----Original Message----- >> From: 389-users-bounces@lists.fedoraproject.org >> [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of >> Reinhard Nappert >> Sent: Tuesday, August 10, 2010 2:42 PM >> To: General discussion list for the 389 Directory server project. >> Subject: Re: [389-users] Multi-Master setup >> >> Rich, on the consumer, I see the following messages: >> >> NSMMReplicationPlugin - agmt="cn=nix2mustrum" (mustrum:389): >> Received error 89: NULL for total update operation >> >> -Reinhard >> >> -----Original Message----- >> From: 389-users-bounces@lists.fedoraproject.org >> [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of >> Rich Megginson >> Sent: Tuesday, August 10, 2010 12:41 PM >> To: General discussion list for the 389 Directory server project. >> Subject: Re: [389-users] Multi-Master setup >> >> Reinhard Nappert wrote: >> >> >> >> >> >> >> >> >>> Hi, >>> >>> I have seen the following message in the errors log file, when >>> I set MMR agreements up: >>> >>> [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - >>> repl_set_mtn_referrals: could not set referrals for replica o=base: >>> 1 >>> [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - >>> multimaster_be_state_change: replica o=base is going offline; >>> disabling replication >>> [10/Aug/2010:11:46:46 -0400] - somehow, there are still 20 >>> entries in the entrycache. :/ >>> [10/Aug/2010:11:46:46 -0400] - WARNING: Import is running with >>> nsslapd-db-private-import-mem on; No other process is allowed >>> to access the database >>> [10/Aug/2010:11:46:48 -0400] - BAD CACHE ASSERTION at >>> ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 >>> [10/Aug/2010:11:46:52 -0400] - Fedora-Directory/1.1.2 >>> B2009.090.1643 starting up >>> [10/Aug/2010:11:46:52 -0400] - Detected Disorderly Shutdown >>> last time DirectoryServer was running, recovering database. >>> >>> After I re-initialize the database from the supplier (setting >>> attribute nsds5BeginReplicaRefresh to start of the agreement >>> object), the database gets correctly imported. >>> >>> Any idea, what is going on? >>> >>> >>> >>> >>> >>> >>> >>> >> No, not sure. But if you can develop a reproducible test case, that would be helpful. >> >> >> >> >> >> >> >> >>> Thanks, >>> -Reinhard >>> >>> --------------------------------------------------------------- >>> - >>> - >>> -- >>> - >>> - >>> - >>> -- >>> >>> -- >>> 389 users mailing list >>> 389-users@lists.fedoraproject.org >>> https://admin.fedoraproject.org/mailman/listinfo/389-users >>> >>> >>> >>> >>> >>> >>> >>> >> -- >> 389 users mailing list >> 389-users@lists.fedoraproject.org >> https://admin.fedoraproject.org/mailman/listinfo/389-users >> -- >> 389 users mailing list >> 389-users@lists.fedoraproject.org >> https://admin.fedoraproject.org/mailman/listinfo/389-users >> -- >> 389 users mailing list >> 389-users@lists.fedoraproject.org >> https://admin.fedoraproject.org/mailman/listinfo/389-users >> >> >> >> >> >> >> >> > -- > 389 users mailing list > 389-users@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/389-users > -- > 389 users mailing list > 389-users@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/389-users > > > > > >

>

389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

Reinhard Nappert wrote:

Rick,

I have seem this again: [23/Sep/2010:11:23:00 -0300] NSMMReplicationPlugin - multimaster_be_state_change: replica o=umc is going offline; disabling replication [23/Sep/2010:11:23:00 -0300] - somehow, there are still 13 entries in the entry cache. :/ [23/Sep/2010:11:23:00 -0300] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [23/Sep/2010:11:23:01 -0300] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0

But this time, I also got an syslog message: Sep 23 11:23:02 ... kernel: ns-slapd[19506]: segfault at 0 rip 2ad0cc3bc05d rsp 5b0350c0 error 6

This my give you a clue.

If so, please let me know. This happens with Fedora-Directory/1.1.2 B2009.090.1643

I don't know. You could try enabling core files: add ulimit -c unlimited to your start-slapd or initscript sysctl -w fs.suid_dumpable=1

Then reproduce the segfault, which should produce a core file in the directory server log directory Then, install the fedora-ds debuginfo package Then, gdb /path/to/ns-slapd /path/to/core.NNNN inside gdb, do thread apply all bt

Thanks, -Reinhard

-----Original Message----- From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of Rich Megginson Sent: Tuesday, August 10, 2010 12:41 PM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] Multi-Master setup

Reinhard Nappert wrote:

...

Hi,

I have seen the following message in the errors log file, when I set MMR agreements up:

[10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=base: 1 [10/Aug/2010:11:46:44 -0400] NSMMReplicationPlugin - multimaster_be_state_change: replica o=base is going offline; disabling replication [10/Aug/2010:11:46:46 -0400] - somehow, there are still 20 entries in the entrycache. :/ [10/Aug/2010:11:46:46 -0400] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [10/Aug/2010:11:46:48 -0400] - BAD CACHE ASSERTION at ../ldap/servers/slapd/back-ldbm/cache.c/765: e->ep_refcnt > 0 [10/Aug/2010:11:46:52 -0400] - Fedora-Directory/1.1.2 B2009.090.1643 starting up [10/Aug/2010:11:46:52 -0400] - Detected Disorderly Shutdown last time DirectoryServer was running, recovering database.

After I re-initialize the database from the supplier (setting attribute nsds5BeginReplicaRefresh to start of the agreement object), the database gets correctly imported.

Any idea, what is going on?

No, not sure. But if you can develop a reproducible test case, that would be helpful.

...

Thanks, -Reinhard

---

--

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users