9:31 a.m.
Afternoon all
I’ve got a slightly strange one with one of our FreeIPA clusters, whereby the topology
suffixes appear to have disappeared.
From what I can see, this is causing replication issues between the hosts, which is
causing us issues with bootstrapping new clients against FreeIPA.
I’m not aware of any config changes that have happened on the FreeIPA hosts that could
have caused this issue, so am a bit stumped atm.
Is someone able to advise next steps on how to investigate the cause and correct the
configuration?
Regards
Gavin
12:24 p.m.
On Wed, Apr 4, 2018 at 4:31 PM, Gavin Williams via FreeIPA-users
<freeipa-users(a)lists.fedorahosted.org> wrote:
Afternoon all
I’ve got a slightly strange one with one of our FreeIPA clusters, whereby the topology
suffixes appear to have disappeared.
How is this manifested? No visible in Web UI, CLI?
From what I can see, this is causing replication issues between the
hosts, which is causing us issues with bootstrapping new clients against FreeIPA.
I’m not aware of any config changes that have happened on the FreeIPA hosts that could
have caused this issue, so am a bit stumped atm.
Is someone able to advise next steps on how to investigate the cause and correct the
configuration?
For anything regarding replication, a good start is to check directory
server error and access logs on both sides.
https://www.freeipa.org/page/Files_to_be_attached_to_bug_report#Directory...
https://www.freeipa.org/page/Troubleshooting#Directory_Server_issues
Next step could be to check for replication conflicts:
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10...
Regards
Gavin
--
Petr Vobornik
4:28 p.m.
Petr
Yeh, I was unable to see the suffixes and replication agreements via the WebUI.
However searching using ldapsearch, they were still present. So I tracked the issue down
to my named user account not having enough permissions. Logged in as ‘admin’ user and was
able to see all the details.
So that just leaves the issue with the fact that replication broke in the first place.
Looking back through the slapd error log, I came across this:
[28/Mar/2018:17:27:04.558588967 +0100] - ERR - NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn - retry (49) the transaction (c
sn=5abbc252002800040000) failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK: Locker killed to
resolve a deadlock))
[28/Mar/2018:17:27:04.575793325 +0100] - ERR - NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn - Failed to write entry with cs
n (5abbc252002800040000); db error - -30993 BDB0068 DB_LOCK_DEADLOCK: Locker killed to
resolve a deadlock
[28/Mar/2018:17:27:04.578285790 +0100] - ERR - NSMMReplicationPlugin -
write_changelog_and_ruv - Can't add a change for ipaUniqueID=6dc4846c-27a
3-11e8-a0a5-fa163e82604c,cn=sudorules,cn=sudo,dc=weareact,dc=net (uniqid:
64da9801-27a311e8-8bfb8904-640ff48c, optype: 8) to changelog csn 5abbc
252002800040000
[28/Mar/2018:17:27:04.595240157 +0100] - ERR - NSMMReplicationPlugin - process_postop -
Failed to apply update (5abbc252002800040000) error (1).
Aborting replication session(conn=453585 op=18)
[28/Mar/2018:17:27:14.160079067 +0100] - ERR - NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn - retry (49) the transaction (c
sn=5abbc252002800040000) failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK: Locker killed to
resolve a deadlock))
[28/Mar/2018:17:27:14.161481168 +0100] - ERR - NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn - Failed to write entry with csn (5abbc252002800040000); db error -
-30993 BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
[28/Mar/2018:17:27:14.162533841 +0100] - ERR - NSMMReplicationPlugin -
write_changelog_and_ruv - Can't add a change for
ipaUniqueID=6dc4846c-27a3-11e8-a0a5-fa163e82604c,cn=sudorules,cn=sudo,dc=weareact,dc=net
(uniqid: 64da9801-27a311e8-8bfb8904-640ff48c, optype: 8) to changelog csn
5abbc252002800040000
[28/Mar/2018:17:27:14.177194703 +0100] - ERR - NSMMReplicationPlugin - process_postop -
Failed to apply update (5abbc252002800040000) error (1). Aborting replication
session(conn=453594 op=6)
Any pointers on identifying possible cause?
Cheers
Gav
On 5 Apr 2018, at 18:24, Petr Vobornik
<pvoborni@redhat.com<mailto:pvoborni@redhat.com>> wrote:
On Wed, Apr 4, 2018 at 4:31 PM, Gavin Williams via FreeIPA-users
<freeipa-users@lists.fedorahosted.org<mailto:freeipa-users@lists.fedorahosted.org>>
wrote:
Afternoon all
I’ve got a slightly strange one with one of our FreeIPA clusters, whereby the topology
suffixes appear to have disappeared.
How is this manifested? No visible in Web UI, CLI?
From what I can see, this is causing replication issues between the hosts, which is
causing us issues with bootstrapping new clients against FreeIPA.
I’m not aware of any config changes that have happened on the FreeIPA hosts that could
have caused this issue, so am a bit stumped atm.
Is someone able to advise next steps on how to investigate the cause and correct the
configuration?
For anything regarding replication, a good start is to check directory
server error and access logs on both sides.
https://www.freeipa.org/page/Files_to_be_attached_to_bug_report#Directory...
https://www.freeipa.org/page/Troubleshooting#Directory_Server_issues
Next step could be to check for replication conflicts:
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10...
Regards
Gavin
--
Petr Vobornik
2:45 a.m.
On 04/05/2018 11:28 PM, Gavin Williams via FreeIPA-users wrote:
Petr
Yeh, I was unable to see the suffixes and replication agreements via
the WebUI.
However searching using ldapsearch, they were still present. So I
tracked the issue down to my named user account not having enough
permissions. Logged in as ‘admin’ user and was able to see all the
details.
So that just leaves the issue with the fact that replication broke in
the first place. Looking back through the slapd error log, I came
across this:
The errors below do not indicate that replication is broken, a
replication session failed and is retried, you can see that the errors
are 10 sec apart and refer to different replication conenctions, so
replication was probably working in between and has probably resumed again.
The underlying problem is that there is concurrent access to the
changelog by incoming connection writing to the changelog and by
outgoing replication connections reading it. The access is protected by
locks at the db (BDB) level and in some situations there can be
deadlocks. The db layer has a mechanism to abort one of the threads and
let the other continue. The aborted thread will log an error and has to
be retried, either immediately or after returning an error to the client
-that is what you are seeing.
Which thread is aborted is determined by the configured deadlock policy,
the default tends to abort the writing one. If these errors occur
frequently and make issues it is worth to change this policy. In the entry:
dn: cn=config,cn=ldbm database,cn=plugins,cn=config
change the attribute: nsslapd-db-deadlock-policy
to
nsslapd-db-deadlock-policy: 6
this will abort the thread with the minimal write locks and will abort
the outgoing repl connection and should have less impact
[28/Mar/2018:17:27:04.558588967 +0100] - ERR - NSMMReplicationPlugin
- changelog program - _cl5WriteOperationTxn - retry (49) the transaction (c
sn=5abbc252002800040000) failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK: Locker killed to
resolve a deadlock))
[28/Mar/2018:17:27:04.575793325 +0100] - ERR - NSMMReplicationPlugin - changelog program
- _cl5WriteOperationTxn - Failed to write entry with cs
n (5abbc252002800040000); db error - -30993 BDB0068 DB_LOCK_DEADLOCK: Locker killed to
resolve a deadlock
[28/Mar/2018:17:27:04.578285790 +0100] - ERR - NSMMReplicationPlugin -
write_changelog_and_ruv - Can't add a change for ipaUniqueID=6dc4846c-27a
3-11e8-a0a5-fa163e82604c,cn=sudorules,cn=sudo,dc=weareact,dc=net (uniqid:
64da9801-27a311e8-8bfb8904-640ff48c, optype: 8) to changelog csn 5abbc
252002800040000
[28/Mar/2018:17:27:04.595240157 +0100] - ERR - NSMMReplicationPlugin - process_postop -
Failed to apply update (5abbc252002800040000) error (1).
Aborting replication session(conn=453585 op=18)
[28/Mar/2018:17:27:14.160079067 +0100] - ERR - NSMMReplicationPlugin - changelog program
- _cl5WriteOperationTxn - retry (49) the transaction (c
sn=5abbc252002800040000) failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK: Locker killed to
resolve a deadlock))
[28/Mar/2018:17:27:14.161481168 +0100] - ERR - NSMMReplicationPlugin - changelog program
- _cl5WriteOperationTxn - Failed to write entry with csn (5abbc252002800040000); db error
- -30993 BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
[28/Mar/2018:17:27:14.162533841 +0100] - ERR - NSMMReplicationPlugin -
write_changelog_and_ruv - Can't add a change for
ipaUniqueID=6dc4846c-27a3-11e8-a0a5-fa163e82604c,cn=sudorules,cn=sudo,dc=weareact,dc=net
(uniqid: 64da9801-27a311e8-8bfb8904-640ff48c, optype: 8) to changelog csn
5abbc252002800040000
[28/Mar/2018:17:27:14.177194703 +0100] - ERR - NSMMReplicationPlugin - process_postop -
Failed to apply update (5abbc252002800040000) error (1). Aborting replication
session(conn=453594 op=6)
Any pointers on identifying possible cause?
Cheers
Gav
> On 5 Apr 2018, at 18:24, Petr Vobornik <pvoborni(a)redhat.com
> <mailto:pvoborni@redhat.com>> wrote:
>
> On Wed, Apr 4, 2018 at 4:31 PM, Gavin Williams via FreeIPA-users
> <freeipa-users(a)lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>> wrote:
>> Afternoon all
>>
>> I’ve got a slightly strange one with one of our FreeIPA clusters,
>> whereby the topology suffixes appear to have disappeared.
>
> How is this manifested? No visible in Web UI, CLI?
>
>> From what I can see, this is causing replication issues between the
>> hosts, which is causing us issues with bootstrapping new clients
>> against FreeIPA.
>>
>> I’m not aware of any config changes that have happened on the
>> FreeIPA hosts that could have caused this issue, so am a bit stumped
>> atm.
>>
>> Is someone able to advise next steps on how to investigate the cause
>> and correct the configuration?
>
> For anything regarding replication, a good start is to check directory
> server error and access logs on both sides.
>
>
https://www.freeipa.org/page/Files_to_be_attached_to_bug_report#Directory...
> https://www.freeipa.org/page/Troubleshooting#Directory_Server_issues
>
> Next step could be to check for replication conflicts:
>
>
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10...
>
>>
>> Regards
>> Gavin
> --
> Petr Vobornik
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric
Shander
11:18 a.m.
Ludwig et al,
Apologies on resurrecting this thread!
Which thread is aborted is determined by the configured deadlock
policy, the default tends to abort the writing one. If these errors occur frequently and
make issues it is worth to change this policy. In the entry:
dn: cn=config,cn=ldbm database,cn=plugins,cn=config
change the attribute: nsslapd-db-deadlock-policy
to
nsslapd-db-deadlock-policy: 6
this will abort the thread with the minimal write locks and will abort the outgoing repl
connection and should have less impact
I've seen a few similar messages in our error logs, but after a period
of time the replication resumed without conflict. I have not found
any remaining issues in the directory server logs after following the
history of this thread and the thread ([Freeipa-users] replica DS
failure deadlock; 10/18/2016).
Would adjusting the nsslapd-threadnumber on both IPA servers
potentially contribute to these messages? During my initial
deployment, I adjusted the value of nsslapd-threadnumber per the
history of the thread ([Freeipa-users] performance scaling of sssd /
freeipa; 01/26/2017). Of course, this value may not be appropriate
for our deployment and I can certainly scale it back.
Thank you for any information!
John DeSantis
Il giorno ven 6 apr 2018 alle ore 03:37 Ludwig Krispenz via
FreeIPA-users <freeipa-users(a)lists.fedorahosted.org> ha scritto:
>
>
> On 04/05/2018 11:28 PM, Gavin Williams via FreeIPA-users wrote:
>
> Petr
>
> Yeh, I was unable to see the suffixes and replication agreements via the WebUI.
>
> However searching using ldapsearch, they were still present. So I tracked the issue
down to my named user account not having enough permissions. Logged in as ‘admin’ user and
was able to see all the details.
>
> So that just leaves the issue with the fact that replication broke in the first
place. Looking back through the slapd error log, I came across this:
>
>
> The errors below do not indicate that replication is broken, a replication session
failed and is retried, you can see that the errors are 10 sec apart and refer to different
replication conenctions, so replication was probably working in between and has probably
resumed again.
>
> The underlying problem is that there is concurrent access to the changelog by
incoming connection writing to the changelog and by outgoing replication connections
reading it. The access is protected by locks at the db (BDB) level and in some situations
there can be deadlocks. The db layer has a mechanism to abort one of the threads and let
the other continue. The aborted thread will log an error and has to be retried, either
immediately or after returning an error to the client -that is what you are seeing.
>
Which thread is aborted is determined by the configured deadlock
policy, the default tends to abort the writing one. If these errors occur frequently and
make issues it is worth to change this policy. In the entry:
dn: cn=config,cn=ldbm database,cn=plugins,cn=config
change the attribute: nsslapd-db-deadlock-policy
to
nsslapd-db-deadlock-policy: 6
this will abort the thread with the minimal write locks and will abort the outgoing repl
connection and should have less impact
>
> [28/Mar/2018:17:27:04.558588967 +0100] - ERR - NSMMReplicationPlugin - changelog
program - _cl5WriteOperationTxn - retry (49) the transaction (c
>
> sn=5abbc252002800040000) failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK: Locker killed
to resolve a deadlock))
>
> [28/Mar/2018:17:27:04.575793325 +0100] - ERR - NSMMReplicationPlugin - changelog
program - _cl5WriteOperationTxn - Failed to write entry with cs
>
> n (5abbc252002800040000); db error - -30993 BDB0068 DB_LOCK_DEADLOCK: Locker killed
to resolve a deadlock
>
> [28/Mar/2018:17:27:04.578285790 +0100] - ERR - NSMMReplicationPlugin -
write_changelog_and_ruv - Can't add a change for ipaUniqueID=6dc4846c-27a
>
> 3-11e8-a0a5-fa163e82604c,cn=sudorules,cn=sudo,dc=weareact,dc=net (uniqid:
64da9801-27a311e8-8bfb8904-640ff48c, optype: 8) to changelog csn 5abbc
>
> 252002800040000
>
> [28/Mar/2018:17:27:04.595240157 +0100] - ERR - NSMMReplicationPlugin - process_postop
- Failed to apply update (5abbc252002800040000) error (1).
>
> Aborting replication session(conn=453585 op=18)
>
> [28/Mar/2018:17:27:14.160079067 +0100] - ERR - NSMMReplicationPlugin - changelog
program - _cl5WriteOperationTxn - retry (49) the transaction (c
>
> sn=5abbc252002800040000) failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK: Locker killed
to resolve a deadlock))
>
> [28/Mar/2018:17:27:14.161481168 +0100] - ERR - NSMMReplicationPlugin - changelog
program - _cl5WriteOperationTxn - Failed to write entry with csn (5abbc252002800040000);
db error - -30993 BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
>
> [28/Mar/2018:17:27:14.162533841 +0100] - ERR - NSMMReplicationPlugin -
write_changelog_and_ruv - Can't add a change for
ipaUniqueID=6dc4846c-27a3-11e8-a0a5-fa163e82604c,cn=sudorules,cn=sudo,dc=weareact,dc=net
(uniqid: 64da9801-27a311e8-8bfb8904-640ff48c, optype: 8) to changelog csn
5abbc252002800040000
>
> [28/Mar/2018:17:27:14.177194703 +0100] - ERR - NSMMReplicationPlugin - process_postop
- Failed to apply update (5abbc252002800040000) error (1). Aborting replication
session(conn=453594 op=6)
>
> Any pointers on identifying possible cause?
>
> Cheers
> Gav
>
> On 5 Apr 2018, at 18:24, Petr Vobornik <pvoborni(a)redhat.com> wrote:
>
> On Wed, Apr 4, 2018 at 4:31 PM, Gavin Williams via FreeIPA-users
> <freeipa-users(a)lists.fedorahosted.org> wrote:
>
> Afternoon all
>
> I’ve got a slightly strange one with one of our FreeIPA clusters, whereby the
topology suffixes appear to have disappeared.
>
>
> How is this manifested? No visible in Web UI, CLI?
>
> From what I can see, this is causing replication issues between the hosts, which is
causing us issues with bootstrapping new clients against FreeIPA.
>
> I’m not aware of any config changes that have happened on the FreeIPA hosts that
could have caused this issue, so am a bit stumped atm.
>
> Is someone able to advise next steps on how to investigate the cause and correct the
configuration?
>
>
> For anything regarding replication, a good start is to check directory
> server error and access logs on both sides.
>
>
https://www.freeipa.org/page/Files_to_be_attached_to_bug_report#Directory...
> https://www.freeipa.org/page/Troubleshooting#Directory_Server_issues
>
> Next step could be to check for replication conflicts:
>
>
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10...
>
>
> Regards
> Gavin
>
> --
> Petr Vobornik
>
>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>
>
> --
> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric
Shander
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
1711
days inactive
2213
days old
freeipa-users@lists.fedorahosted.org
4 comments
4 participants
participants (4)
-
Gavin Williams -
John Desantis -
Ludwig Krispenz -
Petr Vobornik