11:32 p.m.
Hi,
After our LDAP instance has been running for a while large LDAP searches in
our userRoot start hanging. This appears to be caused by the entryrdn index
becoming corrupt since it can be fixed by regenerating the entryrdn index.
To give an example of a command that fails:
ldapsearch -x -b "dc=ourorg,dc=com" -h localhost -D "cn=directory
manager" -w
removed objectclass=posixAccount > /dev/shm/ldap_search_output
This hangs after reading a large number of entries. The number of entries
change each time, but I believe it may be actually failing on the same specific
entry each time, and just reading them in a different order.
This can be fixed by stopping ns-slapdand
moving /var/lib/dirsrv/slapd-ldap-04/db/userRoot/entryrdn.db4 out of the way
before running `./db2index -n userRoot -t entryrdn`, but it will then reoccur
after a hour or two or so.
Our system:
$ cat /etc/redhat-release
CentOS release 6.5 (Final)
$ uname -r
2.6.32-431.3.1.el6.x86_64
$ rpm -q 389-ds
389-ds-1.2.2-1.el6.noarch
In case they're helpful I have stack traces from during both failed and
successful searches. I can send them through if they are useful.
Does anyone have any idea what might be causing this, or how we could go about
fixing it? Should we report it as a bug?
Thank you,
--
TimP
[http://blog.timp.com.au]
10:24 a.m.
On 02/11/2014 10:32 PM, Timothy Pollard wrote:
Hi,
After our LDAP instance has been running for a while large LDAP searches in
our userRoot start hanging. This appears to be caused by the entryrdn index
becoming corrupt since it can be fixed by regenerating the entryrdn index.
To give an example of a command that fails:
ldapsearch -x -b "dc=ourorg,dc=com" -h localhost -D "cn=directory
manager" -w
removed objectclass=posixAccount > /dev/shm/ldap_search_output
This hangs after reading a large number of entries. The number of entries
change each time, but I believe it may be actually failing on the same specific
entry each time, and just reading them in a different order.
This can be fixed by stopping ns-slapdand
moving /var/lib/dirsrv/slapd-ldap-04/db/userRoot/entryrdn.db4 out of the way
before running `./db2index -n userRoot -t entryrdn`, but it will then reoccur
after a hour or two or so.
Our system:
$ cat /etc/redhat-release
CentOS release 6.5 (Final)
$ uname -r
2.6.32-431.3.1.el6.x86_64
$ rpm -q 389-ds
389-ds-1.2.2-1.el6.noarch
rpm -q 389-ds-base
Are these servers running in VMs?
In case they're helpful I have stack traces from during both failed and
successful searches. I can send them through if they are useful.
Yes.
Does anyone have any idea what might be causing this, or how we could go about
fixing it? Should we report it as a bug?
Yes. https://fedorahosted.org/389/newticket
You can attach scripts, logs, stack traces, etc. to the ticket.
Thank you,
--
389 users mailing list
389-users(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
11:22 a.m.
Rich Megginson wrote:
On 02/11/2014 10:32 PM, Timothy Pollard wrote:
> Hi,
>
> After our LDAP instance has been running for a while large LDAP searches in
> our userRoot start hanging. This appears to be caused by the entryrdn index
> becoming corrupt since it can be fixed by regenerating the entryrdn index.
>
> To give an example of a command that fails:
>
> ldapsearch -x -b "dc=ourorg,dc=com" -h localhost -D "cn=directory
manager" -w
> removed objectclass=posixAccount > /dev/shm/ldap_search_output
>
> This hangs after reading a large number of entries. The number of entries
> change each time, but I believe it may be actually failing on the same specific
> entry each time, and just reading them in a different order.
>
> This can be fixed by stopping ns-slapdand
> moving /var/lib/dirsrv/slapd-ldap-04/db/userRoot/entryrdn.db4 out of the way
> before running `./db2index -n userRoot -t entryrdn`, but it will then reoccur
> after a hour or two or so.
>
> Our system:
> $ cat /etc/redhat-release
> CentOS release 6.5 (Final)
> $ uname -r
> 2.6.32-431.3.1.el6.x86_64
> $ rpm -q 389-ds
> 389-ds-1.2.2-1.el6.noarch
rpm -q 389-ds-base
Are these servers running in VMs?
> In case they're helpful I have stack traces from during both failed and
> successful searches. I can send them through if they are useful.
Yes.
> Does anyone have any idea what might be causing this, or how we could go about
> fixing it? Should we report it as a bug?
Yes. https://fedorahosted.org/389/newticket
You can attach scripts, logs, stack traces, etc. to the ticket.
Can we also have
the output from "dbscan -f
/var/lib/dirsrv/slapd-ldap-04/db/userRoot/entryrdn.db4" when the
problem occurs?
And, reindexing entryrdn is necessary for the temporary recovery? For
instance, just restarting the server does not help? I'm wondering
whether the dncache in memory is corrupted or the entryrdn index itself
is ...
Thanks,
--noriko
> Thank you,
>
>
> --
> 389 users mailing list
> 389-users(a)lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
4:06 p.m.
On Wed, 12 Feb 2014 09:22:19 -0800
Noriko Hosoi <nhosoi(a)redhat.com> wrote:
Rich Megginson wrote:
> On 02/11/2014 10:32 PM, Timothy Pollard wrote:
>>
>> Our system:
>> $ cat /etc/redhat-release
>> CentOS release 6.5 (Final)
>> $ uname -r
>> 2.6.32-431.3.1.el6.x86_64
>> $ rpm -q 389-ds
>> 389-ds-1.2.2-1.el6.noarch
> rpm -q 389-ds-base
>
> Are these servers running in VMs?
No they're physical servers.
>
>> In case they're helpful I have stack traces from during both failed and
>> successful searches. I can send them through if they are useful.
>
> Yes.
>> Does anyone have any idea what might be causing this, or how we could go
>> about fixing it? Should we report it as a bug?
>
> Yes. https://fedorahosted.org/389/newticket
>
> You can attach scripts, logs, stack traces, etc. to the ticket.
OK, I've created a new ticket and attached the stacktraces. Thanks.
https://fedorahosted.org/389/ticket/47696
Can we also have the output from "dbscan
-f /var/lib/dirsrv/slapd-ldap-04/db/userRoot/entryrdn.db4" when the problem
occurs?
Oddly it didn't get broken over night, but I did copy the entryrdn.db4 file
last time it broken, so I can take dbscans of the backup of the broken one and
of the currently working one. Unfortunately both files are over 140MB, and the
best compression I can manage gets them down to 16MB each, which is too big to
attach to the ticket.
Any suggestions on how to best share such large files?
And, reindexing entryrdn is necessary for the temporary recovery? For
instance, just restarting the server does not help? I'm wondering whether
the dncache in memory is corrupted or the entryrdn index itself is ...
I'm pretty sure it is; I haven't specifically tested that, but I have restarted
it while it was broken to try to resolve other issues as well.
Thanks,
--noriko
I also forgot to mention a potentially related error message. Occasionally on
entry deletion we see errors like this in our error log:
[12/Feb/2014:20:20:46 +0000] entryrdn-index - _entryrdn_delete_key: Failed to
remove ou=test; has children
[12/Feb/2014:20:20:46 +0000] - database index operation failed BAD 1031, err=-1
Unknown error: -1
Since they mention the entryrdn and indexes I thought they might be related,
but they don't seem to directly cause this problem, since I'm seeing these
errors now, but the system is fine, but I've had it get broken before with none
of these errors between the last re-index and the failure.
--
TimP
[http://blog.timp.com.au]
4:20 p.m.
Hi Timothy,
Timothy Pollard wrote:
Oddly it didn't get broken over night, but I did copy the
entryrdn.db4 file
last time it broken, so I can take dbscans of the backup of the broken one and
of the currently working one. Unfortunately both files are over 140MB, and the
best compression I can manage gets them down to 16MB each, which is too big to
attach to the ticket.
1) I increased the max attachment size. Could you try to
attach the
compressed file to the ticket 47696 one more time?
2) What 389-ds-base version you are running (note: not 389-ds)
$ rpm -q 389-ds-base
3) Could there be any other errors in the error log? Is it possible to
attach it to the ticket?
Thanks!
--noriko
4:59 p.m.
On Wed, 12 Feb 2014 14:20:11 -0800
Noriko Hosoi <nhosoi(a)redhat.com> wrote:
Hi Timothy,
Timothy Pollard wrote:
> Oddly it didn't get broken over night, but I did copy the entryrdn.db4 file
> last time it broken, so I can take dbscans of the backup of the broken one
> and of the currently working one. Unfortunately both files are over 140MB,
> and the best compression I can manage gets them down to 16MB each, which is
> too big to attach to the ticket.
1) I increased the max attachment size. Could you try to attach the
compressed file to the ticket 47696 one more time?
OK, thanks. I hope XZ compression isn't a problem, the Bzip2 version is just
over 18MB, so I can't send that.
2) What 389-ds-base version you are running (note: not 389-ds)
$ rpm -q 389-ds-base
$ rpm -q 389-ds-base
389-ds-base-1.2.11.15-31.el6_5.x86_64
And in case any of this helps:
$ rpm -qa | grep 389
389-ds-base-1.2.11.15-31.el6_5.x86_64
389-ds-console-doc-1.2.6-1.el6.noarch
389-admin-1.1.35-1.el6.x86_64
389-ds-console-1.2.6-1.el6.noarch
389-admin-console-doc-1.1.8-1.el6.noarch
389-ds-1.2.2-1.el6.noarch
389-adminutil-1.1.19-1.el6.x86_64
389-console-1.1.7-1.el6.noarch
389-admin-console-1.1.8-1.el6.noarch
389-dsgw-1.1.11-1.el6.x86_64
389-ds-base-debuginfo-1.2.11.15-31.el6_5.x86_64
389-ds-base-libs-1.2.11.15-31.el6_5.x86_64
3) Could there be any other errors in the error log? Is it possible to attach
it to the ticket?
There's not much, but I'll add part of the error log to the ticket as well.
Thanks!
--noriko
--
389 users mailing list
389-users(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
Thanks,
--
TimP
[http://blog.timp.com.au]
3732
days inactive
3732
days old
389-users@lists.fedoraproject.org
5 comments
3 participants
participants (3)
-
Noriko Hosoi -
Rich Megginson -
Timothy Pollard