4:20 p.m.
Hi Team,
We used following to get the number of rwlocks for /usr/sbin/ns-slapd
process in Centos 7.9 to catch deadlocks:
PID=`pidof ns-slapd`
gdb -ex 'set confirm off' -ex 'set pagination off' -ex 'thread apply
all bt
full' -ex 'quit' /usr/sbin/ns-slapd $PID |& grep '^#0.*lock' |
grep
pthread_rwlock | sort -u
That helped us to detect ns-slapd hang caused by deadlocks.
After migrating to Red Hat 8.6, we had a lot of hangs (dirsvr is running
but not responding) and could not find why. We use the same above method,
however, we are not able to catch anything. I wonder if there is a
different way to count the rwlocks in Red Hat 8.6?
We realize that there are multiple reasons to cause hangs, however, we
would like to rule out the possibility of the deadlock.
The OS and packages:
Red Hat Enterprise Linux release 8.6 (Ootpa)
ipa-server.x86_64 4.9.8-7.module+el8.6.0+14337+19b76db2
@rhel-8-for-x86_64-appstream-rpms
slapi-nis-0.56.6-4.module+el8.6.0+12936+736896b2.x86_64
389-ds-base-libs-1.4.3.28-6.module+el8.6.0+14129+983ceada.x86_64
389-ds-base-1.4.3.28-6.module+el8.6.0+14129+983ceada.x86_64
Many thanks.
Kathy.
5:10 a.m.
New subject: How to check the number of read/write locks on /usr/sbin/ns-slapd process?
Hi Kathy,
The procedure to diagnose hang looks nice. My understanding is that it
assumes that in deadlock situation the more we have threads waiting on a
resource, the more probable we have a hang/deadlock. Now because of the
dynamic of the server itself, on the configuration, on the type of
requests, on the monitoring of rwlock vs simple locks, on the impact on
gdb prone to stop threads on locks, this scripts may fails to detect
hang/deadlock.
ATM I see no specific 8.6 enhancements that can explain why the script
is failing now.
If the server is no longer responsive to a client request (keepalive
req), I suggest you collect pstacks, 'top -H -p `pidof ns-slapd`' and
db_stat -N -CA. If it is eating CPU you may looks at the activity of
thread consuming CPU. If it is not, it is possibly deadlock between
backend/database access.
Best regards
thierry
On 8/30/22 11:20 PM, Kathy Zhu via FreeIPA-users wrote:
Hi Team,
We used following to get the number of rwlocks for /usr/sbin/ns-slapd
process in Centos 7.9 to catch deadlocks:
PID=`pidof ns-slapd`
gdb -ex 'set confirm off' -ex 'set pagination off' -ex 'thread apply
all bt full' -ex 'quit' /usr/sbin/ns-slapd $PID |& grep
'^#0.*lock' |
grep pthread_rwlock | sort -u
That helped us to detect ns-slapd hang caused by deadlocks.
After migrating to Red Hat 8.6, we had a lot of hangs (dirsvr is
running but not responding) and could not find why. We use the same
above method, however, we are not able to catch anything. I wonder if
there is a different way to count the rwlocks in Red Hat 8.6?
We realize that there are multiple reasons to cause hangs, however, we
would like to rule out the possibility of the deadlock.
The OS and packages:
Red Hat Enterprise Linux release 8.6 (Ootpa)
ipa-server.x86_64 4.9.8-7.module+el8.6.0+14337+19b76db2
@rhel-8-for-x86_64-appstream-rpms
slapi-nis-0.56.6-4.module+el8.6.0+12936+736896b2.x86_64
389-ds-base-libs-1.4.3.28-6.module+el8.6.0+14129+983ceada.x86_64
389-ds-base-1.4.3.28-6.module+el8.6.0+14129+983ceada.x86_64
Many thanks.
Kathy.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
6:16 a.m.
New subject: How to check the number of read/write locks on /usr/sbin/ns-slapd process?
On 30-08-2022 23:20, Kathy Zhu via FreeIPA-users wrote:
Hi Team,
We used following to get the number of rwlocks for /usr/sbin/ns-slapd process in Centos
7.9 to catch deadlocks:
PID=`pidof ns-slapd`
gdb -ex 'set confirm off' -ex 'set pagination off' -ex 'thread apply
all bt full' -ex 'quit' /usr/sbin/ns-slapd $PID |& grep
'^#0.*lock' | grep pthread_rwlock | sort -u
That helped us to detect ns-slapd hang caused by deadlocks.
After migrating to Red Hat 8.6, we had a lot of hangs (dirsvr is running but not
responding) and could not find why. We use the same above method, however, we are not able
to catch anything. I wonder if there is a different way to count the rwlocks in Red Hat
8.6?
We realize that there are multiple reasons to cause hangs, however, we would like to rule
out the possibility of the deadlock.
The OS and packages:
Red Hat Enterprise Linux release 8.6 (Ootpa)
ipa-server.x86_64 4.9.8-7.module+el8.6.0+14337+19b76db2
@rhel-8-for-x86_64-appstream-rpms
slapi-nis-0.56.6-4.module+el8.6.0+12936+736896b2.x86_64
389-ds-base-libs-1.4.3.28-6.module+el8.6.0+14129+983ceada.x86_64
389-ds-base-1.4.3.28-6.module+el8.6.0+14129+983ceada.x86_64
Hey Kathy,
May I suggest to also look at the 389-users mailing list
389-users(a)lists.fedoraproject.org
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproje...
For example threads with subject:
* Lock table is out of available lock entries
* Enabling retro changelog maxage with 3 million entries make dirsrv not respond
anymore
* DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
I have reported about hangs (deadlocks). In our case when Retro Changelog trimming
is enabled we get a hangs within a few minutes. As far as I know there is no solution
yet.
That is why we have disabled this trimming. (And our Retro Changelog has accumulated to
more
than Gb of data.)
--
Kees
12:08 p.m.
New subject: How to check the number of read/write locks on /usr/sbin/ns-slapd process?
Hi Thierry and Kees,
Thank you for your replies. I provided the outputs of "pstack $(pidof
ns-slapd)" to Red Hat Support Center, and they were able to find the root
cause for a particular hang issue, it turns out to be a bug. I attached the
details. I will continue to troubleshoot with them. Hangs happen a lot in
our environment. I will reference your comments to collect more data to
confirm that it is the same issue of all hang's or if there is something
else.
Many thanks!
Kathy.
Red Hat SME found an issue related to our hangs. Here are the details.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The IPA schema compat plugin for RHDS deadlocks on delete post op. One
thread is trying to acquire a write lock while in delete post op while
several other threads are trying to acquire read locks in bind ops. It
is not clear which thread is actually holding the lock/s. No FATAL
error messages are reported in the logs for lock/unlock API. Because
of this issue the RHDS gets into a "hang" like state where it is not
responsive to operations which is obviously a problem. There is one
thread that is doing post op plugin ops for add op tho it is not in
schema compat plugin. And there are a few threads that just stuck in
backend search ops inside bdb code likely because the locks they need
are held by thread/s stuck in schema compat plugin operations.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
A bug was opened to further analysis from our Developers.
- [2124214 – schema compat plugin deadlock on delete post
op](https://bugzilla.redhat.com/show_bug.cgi?id=2124214)
On Tue, Sep 6, 2022 at 4:16 AM Kees Bakker via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org> wrote:
On 30-08-2022 23:20, Kathy Zhu via FreeIPA-users wrote:
Hi Team,
We used following to get the number of rwlocks for /usr/sbin/ns-slapd
process in Centos 7.9 to catch deadlocks:
PID=`pidof ns-slapd`
gdb -ex 'set confirm off' -ex 'set pagination off' -ex 'thread apply
all
bt full' -ex 'quit' /usr/sbin/ns-slapd $PID |& grep '^#0.*lock' |
grep
pthread_rwlock | sort -u
That helped us to detect ns-slapd hang caused by deadlocks.
After migrating to Red Hat 8.6, we had a lot of hangs (dirsvr is running
but not responding) and could not find why. We use the same above method,
however, we are not able to catch anything. I wonder if there is a
different way to count the rwlocks in Red Hat 8.6?
We realize that there are multiple reasons to cause hangs, however, we
would like to rule out the possibility of the deadlock.
The OS and packages:
Red Hat Enterprise Linux release 8.6 (Ootpa)
ipa-server.x86_64 4.9.8-7.module+el8.6.0+14337+19b76db2
@rhel-8-for-x86_64-appstream-rpms
slapi-nis-0.56.6-4.module+el8.6.0+12936+736896b2.x86_64
389-ds-base-libs-1.4.3.28-6.module+el8.6.0+14129+983ceada.x86_64
389-ds-base-1.4.3.28-6.module+el8.6.0+14129+983ceada.x86_64
Hey Kathy,
May I suggest to also look at the 389-users mailing list
389-users(a)lists.fedoraproject.org
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproje...
For example threads with subject:
* Lock table is out of available lock entries
* Enabling retro changelog maxage with 3 million entries make dirsrv not
respond anymore
* DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
I have reported about hangs (deadlocks). In our case when Retro Changelog
trimming
is enabled we get a hangs within a few minutes. As far as I know there is
no solution yet.
That is why we have disabled this trimming. (And our Retro Changelog has
accumulated to more
than Gb of data.)
--
Kees
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
597
days inactive
604
days old
freeipa-users@lists.fedorahosted.org
3 comments
3 participants
participants (3)
-
Kathy Zhu -
Kees Bakker -
Thierry Bordaz