On 07/23/2014 11:24 AM, Thomas Walker wrote:
Hi, I've recently been trying to hunt down some odd performance
problems with our installation of 389 LDAP (currently 1.3.2.19 but been following recent
debian unstable). We've been seeing long delays (tens of seconds at times) handling
even the simplest new bind()s while the server otherwise has idle worker threads (and
other non-idle worker threads servicing existing conenctions).
Upon grabbing some userland thread stacks during these "hangs" when no new
external connections could be established, I saw what looked to be the thread associated
with slapd_daemon() in ldap/servers/slapd/daemon.c hung up in setup_pr_read_pds() walking
the list of active connections acquiring connection locks (c->c_mutex) sequentially in
the process. I stuck some calls to clock_gettime() around the PR_Lock(c->c_mutex) call
or or about ldap/servers/slapd/daemon.c:1690 and warned when we waitied for more than a
set duration:
[22/Jul/2014:17:37:05 +0000] - setup_pr_read_pds: (fd=192) waited 995.375473 msecs for
lock
[22/Jul/2014:17:37:08 +0000] - setup_pr_read_pds: (fd=202) waited 3003.548263 msecs for
lock
[22/Jul/2014:17:37:10 +0000] - setup_pr_read_pds: (fd=181) waited 1997.828897 msecs for
lock
<up to 20-30 seconds in some extreme cases>
It looks like this could hang for up to CONN_TURBO_TIMEOUT_INTERVAL (default 1 second)
per thread in turbo (up to 50% of worker pool by default). While stuck there, it isn't
calling handle_listeners() to pull new connections off of the well known port.
Perhaps handle_listeners() should run off in its own thread, away from this connection
maitenance? (or if it must be there, a non-blocking PRP_TryLock() or somesuch?)
Not sure. This is a complex area.
Are you using IdM/FreeIPA or just plain 389?
Some stack traces taking during the run might be helpful. Please see
http://port389.org/wiki/FAQ#Debugging_Hangs
I guess you'll have to replace "yum" with "apt" and I'm not
sure how the
debuginfo packages are handled on Debian.
TIA
Thomas
--
389 users mailing list
389-users(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users