12:24 p.m.
Hi, I've recently been trying to hunt down some odd performance problems with our
installation of 389 LDAP (currently 1.3.2.19 but been following recent debian unstable).
We've been seeing long delays (tens of seconds at times) handling even the simplest
new bind()s while the server otherwise has idle worker threads (and other non-idle worker
threads servicing existing conenctions).
Upon grabbing some userland thread stacks during these "hangs" when no new
external connections could be established, I saw what looked to be the thread associated
with slapd_daemon() in ldap/servers/slapd/daemon.c hung up in setup_pr_read_pds() walking
the list of active connections acquiring connection locks (c->c_mutex) sequentially in
the process. I stuck some calls to clock_gettime() around the PR_Lock(c->c_mutex) call
or or about ldap/servers/slapd/daemon.c:1690 and warned when we waitied for more than a
set duration:
[22/Jul/2014:17:37:05 +0000] - setup_pr_read_pds: (fd=192) waited 995.375473 msecs for
lock
[22/Jul/2014:17:37:08 +0000] - setup_pr_read_pds: (fd=202) waited 3003.548263 msecs for
lock
[22/Jul/2014:17:37:10 +0000] - setup_pr_read_pds: (fd=181) waited 1997.828897 msecs for
lock
<up to 20-30 seconds in some extreme cases>
It looks like this could hang for up to CONN_TURBO_TIMEOUT_INTERVAL (default 1 second) per
thread in turbo (up to 50% of worker pool by default). While stuck there, it isn't
calling handle_listeners() to pull new connections off of the well known port.
Perhaps handle_listeners() should run off in its own thread, away from this connection
maitenance? (or if it must be there, a non-blocking PRP_TryLock() or somesuch?)
TIA
Thomas
7:23 p.m.
On 07/23/2014 11:24 AM, Thomas Walker wrote:
Hi, I've recently been trying to hunt down some odd performance
problems with our installation of 389 LDAP (currently 1.3.2.19 but been following recent
debian unstable). We've been seeing long delays (tens of seconds at times) handling
even the simplest new bind()s while the server otherwise has idle worker threads (and
other non-idle worker threads servicing existing conenctions).
Upon grabbing some userland thread stacks during these "hangs" when no new
external connections could be established, I saw what looked to be the thread associated
with slapd_daemon() in ldap/servers/slapd/daemon.c hung up in setup_pr_read_pds() walking
the list of active connections acquiring connection locks (c->c_mutex) sequentially in
the process. I stuck some calls to clock_gettime() around the PR_Lock(c->c_mutex) call
or or about ldap/servers/slapd/daemon.c:1690 and warned when we waitied for more than a
set duration:
[22/Jul/2014:17:37:05 +0000] - setup_pr_read_pds: (fd=192) waited 995.375473 msecs for
lock
[22/Jul/2014:17:37:08 +0000] - setup_pr_read_pds: (fd=202) waited 3003.548263 msecs for
lock
[22/Jul/2014:17:37:10 +0000] - setup_pr_read_pds: (fd=181) waited 1997.828897 msecs for
lock
<up to 20-30 seconds in some extreme cases>
It looks like this could hang for up to CONN_TURBO_TIMEOUT_INTERVAL (default 1 second)
per thread in turbo (up to 50% of worker pool by default). While stuck there, it isn't
calling handle_listeners() to pull new connections off of the well known port.
Perhaps handle_listeners() should run off in its own thread, away from this connection
maitenance? (or if it must be there, a non-blocking PRP_TryLock() or somesuch?)
Not sure. This is a complex area.
Are you using IdM/FreeIPA or just plain 389?
Some stack traces taking during the run might be helpful. Please see
http://port389.org/wiki/FAQ#Debugging_Hangs
I guess you'll have to replace "yum" with "apt" and I'm not
sure how the
debuginfo packages are handled on Debian.
TIA
Thomas
--
389 users mailing list
389-users(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
10:08 a.m.
On Wed, Jul 23, 2014 at 06:23:24PM -0600, Rich Megginson wrote:
Not sure. This is a complex area.
Are you using IdM/FreeIPA or just plain 389?
Just plain old 389 (backend for sssd) with a load balancer inbetween (which is flapping
when we get erratic response times to simple bind attempts).
A lightly loaded server is fine but once we get more than a frew dozen connections, the
pauses become a real problem...
Some stack traces taking during the run might be helpful. Please
see http://port389.org/wiki/FAQ#Debugging_Hangs
I guess you'll have to replace "yum" with "apt" and I'm not
sure how
the debuginfo packages are handled on Debian.
Ah, missed that bit of the website. Attaching a tarball of dumps from a hang this
morning. I had an external process "pinging" the LDAP server doing lookups for
a UID 'bazbar.$i' via loopback every few seconds. It's request at 1406210089
went unregistered by the server until 1406210106 (log timestamps below are GMT):
[24/Jul/2014:13:55:06 +0000] conn=301 fd=182 slot=182 connection from ::1 to ::1
[24/Jul/2014:13:55:06 +0000] conn=301 op=0 BIND dn="" method=128 version=3
[24/Jul/2014:13:55:06 +0000] conn=301 op=0 RESULT err=0 tag=97 nentries=0 etime=0
dn=""
[24/Jul/2014:13:55:06 +0000] conn=301 op=1 SRCH
base="ou=People,dc=n,dc=twosigma,dc=com" scope=2
filter="(&(uid=bazbar.7)(objectClass=posixAccount))" attrs=ALL
[24/Jul/2014:13:55:06 +0000] conn=301 op=1 RESULT err=0 tag=101 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=301 op=2 UNBIND
[24/Jul/2014:13:55:06 +0000] conn=301 op=2 fd=182 closed - U1
I've obfuscated a few of the buffers in the stack dumps to keep my security folks
happy, shouldn't be any trouble though. These "hangs" always seem to be
followed by a flurry of angry clients when things unjam:
[24/Jul/2014:13:55:04 +0000] conn=299 op=1 ABANDON targetop=NOTFOUND msgid=1
[24/Jul/2014:13:55:06 +0000] conn=302 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=307 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=308 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=309 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=313 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=315 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=316 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=317 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=318 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=319 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=320 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=321 op=1 ABANDON targetop=NOTFOUND msgid=1
[24/Jul/2014:13:55:06 +0000] conn=322 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
even a few TCP resets:
[24/Jul/2014:13:55:09 +0000] conn=323 op=-1 fd=182 closed error 104 (Connection reset by
peer) - TCP connection reset by peer.
[24/Jul/2014:13:55:09 +0000] conn=326 op=-1 fd=165 closed error 104 (Connection reset by
peer) - TCP connection reset by peer.
[24/Jul/2014:13:55:11 +0000] conn=330 op=-1 fd=276 closed error 104 (Connection reset by
peer) - TCP connection reset by peer.
[24/Jul/2014:13:55:11 +0000] conn=332 op=-1 fd=190 closed error 104 (Connection reset by
peer) - TCP connection reset by peer.
11:31 a.m.
Ah, as you'll note from the stacktraces in fact, I've managed to turn on
connection debug output and am also attaching the section of "errors" for the
same period of time that contains all the conncection logging info...
Interestingly, it registers a few new "connections" during the pause:
[24/Jul/2014:13:54:56 +0000] - new connection on 278
[24/Jul/2014:13:54:58 +0000] - new connection on 165
[24/Jul/2014:13:54:59 +0000] - new connection on 182
[24/Jul/2014:13:55:02 +0000] - new connection on 190
[24/Jul/2014:13:55:04 +0000] - new connection on 165
and then a a few dozen new conenctions at 13:55:06 when the flood gates seem to re-open.
On Thu, Jul 24, 2014 at 11:08:15AM -0400, Thomas Walker wrote:
On Wed, Jul 23, 2014 at 06:23:24PM -0600, Rich Megginson wrote:
>
> Not sure. This is a complex area.
>
> Are you using IdM/FreeIPA or just plain 389?
Just plain old 389 (backend for sssd) with a load balancer inbetween (which is flapping
when we get erratic response times to simple bind attempts).
A lightly loaded server is fine but once we get more than a frew dozen connections, the
pauses become a real problem...
11:55 a.m.
On 07/24/2014 09:08 AM, Thomas Walker wrote:
On Wed, Jul 23, 2014 at 06:23:24PM -0600, Rich Megginson wrote:
> Not sure. This is a complex area.
>
> Are you using IdM/FreeIPA or just plain 389?
Just plain old 389 (backend for sssd) with a load balancer inbetween (which is flapping
when we get erratic response times to simple bind attempts).
A lightly loaded server is fine but once we get more than a frew dozen connections, the
pauses become a real problem...
> Some stack traces taking during the run might be helpful. Please
> see http://port389.org/wiki/FAQ#Debugging_Hangs
> I guess you'll have to replace "yum" with "apt" and I'm
not sure how
> the debuginfo packages are handled on Debian.
>
Ah, missed that bit of the website. Attaching a tarball of dumps from a hang this
morning. I had an external process "pinging" the LDAP server doing lookups for
a UID 'bazbar.$i' via loopback every few seconds. It's request at 1406210089
went unregistered by the server until 1406210106 (log timestamps below are GMT):
[24/Jul/2014:13:55:06 +0000] conn=301 fd=182 slot=182 connection from ::1 to ::1
[24/Jul/2014:13:55:06 +0000] conn=301 op=0 BIND dn="" method=128 version=3
[24/Jul/2014:13:55:06 +0000] conn=301 op=0 RESULT err=0 tag=97 nentries=0 etime=0
dn=""
[24/Jul/2014:13:55:06 +0000] conn=301 op=1 SRCH
base="ou=People,dc=n,dc=twosigma,dc=com" scope=2
filter="(&(uid=bazbar.7)(objectClass=posixAccount))" attrs=ALL
[24/Jul/2014:13:55:06 +0000] conn=301 op=1 RESULT err=0 tag=101 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=301 op=2 UNBIND
[24/Jul/2014:13:55:06 +0000] conn=301 op=2 fd=182 closed - U1
I've obfuscated a few of the buffers in the stack dumps to keep my security folks
happy, shouldn't be any trouble though. These "hangs" always seem to be
followed by a flurry of angry clients when things unjam:
[24/Jul/2014:13:55:04 +0000] conn=299 op=1 ABANDON targetop=NOTFOUND msgid=1
[24/Jul/2014:13:55:06 +0000] conn=302 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=307 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=308 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=309 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=313 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=315 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=316 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=317 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=318 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=319 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=320 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
[24/Jul/2014:13:55:06 +0000] conn=321 op=1 ABANDON targetop=NOTFOUND msgid=1
[24/Jul/2014:13:55:06 +0000] conn=322 op=1 ABANDON targetop=0 msgid=1 nentries=0 etime=0
even a few TCP resets:
[24/Jul/2014:13:55:09 +0000] conn=323 op=-1 fd=182 closed error 104 (Connection reset by
peer) - TCP connection reset by peer.
[24/Jul/2014:13:55:09 +0000] conn=326 op=-1 fd=165 closed error 104 (Connection reset by
peer) - TCP connection reset by peer.
[24/Jul/2014:13:55:11 +0000] conn=330 op=-1 fd=276 closed error 104 (Connection reset by
peer) - TCP connection reset by peer.
[24/Jul/2014:13:55:11 +0000] conn=332 op=-1 fd=190 closed error 104 (Connection reset by
peer) - TCP connection reset by peer.
Some initial observations based on the stack traces:
1) I think you are running too many threads. Having too many threads
causes too much thread contention, which leads to performance problems.
Did you set nsslapd-threadnumber? If so, try setting it to 2 times the
number of cores on your system. If you didn't touch
nsslapd-threadnumber, then it must be a bug, because the default number
is 30 (for historical reasons).
2) Thread 1 summary:
fsync fsync ?? vslapd_log_error slapd_log_error_proc_internal
slapd_log_error_proc handle_pr_read_ready slapd_daemon main
When doing stack traces, it is best to do it with no additional error
log levels. This fsync() in the daemon main loop is very expensive.
3) There are a lot of stack traces waiting on locks related to virtual
attribute lookups, for example:
Thread 86
?? ?? ?? ?? PR_vsmprintf slapi_ch_smprintf
vattr_map_namespace_sp_getlist slapi_vattr_namespace_values_get_sp
send_specific_attrs send_ldap_search_entry_ext send_ldap_search_entry
iterate send_results_ext op_shared_search do_search
connection_dispatch_operation ?? start_thread clone ??
You might want to see if this is a source of your performance problems.
Are you using Class of Service, Roles, per-subtree Password Policy?
Try setting nsslapd-ignore-virtual-attrs: on in cn=config to disable
virtual attribute lookups. At least, that will either rule out vattrs
as the source of the performance problems, or point out that we really
need to address this performance sooner rather than later.
4) There are a lot of threads waiting for some sort of input from the
client e.g.
Thread 98
poll poll ?? connection_read_operation connection_threadmain ??
start_thread clone ??
This means the client has initiated an operation but has not completed
sending the data, either due to client problems, or there is some sort
of network problem preventing the server's receipt of the data.
Here is the list of all of the "interesting" threads (that is, threads
not just sitting in connection_wait_for_new_work):
Thread 126
poll poll ?? slapd_poll write_function openldap_write_function
ber_int_sb_write ber_flush2 flush_ber send_ldap_search_entry_ext
send_ldap_search_entry iterate send_results_ext op_shared_search
do_search connection_dispatch_operation ?? start_thread clone ??
Thread 124
__lll_lock_wait __lll_lock_wait pthread_rwlock_rdlock
slapi_rwlock_rdlock vattr_map_lookup vattr_map_namespace_sp_getlist
vattr_test_filter slapi_vattr_filter_test_ext_internal
vattr_test_filter_list slapi_vattr_filter_test_ext_internal
slapi_vattr_filter_test_ext slapi_vattr_filter_test
ldbm_back_next_search_entry_ext iterate send_results_ext
op_shared_search do_search connection_dispatch_operation ?? start_thread
clone ??
Thread 120
__lll_lock_wait __lll_lock_wait pthread_rwlock_rdlock
slapi_rwlock_rdlock vattr_map_lookup vattr_map_namespace_sp_getlist
vattr_test_filter slapi_vattr_filter_test_ext_internal
vattr_test_filter_list slapi_vattr_filter_test_ext_internal
slapi_vattr_filter_test_ext slapi_vattr_filter_test
ldbm_back_next_search_entry_ext iterate send_results_ext
op_shared_search do_search connection_dispatch_operation ?? start_thread
clone ??
Thread 115
__lll_lock_wait __lll_lock_wait _L_lock_1145 pthread_mutex_lock PR_Lock
connection_threadmain ?? start_thread clone ??
Thread 114
__lll_lock_wait __lll_lock_wait pthread_rwlock_rdlock
slapi_rwlock_rdlock vattr_map_lookup vattr_map_namespace_sp_getlist
vattr_test_filter slapi_vattr_filter_test_ext_internal
slapi_vattr_filter_test_ext_internal vattr_test_filter_list
slapi_vattr_filter_test_ext_internal slapi_vattr_filter_test_ext
slapi_vattr_filter_test ldbm_back_next_search_entry_ext iterate
send_results_ext op_shared_search do_search
connection_dispatch_operation ?? start_thread clone ??
Thread 113
__lll_lock_wait __lll_lock_wait _L_lock_1145 pthread_mutex_lock PR_Lock
connection_threadmain ?? start_thread clone ??
Thread 112
__lll_lock_wait __lll_lock_wait _L_lock_1145 pthread_mutex_lock PR_Lock
connection_set_ssl_ssf connection_dispatch_operation
connection_threadmain ?? start_thread clone ??
Thread 107
pthread_rwlock_rdlock pthread_rwlock_rdlock slapi_rwlock_rdlock
vattr_map_lookup vattr_map_namespace_sp_getlist
slapi_vattr_namespace_values_get_sp send_specific_attrs
send_ldap_search_entry_ext send_ldap_search_entry iterate
send_results_ext op_shared_search do_search
connection_dispatch_operation ?? start_thread clone ??
Thread 106
__lll_lock_wait __lll_lock_wait pthread_rwlock_rdlock
slapi_rwlock_rdlock vattr_map_lookup vattr_map_namespace_sp_getlist
vattr_test_filter slapi_vattr_filter_test_ext_internal
slapi_vattr_filter_test_ext_internal vattr_test_filter_list
slapi_vattr_filter_test_ext_internal slapi_vattr_filter_test_ext
slapi_vattr_filter_test ldbm_back_next_search_entry_ext iterate
send_results_ext op_shared_search do_search
connection_dispatch_operation ?? start_thread clone ??
Thread 105
poll poll ?? slapd_poll write_function openldap_write_function
ber_int_sb_write ber_flush2 flush_ber send_ldap_search_entry_ext
send_ldap_search_entry iterate send_results_ext op_shared_search
do_search connection_dispatch_operation ?? start_thread clone ??
Thread 101
__lll_lock_wait __lll_lock_wait pthread_rwlock_rdlock
slapi_rwlock_rdlock aclanom_get_suffix_info acl_access_allowed
acl_access_allowed_main plugin_call_acl_plugin test_filter_access
slapi_vattr_filter_test_ext_internal vattr_test_filter_list
slapi_vattr_filter_test_ext_internal slapi_vattr_filter_test_ext
slapi_vattr_filter_test ldbm_back_next_search_entry_ext iterate
send_results_ext op_shared_search do_search
connection_dispatch_operation ?? start_thread clone ??
Thread 99
__lll_lock_wait __lll_lock_wait pthread_rwlock_rdlock
slapi_rwlock_rdlock vattr_map_lookup vattr_map_namespace_sp_getlist
slapi_vattr_namespace_values_get_sp send_specific_attrs
send_ldap_search_entry_ext send_ldap_search_entry iterate
send_results_ext op_shared_search do_search
connection_dispatch_operation ?? start_thread clone ??
Thread 98
poll poll ?? connection_read_operation connection_threadmain ??
start_thread clone ??
Thread 97
poll poll ?? connection_read_operation connection_threadmain ??
start_thread clone ??
Thread 90
__lll_lock_wait __lll_lock_wait _L_lock_1145 pthread_mutex_lock PR_Lock
connection_threadmain ?? start_thread clone ??
Thread 86
?? ?? ?? ?? PR_vsmprintf slapi_ch_smprintf
vattr_map_namespace_sp_getlist slapi_vattr_namespace_values_get_sp
send_specific_attrs send_ldap_search_entry_ext send_ldap_search_entry
iterate send_results_ext op_shared_search do_search
connection_dispatch_operation ?? start_thread clone ??
Thread 81
__lll_lock_wait __lll_lock_wait pthread_rwlock_rdlock
slapi_rwlock_rdlock aclanom_get_suffix_info acl_access_allowed
acl_access_allowed_main plugin_call_acl_plugin test_filter_access
slapi_vattr_filter_test_ext_internal vattr_test_filter_list
slapi_vattr_filter_test_ext_internal slapi_vattr_filter_test_ext
slapi_vattr_filter_test ldbm_back_next_search_entry_ext iterate
send_results_ext op_shared_search do_search
connection_dispatch_operation ?? start_thread clone ??
Thread 1
fsync fsync ?? vslapd_log_error slapd_log_error_proc_internal
slapd_log_error_proc handle_pr_read_ready slapd_daemon main
--
389 users mailing list
389-users(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
1:50 p.m.
On Thu, Jul 24, 2014 at 10:55:54AM -0600, Rich Megginson wrote:
1) I think you are running too many threads. Having too many threads causes
too much thread contention, which leads to performance problems. Did you set
nsslapd-threadnumber? If so, try setting it to 2 times the number of cores on
your system. If you didn't touch nsslapd-threadnumber, then it must be a bug,
because the default number is 30 (for historical reasons).
Thanks, inherited config from a predecessor. Cranked that back down significantly.
2) Thread 1 summary:
fsync fsync ?? vslapd_log_error slapd_log_error_proc_internal
slapd_log_error_proc handle_pr_read_ready slapd_daemon main
When doing stack traces, it is best to do it with no additional error log
levels. This fsync() in the daemon main loop is very expensive.
Done.
3) There are a lot of stack traces waiting on locks related to virtual
attribute lookups, for example:
Thread 86
?? ?? ?? ?? PR_vsmprintf slapi_ch_smprintf vattr_map_namespace_sp_getlist
slapi_vattr_namespace_values_get_sp send_specific_attrs
send_ldap_search_entry_ext send_ldap_search_entry iterate send_results_ext
op_shared_search do_search connection_dispatch_operation ?? start_thread clone
??
And done.
Attaching some new stacktraces with those out of the way.
2:50 p.m.
On 07/24/2014 12:50 PM, Thomas Walker wrote:
On Thu, Jul 24, 2014 at 10:55:54AM -0600, Rich Megginson wrote:
> 1) I think you are running too many threads. Having too many threads causes
> too much thread contention, which leads to performance problems. Did you set
> nsslapd-threadnumber? If so, try setting it to 2 times the number of cores on
> your system. If you didn't touch nsslapd-threadnumber, then it must be a bug,
> because the default number is 30 (for historical reasons).
Thanks, inherited config from a predecessor. Cranked that back down significantly.
> 2) Thread 1 summary:
> fsync fsync ?? vslapd_log_error slapd_log_error_proc_internal
> slapd_log_error_proc handle_pr_read_ready slapd_daemon main
>
> When doing stack traces, it is best to do it with no additional error log
> levels. This fsync() in the daemon main loop is very expensive.
Done.
> 3) There are a lot of stack traces waiting on locks related to virtual
> attribute lookups, for example:
> Thread 86
> ?? ?? ?? ?? PR_vsmprintf slapi_ch_smprintf vattr_map_namespace_sp_getlist
> slapi_vattr_namespace_values_get_sp send_specific_attrs
> send_ldap_search_entry_ext send_ldap_search_entry iterate send_results_ext
> op_shared_search do_search connection_dispatch_operation ?? start_thread clone
> ??
And done.
Attaching some new stacktraces with those out of the way.
Did the above make any significant difference with respect to the
performance?
--
389 users mailing list
389-users(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
3:14 p.m.
On Thu, Jul 24, 2014 at 01:50:39PM -0600, Rich Megginson wrote:
Did the above make any significant difference with respect to the performance?
The problem is definitely load related and transient so it is hard to say immediately. I
definitely still see many multisecond bind times, althought they're more of the 3-6
second variety so far (as opposed to teens in the teens like this morning). Since it
comes and goes, I'll need to let it run for awhile to see how things average out.
8:59 a.m.
On Thu, Jul 24, 2014 at 04:14:13PM -0400, Thomas Walker wrote:
On Thu, Jul 24, 2014 at 01:50:39PM -0600, Rich Megginson wrote:
>
>
> Did the above make any significant difference with respect to the performance?
>
>
The problem is definitely load related and transient so it is hard to say immediately. I
definitely still see many multisecond bind times, althought they're more of the 3-6
second variety so far (as opposed to teens in the teens like this morning). Since it
comes and goes, I'll need to let it run for awhile to see how things average out.
Can definitely confirm that it has made a substantial difference... Max and average
response times are about 25% of what they were. Still get into high single digit seconds
in some cases, but a significant improvement to be sure.
2:04 p.m.
On 07/25/2014 07:59 AM, Thomas Walker wrote:
On Thu, Jul 24, 2014 at 04:14:13PM -0400, Thomas Walker wrote:
> On Thu, Jul 24, 2014 at 01:50:39PM -0600, Rich Megginson wrote:
>>
>> Did the above make any significant difference with respect to the performance?
>>
>>
> The problem is definitely load related and transient so it is hard to say
immediately. I definitely still see many multisecond bind times, althought they're
more of the 3-6 second variety so far (as opposed to teens in the teens like this
morning). Since it comes and goes, I'll need to let it run for awhile to see how
things average out.
Can definitely confirm that it has made a substantial difference... Max and average
response times are about 25% of what they were. Still get into high single digit seconds
in some cases, but a significant improvement to be sure.
Ok. I suggest that you open a ticket at
https://fedorahosted.org/389/newticket so that we can track this issue,
discuss, attach files, etc.
--
389 users mailing list
389-users(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
3556
days inactive
3558
days old
389-users@lists.fedoraproject.org
9 comments
2 participants
participants (2)
-
Rich Megginson -
Thomas Walker