9:46 p.m.
Hi,
I have been running FDS/389 on a F11 xen DomU for several months. I use
it as the backend for UNIX username/passwords and also for redMine (a
Ruby on Rails bug tracker) for http://www.gnucapplus.org/.
This VM would regularly lock up every week or so when 389 was still
called FDS. I've since upgraded to 389 by issuing 'yum upgrade' as well
as running the 'setup-...-.pl -u' script and now it barely goes a day
before crashing. When ldap crashes, the whole box basically becomes
unresponsive.
I left the Xen hardware console open to see what was up and the only
thing I could conclude was that 389 was crashing (if I issued a service
start it came back to life). Doing anything like a top or ls will
completely kill the box. Likewise, the logs show nothing at or before
the time of crash. I suspected too few file descriptors but changing
that to a very high number had no impact.
I was about to do a rip and replace with OpenLDAP which I use very
sucesessfully for our corporate systems but figured I ought to see if
anyone here can help or if I can submit any kind of meaningful bug
report first. I assume I will need to run 389's slapd without
daemonizing it and hope it spits something useful out to stderr. Any
advice here would be greatly appreciated, as would any success stories
of using 389 on F11.
I'm not subscribed to the list so please CC.
Regards,
Kevin Bowing
11:43 a.m.
On 09/10/2009 07:46 PM, Kevin Bowling wrote:
Hi,
I have been running FDS/389 on a F11 xen DomU for several months. I
use it as the backend for UNIX username/passwords and also for redMine
(a Ruby on Rails bug tracker) for http://www.gnucapplus.org/.
This VM would regularly lock up every week or so when 389 was still
called FDS. I've since upgraded to 389 by issuing 'yum upgrade' as
well as running the 'setup-...-.pl -u' script and now it barely goes a
day before crashing. When ldap crashes, the whole box basically
becomes unresponsive.
I left the Xen hardware console open to see what was up and the only
thing I could conclude was that 389 was crashing (if I issued a
service start it came back to life). Doing anything like a top or ls
will completely kill the box. Likewise, the logs show nothing at or
before the time of crash. I suspected too few file descriptors but
changing that to a very high number had no impact.
I was about to do a rip and replace with OpenLDAP which I use very
sucesessfully for our corporate systems but figured I ought to see if
anyone here can help or if I can submit any kind of meaningful bug
report first. I assume I will need to run 389's slapd without
daemonizing it and hope it spits something useful out to stderr. Any
advice here would be greatly appreciated, as would any success stories
of using 389 on F11.
Hello Kevin,
You specified the platform "F11 xen DomU". Did you have a chance to run
the 389 server on any other platforms? I'm wondering if the crash is
observed only on the specific platform or not. Is the server running on
the 64-bit machine or 32-bit?
If you start the server with "-d 1" option, the server will run as the
trace mode. (E.g., /usr/lib[64]/dirsrv/slapd-YOURID/start-slapd -d 1)
I'm afraid it might be a memory leak. When you restart the 389 server,
could you check the size of ns-slapd some time like every hour and see
if the server size keeps growing or stops? Also, the server quits if it
fails to write to the errors log. If it happens, it's logged in the
system log. Does the messages file on the system happen to have some
logs related to the 389 server?
Thanks,
--noriko
I'm not subscribed to the list so please CC.
Regards,
Kevin Bowing
--
389 users mailing list
389-users(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-directory-users
12:14 p.m.
On 9/11/2009 12:43 PM, Noriko Hosoi wrote:
On 09/10/2009 07:46 PM, Kevin Bowling wrote:
> Hi,
>
> I have been running FDS/389 on a F11 xen DomU for several months. I
> use it as the backend for UNIX username/passwords and also for
> redMine (a Ruby on Rails bug tracker) for http://www.gnucapplus.org/.
>
> This VM would regularly lock up every week or so when 389 was still
> called FDS. I've since upgraded to 389 by issuing 'yum upgrade' as
> well as running the 'setup-...-.pl -u' script and now it barely goes
> a day before crashing. When ldap crashes, the whole box basically
> becomes unresponsive.
>
> I left the Xen hardware console open to see what was up and the only
> thing I could conclude was that 389 was crashing (if I issued a
> service start it came back to life). Doing anything like a top or ls
> will completely kill the box. Likewise, the logs show nothing at or
> before the time of crash. I suspected too few file descriptors but
> changing that to a very high number had no impact.
>
> I was about to do a rip and replace with OpenLDAP which I use very
> sucesessfully for our corporate systems but figured I ought to see if
> anyone here can help or if I can submit any kind of meaningful bug
> report first. I assume I will need to run 389's slapd without
> daemonizing it and hope it spits something useful out to stderr. Any
> advice here would be greatly appreciated, as would any success
> stories of using 389 on F11.
Hello Kevin,
You specified the platform "F11 xen DomU". Did you have a chance to
run the 389 server on any other platforms? I'm wondering if the crash
is observed only on the specific platform or not. Is the server
running on the 64-bit machine or 32-bit?
If you start the server with "-d 1" option, the server will run as the
trace mode. (E.g., /usr/lib[64]/dirsrv/slapd-YOURID/start-slapd -d 1)
I'm afraid it might be a memory leak. When you restart the 389
server, could you check the size of ns-slapd some time like every hour
and see if the server size keeps growing or stops? Also, the server
quits if it fails to write to the errors log. If it happens, it's
logged in the system log. Does the messages file on the system
happen to have some logs related to the 389 server?
Thanks,
--noriko
>
> I'm not subscribed to the list so please CC.
>
> Regards,
>
> Kevin Bowing
>
> --
> 389 users mailing list
> 389-users(a)redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-directory-users
I captured some output while running in trace, see the end of this
message. The system is 64-bit, I have not run on any other boxes. A
cursory look with top showed only 10MB or so RSS memory.
Regards,
Kevin
[11/Sep/2009:09:58:44 -0700] - => id2entry( 48 )
[11/Sep/2009:09:58:44 -0700] - <= id2entry 7f025401f5a0 (cache)
[11/Sep/2009:09:58:44 -0700] - => id2entry( 50 )
[11/Sep/2009:09:58:44 -0700] - <= id2entry 7f0254021190 (cache)
[11/Sep/2009:09:58:44 -0700] - => slapi_reslimit_get_integer_limit()
conn=0xa856beb0, handle=3
[11/Sep/2009:09:58:44 -0700] - <= slapi_reslimit_get_integer_limit()
returning NO VALUE
[11/Sep/2009:09:58:44 -0700] - => slapi_reslimit_get_integer_limit()
conn=0xa856bc60, handle=3
[11/Sep/2009:09:58:44 -0700] - <= slapi_reslimit_get_integer_limit()
returning NO VALUE
[11/Sep/2009:09:58:44 -0700] - => slapi_reslimit_get_integer_limit()
conn=0xa856bd88, handle=3
[11/Sep/2009:09:58:44 -0700] - <= slapi_reslimit_get_integer_limit()
returning NO VALUE
[11/Sep/2009:09:58:44 -0700] - => slapi_reslimit_get_integer_limit()
conn=0xa856bb38, handle=3
[11/Sep/2009:09:58:44 -0700] - <= slapi_reslimit_get_integer_limit()
returning NO VALUE
[11/Sep/2009:09:58:44 -0700] - => slapi_reslimit_get_integer_limit()
conn=0xa856ba10, handle=3
[11/Sep/2009:09:58:44 -0700] - <= slapi_reslimit_get_integer_limit()
returning NO VALUE
[11/Sep/2009:09:58:44 -0700] - => send_ldap_result 0::
[11/Sep/2009:09:58:44 -0700] - <= send_ldap_result
[11/Sep/2009:09:58:50 -0700] - ldbm backend flushing
[11/Sep/2009:09:58:50 -0700] - ldbm backend done flushing
[11/Sep/2009:09:58:50 -0700] - ldbm backend flushing
[11/Sep/2009:09:58:50 -0700] - ldbm backend done flushing
[11/Sep/2009:09:59:20 -0700] - ldbm backend flushing
[11/Sep/2009:09:59:20 -0700] - ldbm backend done flushing
[11/Sep/2009:09:59:20 -0700] - ldbm backend flushing
[11/Sep/2009:09:59:20 -0700] - ldbm backend done flushing
[11/Sep/2009:09:59:50 -0700] - ldbm backend flushing
[11/Sep/2009:09:59:50 -0700] - ldbm backend done flushing
[11/Sep/2009:09:59:50 -0700] - ldbm backend flushing
[11/Sep/2009:09:59:50 -0700] - ldbm backend done flushing
[11/Sep/2009:10:00:20 -0700] - ldbm backend flushing
[11/Sep/2009:10:00:20 -0700] - ldbm backend done flushing
[11/Sep/2009:10:00:20 -0700] - ldbm backend flushing
[11/Sep/2009:10:00:20 -0700] - ldbm backend done flushing
[11/Sep/2009:10:00:50 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:03 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:03 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:04 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:35 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:39 -0700] - slapd shutting down - signaling operation
threads
[11/Sep/2009:10:01:40 -0700] - slapd shutting down - waiting for 30
threads to terminate
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - slapd shutting down - waiting for 29
threads to terminate
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - slapd shutting down - waiting for 28
threads to terminate
[11/Sep/2009:10:01:41 -0700] - slapd shutting down - closing down
internal subsystems and plugins
[11/Sep/2009:10:01:41 -0700] - slapd shutting down - waiting for
backends to close down
[11/Sep/2009:10:01:42 -0700] - => slapi_control_present (looking for
1.3.6.1.4.1.42.2.27.8.5.1)
[11/Sep/2009:10:01:42 -0700] - <= slapi_control_present 0 (NO CONTROLS)
[11/Sep/2009:10:01:42 -0700] - modify_update_last_modified_attr
[11/Sep/2009:10:01:42 -0700] - Calling plugin 'Distributed Numeric
Assignment internal preop plugin' #0 type 421
[11/Sep/2009:10:01:42 -0700] dna-plugin - --> dna_pre_op
[11/Sep/2009:10:01:42 -0700] dna-plugin - <-- dna_pre_op
[11/Sep/2009:10:01:42 -0700] - Calling plugin 'Legacy replication
internal preoperation plugin' #1 type 421
[11/Sep/2009:10:01:42 -0700] - Calling plugin 'Multimaster replication
internal preoperation plugin' #2 type 421
[11/Sep/2009:10:01:42 -0700] - => entry_apply_mods
[11/Sep/2009:10:01:42 -0700] - <= entry_apply_mods 0
[11/Sep/2009:10:01:42 -0700] - => send_ldap_result 0::
[11/Sep/2009:10:01:42 -0700] - <= send_ldap_result
[11/Sep/2009:10:01:42 -0700] - ps_service_persistent_searches: entry
"cn=uniqueid generator,cn=config" not enqueued on any persistent search
lists
[11/Sep/2009:10:01:42 -0700] - Calling plugin 'Class of Service
internalpostoperation plugin' #0 type 521
[11/Sep/2009:10:01:42 -0700] - --> cos_post_op
[11/Sep/2009:10:01:42 -0700] - --> cos_cache_change_notify
[11/Sep/2009:10:01:42 -0700] - --> cos_cache_template_index_bsearch
[11/Sep/2009:10:01:42 -0700] - --> cos_cache_getref
[11/Sep/2009:10:01:42 -0700] - <-- cos_cache_getref
[11/Sep/2009:10:01:42 -0700] - <-- cos_cache_template_index_bsearch
[11/Sep/2009:10:01:42 -0700] - <-- cos_cache_change_notify
[11/Sep/2009:10:01:42 -0700] - <-- cos_post_op
[11/Sep/2009:10:01:42 -0700] - Calling plugin 'Legacy replication
internal postoperation plugin' #1 type 521
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Multimaster replication
internal postoperation plugin' #2 type 521
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Retrocl internal
postoperation plugin' #3 type 521
not applying change if not logging
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Roles
internalpostoperation plugin' #4 type 521
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Legacy Replication
Plugin' #0 type 210
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Roles Plugin' #0 type 210
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Multimaster Replication
Plugin' #0 type 210
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'HTTP Client' #0 type 210
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Class of Service' #0 type 210
[11/Sep/2009:10:01:43 -0700] - --> cos_close
[11/Sep/2009:10:01:43 -0700] - --> cos_cache_stop
[11/Sep/2009:10:01:43 -0700] - <-- cos_cache_wait_on_change thread exit
[11/Sep/2009:10:01:43 -0700] - --> cos_cache_release
[11/Sep/2009:10:01:43 -0700] - <-- cos_cache_release
[11/Sep/2009:10:01:43 -0700] - <-- cos_cache_stop
[11/Sep/2009:10:01:43 -0700] - <-- cos_close
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'ACL Plugin' #0 type 210
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Views' #0 type 210
[11/Sep/2009:10:01:43 -0700] views-plugin - --> views_close
[11/Sep/2009:10:01:43 -0700] views-plugin - --> views_cache_free
[11/Sep/2009:10:01:43 -0700] views-plugin - <-- views_cache_free
[11/Sep/2009:10:01:43 -0700] views-plugin - <-- views_close
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'State Change Plugin' #0
type 210
[11/Sep/2009:10:01:43 -0700] statechange-plugin - --> statechange_close
[11/Sep/2009:10:01:43 -0700] statechange-plugin - <-- statechange_close
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'ldbm database' #0 type 210
[11/Sep/2009:10:01:43 -0700] - ldbm backend syncing
[11/Sep/2009:10:01:43 -0700] - Waiting for 4 database threads to stop
[11/Sep/2009:10:01:43 -0700] - Leaving deadlock_threadmain
[11/Sep/2009:10:01:44 -0700] - Leaving checkpoint_threadmain before
checkpoint
[11/Sep/2009:10:01:44 -0700] - Checkpointing database ...
[11/Sep/2009:10:01:44 -0700] - Leaving checkpoint_threadmain
[11/Sep/2009:10:01:44 -0700] - Leaving trickle_threadmain priv
[11/Sep/2009:10:01:44 -0700] - Leaving perf_threadmain
[11/Sep/2009:10:01:45 -0700] - All database threads now stopped
[11/Sep/2009:10:01:45 -0700] - ldbm backend done syncing
[11/Sep/2009:10:01:45 -0700] - Calling plugin 'chaining database' #0
type 210
[11/Sep/2009:10:01:45 -0700] - Removed [1] entries from the dse tree.
[11/Sep/2009:10:01:45 -0700] - Removed [166] entries from the dse tree.
[11/Sep/2009:10:01:45 -0700] - ldbm backend cleaning up
[11/Sep/2009:10:01:45 -0700] - ldbm backend cleaning up
[11/Sep/2009:10:01:45 -0700] - slapd shutting down - backends closed down
[11/Sep/2009:10:01:45 -0700] - => reslimit_update_from_entry()
conn=0xa856ba10, entry=0x0
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 0 (based on nsLookThroughLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 1 (based on nsSizeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 2 (based on nsTimeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 3 (based on nsIdleTimeout)
[11/Sep/2009:10:01:45 -0700] - <= reslimit_update_from_entry() returning
status 0
[11/Sep/2009:10:01:45 -0700] - => reslimit_update_from_entry()
conn=0xa856bb38, entry=0x0
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 0 (based on nsLookThroughLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 1 (based on nsSizeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 2 (based on nsTimeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 3 (based on nsIdleTimeout)
[11/Sep/2009:10:01:45 -0700] - <= reslimit_update_from_entry() returning
status 0
[11/Sep/2009:10:01:45 -0700] - => reslimit_update_from_entry()
conn=0xa856bd88, entry=0x0
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 0 (based on nsLookThroughLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 1 (based on nsSizeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 2 (based on nsTimeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 3 (based on nsIdleTimeout)
[11/Sep/2009:10:01:45 -0700] - <= reslimit_update_from_entry() returning
status 0
[11/Sep/2009:10:01:45 -0700] - => reslimit_update_from_entry()
conn=0xa856beb0, entry=0x0
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 0 (based on nsLookThroughLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 1 (based on nsSizeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 2 (based on nsTimeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 3 (based on nsIdleTimeout)
[11/Sep/2009:10:01:45 -0700] - <= reslimit_update_from_entry() returning
status 0
[11/Sep/2009:10:01:45 -0700] - slapd stopped.
12:23 p.m.
On 09/11/2009 10:14 AM, Kevin Bowling wrote:
[...]
I captured some output while running in trace, see the end of this
message. The system is 64-bit, I have not run on any other boxes. A
cursory look with top showed only 10MB or so RSS memory.
Your server received a
shutdown signal. It looks to me it's a normal
shutdown... (Not a server crash...)
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
*[11/Sep/2009:10:01:39 -0700] - slapd shutting down - signaling
operation threads *
*[11/Sep/2009:10:01:40 -0700] - slapd shutting down - waiting for 30
threads to terminate *
*[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - slapd shutting down - waiting for 29
threads to terminate *
[...]
Regards,
Kevin
[11/Sep/2009:09:58:44 -0700] - => id2entry( 48 )
[11/Sep/2009:09:58:44 -0700] - <= id2entry 7f025401f5a0 (cache)
[11/Sep/2009:09:58:44 -0700] - => id2entry( 50 )
[11/Sep/2009:09:58:44 -0700] - <= id2entry 7f0254021190 (cache)
[11/Sep/2009:09:58:44 -0700] - => slapi_reslimit_get_integer_limit()
conn=0xa856beb0, handle=3
[11/Sep/2009:09:58:44 -0700] - <= slapi_reslimit_get_integer_limit()
returning NO VALUE
[11/Sep/2009:09:58:44 -0700] - => slapi_reslimit_get_integer_limit()
conn=0xa856bc60, handle=3
[11/Sep/2009:09:58:44 -0700] - <= slapi_reslimit_get_integer_limit()
returning NO VALUE
[11/Sep/2009:09:58:44 -0700] - => slapi_reslimit_get_integer_limit()
conn=0xa856bd88, handle=3
[11/Sep/2009:09:58:44 -0700] - <= slapi_reslimit_get_integer_limit()
returning NO VALUE
[11/Sep/2009:09:58:44 -0700] - => slapi_reslimit_get_integer_limit()
conn=0xa856bb38, handle=3
[11/Sep/2009:09:58:44 -0700] - <= slapi_reslimit_get_integer_limit()
returning NO VALUE
[11/Sep/2009:09:58:44 -0700] - => slapi_reslimit_get_integer_limit()
conn=0xa856ba10, handle=3
[11/Sep/2009:09:58:44 -0700] - <= slapi_reslimit_get_integer_limit()
returning NO VALUE
[11/Sep/2009:09:58:44 -0700] - => send_ldap_result 0::
[11/Sep/2009:09:58:44 -0700] - <= send_ldap_result
[11/Sep/2009:09:58:50 -0700] - ldbm backend flushing
[11/Sep/2009:09:58:50 -0700] - ldbm backend done flushing
[11/Sep/2009:09:58:50 -0700] - ldbm backend flushing
[11/Sep/2009:09:58:50 -0700] - ldbm backend done flushing
[11/Sep/2009:09:59:20 -0700] - ldbm backend flushing
[11/Sep/2009:09:59:20 -0700] - ldbm backend done flushing
[11/Sep/2009:09:59:20 -0700] - ldbm backend flushing
[11/Sep/2009:09:59:20 -0700] - ldbm backend done flushing
[11/Sep/2009:09:59:50 -0700] - ldbm backend flushing
[11/Sep/2009:09:59:50 -0700] - ldbm backend done flushing
[11/Sep/2009:09:59:50 -0700] - ldbm backend flushing
[11/Sep/2009:09:59:50 -0700] - ldbm backend done flushing
[11/Sep/2009:10:00:20 -0700] - ldbm backend flushing
[11/Sep/2009:10:00:20 -0700] - ldbm backend done flushing
[11/Sep/2009:10:00:20 -0700] - ldbm backend flushing
[11/Sep/2009:10:00:20 -0700] - ldbm backend done flushing
[11/Sep/2009:10:00:50 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:03 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:03 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:04 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:35 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:39 -0700] - slapd shutting down - signaling
operation threads
[11/Sep/2009:10:01:40 -0700] - slapd shutting down - waiting for 30
threads to terminate
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - slapd shutting down - waiting for 29
threads to terminate
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:41 -0700] - slapd shutting down - waiting for 28
threads to terminate
[11/Sep/2009:10:01:41 -0700] - slapd shutting down - closing down
internal subsystems and plugins
[11/Sep/2009:10:01:41 -0700] - slapd shutting down - waiting for
backends to close down
[11/Sep/2009:10:01:42 -0700] - => slapi_control_present (looking for
1.3.6.1.4.1.42.2.27.8.5.1)
[11/Sep/2009:10:01:42 -0700] - <= slapi_control_present 0 (NO CONTROLS)
[11/Sep/2009:10:01:42 -0700] - modify_update_last_modified_attr
[11/Sep/2009:10:01:42 -0700] - Calling plugin 'Distributed Numeric
Assignment internal preop plugin' #0 type 421
[11/Sep/2009:10:01:42 -0700] dna-plugin - --> dna_pre_op
[11/Sep/2009:10:01:42 -0700] dna-plugin - <-- dna_pre_op
[11/Sep/2009:10:01:42 -0700] - Calling plugin 'Legacy replication
internal preoperation plugin' #1 type 421
[11/Sep/2009:10:01:42 -0700] - Calling plugin 'Multimaster replication
internal preoperation plugin' #2 type 421
[11/Sep/2009:10:01:42 -0700] - => entry_apply_mods
[11/Sep/2009:10:01:42 -0700] - <= entry_apply_mods 0
[11/Sep/2009:10:01:42 -0700] - => send_ldap_result 0::
[11/Sep/2009:10:01:42 -0700] - <= send_ldap_result
[11/Sep/2009:10:01:42 -0700] - ps_service_persistent_searches: entry
"cn=uniqueid generator,cn=config" not enqueued on any persistent
search lists
[11/Sep/2009:10:01:42 -0700] - Calling plugin 'Class of Service
internalpostoperation plugin' #0 type 521
[11/Sep/2009:10:01:42 -0700] - --> cos_post_op
[11/Sep/2009:10:01:42 -0700] - --> cos_cache_change_notify
[11/Sep/2009:10:01:42 -0700] - --> cos_cache_template_index_bsearch
[11/Sep/2009:10:01:42 -0700] - --> cos_cache_getref
[11/Sep/2009:10:01:42 -0700] - <-- cos_cache_getref
[11/Sep/2009:10:01:42 -0700] - <-- cos_cache_template_index_bsearch
[11/Sep/2009:10:01:42 -0700] - <-- cos_cache_change_notify
[11/Sep/2009:10:01:42 -0700] - <-- cos_post_op
[11/Sep/2009:10:01:42 -0700] - Calling plugin 'Legacy replication
internal postoperation plugin' #1 type 521
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Multimaster replication
internal postoperation plugin' #2 type 521
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Retrocl internal
postoperation plugin' #3 type 521
not applying change if not logging
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Roles
internalpostoperation plugin' #4 type 521
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Legacy Replication
Plugin' #0 type 210
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Roles Plugin' #0 type 210
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Multimaster Replication
Plugin' #0 type 210
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'HTTP Client' #0 type 210
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Class of Service' #0
type 210
[11/Sep/2009:10:01:43 -0700] - --> cos_close
[11/Sep/2009:10:01:43 -0700] - --> cos_cache_stop
[11/Sep/2009:10:01:43 -0700] - <-- cos_cache_wait_on_change thread exit
[11/Sep/2009:10:01:43 -0700] - --> cos_cache_release
[11/Sep/2009:10:01:43 -0700] - <-- cos_cache_release
[11/Sep/2009:10:01:43 -0700] - <-- cos_cache_stop
[11/Sep/2009:10:01:43 -0700] - <-- cos_close
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'ACL Plugin' #0 type 210
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'Views' #0 type 210
[11/Sep/2009:10:01:43 -0700] views-plugin - --> views_close
[11/Sep/2009:10:01:43 -0700] views-plugin - --> views_cache_free
[11/Sep/2009:10:01:43 -0700] views-plugin - <-- views_cache_free
[11/Sep/2009:10:01:43 -0700] views-plugin - <-- views_close
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'State Change Plugin' #0
type 210
[11/Sep/2009:10:01:43 -0700] statechange-plugin - --> statechange_close
[11/Sep/2009:10:01:43 -0700] statechange-plugin - <-- statechange_close
[11/Sep/2009:10:01:43 -0700] - Calling plugin 'ldbm database' #0 type 210
[11/Sep/2009:10:01:43 -0700] - ldbm backend syncing
[11/Sep/2009:10:01:43 -0700] - Waiting for 4 database threads to stop
[11/Sep/2009:10:01:43 -0700] - Leaving deadlock_threadmain
[11/Sep/2009:10:01:44 -0700] - Leaving checkpoint_threadmain before
checkpoint
[11/Sep/2009:10:01:44 -0700] - Checkpointing database ...
[11/Sep/2009:10:01:44 -0700] - Leaving checkpoint_threadmain
[11/Sep/2009:10:01:44 -0700] - Leaving trickle_threadmain priv
[11/Sep/2009:10:01:44 -0700] - Leaving perf_threadmain
[11/Sep/2009:10:01:45 -0700] - All database threads now stopped
[11/Sep/2009:10:01:45 -0700] - ldbm backend done syncing
[11/Sep/2009:10:01:45 -0700] - Calling plugin 'chaining database' #0
type 210
[11/Sep/2009:10:01:45 -0700] - Removed [1] entries from the dse tree.
[11/Sep/2009:10:01:45 -0700] - Removed [166] entries from the dse tree.
[11/Sep/2009:10:01:45 -0700] - ldbm backend cleaning up
[11/Sep/2009:10:01:45 -0700] - ldbm backend cleaning up
[11/Sep/2009:10:01:45 -0700] - slapd shutting down - backends closed down
[11/Sep/2009:10:01:45 -0700] - => reslimit_update_from_entry()
conn=0xa856ba10, entry=0x0
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 0 (based on nsLookThroughLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 1 (based on nsSizeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 2 (based on nsTimeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 3 (based on nsIdleTimeout)
[11/Sep/2009:10:01:45 -0700] - <= reslimit_update_from_entry()
returning status 0
[11/Sep/2009:10:01:45 -0700] - => reslimit_update_from_entry()
conn=0xa856bb38, entry=0x0
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 0 (based on nsLookThroughLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 1 (based on nsSizeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 2 (based on nsTimeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 3 (based on nsIdleTimeout)
[11/Sep/2009:10:01:45 -0700] - <= reslimit_update_from_entry()
returning status 0
[11/Sep/2009:10:01:45 -0700] - => reslimit_update_from_entry()
conn=0xa856bd88, entry=0x0
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 0 (based on nsLookThroughLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 1 (based on nsSizeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 2 (based on nsTimeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 3 (based on nsIdleTimeout)
[11/Sep/2009:10:01:45 -0700] - <= reslimit_update_from_entry()
returning status 0
[11/Sep/2009:10:01:45 -0700] - => reslimit_update_from_entry()
conn=0xa856beb0, entry=0x0
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 0 (based on nsLookThroughLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 1 (based on nsSizeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 2 (based on nsTimeLimit)
[11/Sep/2009:10:01:45 -0700] - reslimit_update_from_entry(): setting
limit for handle 3 (based on nsIdleTimeout)
[11/Sep/2009:10:01:45 -0700] - <= reslimit_update_from_entry()
returning status 0
[11/Sep/2009:10:01:45 -0700] - slapd stopped.
12:25 p.m.
On 9/11/2009 1:23 PM, Noriko Hosoi wrote:
On 09/11/2009 10:14 AM, Kevin Bowling wrote:
> [...]
>
> I captured some output while running in trace, see the end of this
> message. The system is 64-bit, I have not run on any other boxes. A
> cursory look with top showed only 10MB or so RSS memory.
Your server received a shutdown signal. It looks to me it's a normal
shutdown... (Not a server crash...)
I apologize, I had a script in cron.hourly to restart the service in one
of my attempts to circumvent the crash. I've removed this and am
running with tracing enabled again, hopefully to catch the culprit!
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend flushing
[11/Sep/2009:10:01:39 -0700] - ldbm backend done flushing
*[11/Sep/2009:10:01:39 -0700] - slapd shutting down - signaling
operation threads *
*[11/Sep/2009:10:01:40 -0700] - slapd shutting down - waiting for 30
threads to terminate *
*[11/Sep/2009:10:01:40 -0700] - op_thread received shutdown signal
[11/Sep/2009:10:01:40 -0700] - slapd shutting down - waiting for 29
threads to terminate *
[...]
12:37 p.m.
On 9/11/2009 12:43 PM, Noriko Hosoi wrote:
On 09/10/2009 07:46 PM, Kevin Bowling wrote:
> Hi,
>
> I have been running FDS/389 on a F11 xen DomU for several months. I
> use it as the backend for UNIX username/passwords and also for
> redMine (a Ruby on Rails bug tracker) for http://www.gnucapplus.org/.
>
> This VM would regularly lock up every week or so when 389 was still
> called FDS. I've since upgraded to 389 by issuing 'yum upgrade' as
> well as running the 'setup-...-.pl -u' script and now it barely goes
> a day before crashing. When ldap crashes, the whole box basically
> becomes unresponsive.
>
> I left the Xen hardware console open to see what was up and the only
> thing I could conclude was that 389 was crashing (if I issued a
> service start it came back to life). Doing anything like a top or ls
> will completely kill the box. Likewise, the logs show nothing at or
> before the time of crash. I suspected too few file descriptors but
> changing that to a very high number had no impact.
>
> I was about to do a rip and replace with OpenLDAP which I use very
> sucesessfully for our corporate systems but figured I ought to see if
> anyone here can help or if I can submit any kind of meaningful bug
> report first. I assume I will need to run 389's slapd without
> daemonizing it and hope it spits something useful out to stderr. Any
> advice here would be greatly appreciated, as would any success
> stories of using 389 on F11.
Hello Kevin,
You specified the platform "F11 xen DomU". Did you have a chance to
run the 389 server on any other platforms? I'm wondering if the crash
is observed only on the specific platform or not. Is the server
running on the 64-bit machine or 32-bit?
If you start the server with "-d 1" option, the server will run as the
trace mode. (E.g., /usr/lib[64]/dirsrv/slapd-YOURID/start-slapd -d 1)
I'm afraid it might be a memory leak. When you restart the 389
server, could you check the size of ns-slapd some time like every hour
and see if the server size keeps growing or stops? Also, the server
quits if it fails to write to the errors log. If it happens, it's
logged in the system log. Does the messages file on the system
happen to have some logs related to the 389 server?
Thanks,
--noriko
>
> I'm not subscribed to the list so please CC.
>
> Regards,
>
> Kevin Bowing
It was stable for 17 days while running with debug enabled to console.
I upgraded to the F11 2.6.30 kernel rebase, and now I get some debugging
info on the console. I'm taking a wild guess that it is timing
related. Where should I place a bug report?
Regards,
Kevin
[root@buildbox-a2 ~]# xm console 8
INFO: task kjournald:61 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff88003e932000 0 61 2
ffff88003e919d40 0000000000000246 ffffffff8100e45c 0000000000000000
000000001cee5db8 ffff88003e919d20 ffffffff8100ee82 0000000000000202
ffff88003e9c83a8 000000000000e2e8 ffff88003e9c83a8 0000000000012d00
Call Trace:
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff8100ee6f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff811b8b33>] journal_commit_transaction+0x13d/0xe42
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff810632bc>] ? try_to_del_timer_sync+0x69/0x87
[<ffffffff811bcdf7>] kjournald+0xfd/0x253
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff811bccfa>] ? kjournald+0x0/0x253
[<ffffffff81070709>] kthread+0x6d/0xae
[<ffffffff8101313a>] child_rip+0xa/0x20
[<ffffffff81012afd>] ? restore_args+0x0/0x30
[<ffffffff81013130>] ? child_rip+0x0/0x20
INFO: task ns-slapd:1034 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ns-slapd D ffffc20000000000 0 1034 1
ffff88003dd87908 0000000000000282 ffff88003dd87868 ffffffff8100ed0d
ffff88003dd86000 00000000e59205a0 ffff88003dd87888 ffffffff8107957a
ffff88003d4fe0e8 000000000000e2e8 ffff88003d4fe0e8 0000000000012d00
Call Trace:
[<ffffffff8100ed0d>] ? xen_clocksource_get_cycles+0x1c/0x32
[<ffffffff8107957a>] ? clocksource_read+0x22/0x38
[<ffffffff81074986>] ? ktime_get_ts+0x61/0x7d
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff81496c62>] io_schedule+0x44/0x6c
[<ffffffff8113b141>] sync_buffer+0x53/0x6b
[<ffffffff81497294>] __wait_on_bit_lock+0x55/0xb2
[<ffffffff810d2d1f>] ? find_get_page+0x64/0xa3
[<ffffffff8149736e>] out_of_line_wait_on_bit_lock+0x7d/0x9c
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81070c5a>] ? wake_bit_function+0x0/0x5a
[<ffffffff8113b380>] __lock_buffer+0x3d/0x53
[<ffffffff811b6eda>] lock_buffer+0x49/0x64
[<ffffffff811b7a15>] do_get_write_access+0x82/0x3f3
[<ffffffff811bbdb3>] ? journal_add_journal_head+0xce/0x162
[<ffffffff811b7dc0>] journal_get_write_access+0x3a/0x65
[<ffffffff8118c209>] __ext3_journal_get_write_access+0x34/0x74
[<ffffffff8117e464>] ext3_reserve_inode_write+0x50/0xaa
[<ffffffff8117e50d>] ext3_mark_inode_dirty+0x4f/0x80
[<ffffffff8117e6b8>] ext3_dirty_inode+0x79/0xa7
[<ffffffff81135095>] __mark_inode_dirty+0x45/0x190
[<ffffffff81129603>] file_update_time+0xc0/0x113
[<ffffffff810eb167>] do_wp_page+0x610/0x658
[<ffffffff81INFO: task kjournald:61 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff88003e932000 0 61 2
ffff88003e919d40 0000000000000246 ffffffff8100e45c 0000000000000000
000000001cee5db8 ffff88003e919d20 ffffffff8100ee82 0000000000000202
ffff88003e9c83a8 000000000000e2e8 ffff88003e9c83a8 0000000000012d00
Call Trace:
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff8100ee6f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff811b8b33>] journal_commit_transaction+0x13d/0xe42
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff810632bc>] ? try_to_del_timer_sync+0x69/0x87
[<ffffffff811bcdf7>] kjournald+0xfd/0x253
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff811bccfa>] ? kjournald+0x0/0x253
[<ffffffff81070709>] kthread+0x6d/0xae
[<ffffffff8101313a>] child_rip+0xa/0x20
[<ffffffff81012afd>] ? restore_args+0x0/0x30
[<ffffffff81013130>] ? child_rip+0x0/0x20
INFO: task ns-slapd:1034 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ns-slapd D ffffc20000000000 0 1034 1
ffff88003dd87908 0000000000000282 ffff88003dd87868 ffffffff8100ed0d
ffff88003dd86000 00000000e59205a0 ffff88003dd87888 ffffffff8107957a
ffff88003d4fe0e8 000000000000e2e8 ffff88003d4fe0e8 0000000000012d00
Call Trace:
[<ffffffff8100ed0d>] ? xen_clocksource_get_cycles+0x1c/0x32
[<ffffffff8107957a>] ? clocksource_read+0x22/0x38
[<ffffffff81074986>] ? ktime_get_ts+0x61/0x7d
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff81496c62>] io_schedule+0x44/0x6c
[<ffffffff8113b141>] sync_buffer+0x53/0x6b
[<ffffffff81497294>] __wait_on_bit_lock+0x55/0xb2
[<ffffffff810d2d1f>] ? find_get_page+0x64/0xa3
[<ffffffff8149736e>] out_of_line_wait_on_bit_lock+0x7d/0x9c
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81070c5a>] ? wake_bit_function+0x0/0x5a
[<ffffffff8113b380>] __lock_buffer+0x3d/0x53
[<ffffffff811b6eda>] lock_buffer+0x49/0x64
[<ffffffff811b7a15>] do_get_write_access+0x82/0x3f3
[<ffffffff811bbdb3>] ? journal_add_journal_head+0xce/0x162
[<ffffffff811b7dc0>] journal_get_write_access+0x3a/0x65
[<ffffffff8118c209>] __ext3_journal_get_write_access+0x34/0x74
[<ffffffff8117e464>] ext3_reserve_inode_write+0x50/0xaa
[<ffffffff8117e50d>] ext3_mark_inode_dirty+0x4f/0x80
[<ffffffff8117e6b8>] ext3_dirty_inode+0x79/0xa7
[<ffffffff81135095>] __mark_inode_dirty+0x45/0x190
[<ffffffff81129603>] file_update_time+0xc0/0x113
[<ffffffff810eb167>] do_wp_page+0x610/0x658
[<ffffffff8100bc21>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[<ffffffff810eccd9>] handle_mm_fault+0x6a2/0x72e
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff8149be99>] do_page_fault+0x226/0x24f
[<ffffffff81499965>] page_fault+0x25/0x30
INFO: task ns-slapd:1040 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ns-slapd D ffff88003e932024 0 1040 1
ffff88003bc119f8 0000000000000282 ffffffff8100e45c ffffc20000025410
00000000f1efb74c ffff88003bc119d8 ffffffff8100ee82 0000000000000004
ffff88003bc0b248 000000000000e2e8 ffff88003bc0b248 0000000000012d00
Call Trace:
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff8100ee6f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff811b841d>] start_this_handle+0x2d4/0x373
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff811b865d>] journal_start+0xb7/0x106
[<ffffffff81187903>] ext3_journal_start_sb+0x62/0x78
[<ffffffff8117d60b>] ext3_journal_start+0x28/0x3e
[<ffffffff8117e67d>] ext3_dirty_inode+0x3e
1:38 p.m.
On 09/30/2009 10:37 AM, Kevin Bowling wrote:
> [...]
It was stable for 17 days while running with debug enabled to
console. I upgraded to the F11 2.6.30 kernel rebase, and now I get
some debugging info on the console. I'm taking a wild guess that it
is timing related. Where should I place a bug report?
Hi,
Does this mean the 389 server did not get crashed, but blocked possibly
by Xen OS?
INFO: task ns-slapd:1034 blocked for more than 120 seconds.
It looks some other daemon was also affected.
INFO: task kjournald:61 blocked for more than 120 seconds.
If it's a Xen issue, you could go to this page,
https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
then choose "xen" for Component.
Thanks,
--noriko
Regards,
Kevin
[root@buildbox-a2 ~]# xm console 8
INFO: task kjournald:61 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff88003e932000 0 61 2
ffff88003e919d40 0000000000000246 ffffffff8100e45c 0000000000000000
000000001cee5db8 ffff88003e919d20 ffffffff8100ee82 0000000000000202
ffff88003e9c83a8 000000000000e2e8 ffff88003e9c83a8 0000000000012d00
Call Trace:
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff8100ee6f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff811b8b33>] journal_commit_transaction+0x13d/0xe42
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff810632bc>] ? try_to_del_timer_sync+0x69/0x87
[<ffffffff811bcdf7>] kjournald+0xfd/0x253
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff811bccfa>] ? kjournald+0x0/0x253
[<ffffffff81070709>] kthread+0x6d/0xae
[<ffffffff8101313a>] child_rip+0xa/0x20
[<ffffffff81012afd>] ? restore_args+0x0/0x30
[<ffffffff81013130>] ? child_rip+0x0/0x20
INFO: task ns-slapd:1034 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ns-slapd D ffffc20000000000 0 1034 1
ffff88003dd87908 0000000000000282 ffff88003dd87868 ffffffff8100ed0d
ffff88003dd86000 00000000e59205a0 ffff88003dd87888 ffffffff8107957a
ffff88003d4fe0e8 000000000000e2e8 ffff88003d4fe0e8 0000000000012d00
Call Trace:
[<ffffffff8100ed0d>] ? xen_clocksource_get_cycles+0x1c/0x32
[<ffffffff8107957a>] ? clocksource_read+0x22/0x38
[<ffffffff81074986>] ? ktime_get_ts+0x61/0x7d
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff81496c62>] io_schedule+0x44/0x6c
[<ffffffff8113b141>] sync_buffer+0x53/0x6b
[<ffffffff81497294>] __wait_on_bit_lock+0x55/0xb2
[<ffffffff810d2d1f>] ? find_get_page+0x64/0xa3
[<ffffffff8149736e>] out_of_line_wait_on_bit_lock+0x7d/0x9c
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81070c5a>] ? wake_bit_function+0x0/0x5a
[<ffffffff8113b380>] __lock_buffer+0x3d/0x53
[<ffffffff811b6eda>] lock_buffer+0x49/0x64
[<ffffffff811b7a15>] do_get_write_access+0x82/0x3f3
[<ffffffff811bbdb3>] ? journal_add_journal_head+0xce/0x162
[<ffffffff811b7dc0>] journal_get_write_access+0x3a/0x65
[<ffffffff8118c209>] __ext3_journal_get_write_access+0x34/0x74
[<ffffffff8117e464>] ext3_reserve_inode_write+0x50/0xaa
[<ffffffff8117e50d>] ext3_mark_inode_dirty+0x4f/0x80
[<ffffffff8117e6b8>] ext3_dirty_inode+0x79/0xa7
[<ffffffff81135095>] __mark_inode_dirty+0x45/0x190
[<ffffffff81129603>] file_update_time+0xc0/0x113
[<ffffffff810eb167>] do_wp_page+0x610/0x658
[<ffffffff81INFO: task kjournald:61 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff88003e932000 0 61 2
ffff88003e919d40 0000000000000246 ffffffff8100e45c 0000000000000000
000000001cee5db8 ffff88003e919d20 ffffffff8100ee82 0000000000000202
ffff88003e9c83a8 000000000000e2e8 ffff88003e9c83a8 0000000000012d00
Call Trace:
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff8100ee6f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff811b8b33>] journal_commit_transaction+0x13d/0xe42
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff810632bc>] ? try_to_del_timer_sync+0x69/0x87
[<ffffffff811bcdf7>] kjournald+0xfd/0x253
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff811bccfa>] ? kjournald+0x0/0x253
[<ffffffff81070709>] kthread+0x6d/0xae
[<ffffffff8101313a>] child_rip+0xa/0x20
[<ffffffff81012afd>] ? restore_args+0x0/0x30
[<ffffffff81013130>] ? child_rip+0x0/0x20
INFO: task ns-slapd:1034 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ns-slapd D ffffc20000000000 0 1034 1
ffff88003dd87908 0000000000000282 ffff88003dd87868 ffffffff8100ed0d
ffff88003dd86000 00000000e59205a0 ffff88003dd87888 ffffffff8107957a
ffff88003d4fe0e8 000000000000e2e8 ffff88003d4fe0e8 0000000000012d00
Call Trace:
[<ffffffff8100ed0d>] ? xen_clocksource_get_cycles+0x1c/0x32
[<ffffffff8107957a>] ? clocksource_read+0x22/0x38
[<ffffffff81074986>] ? ktime_get_ts+0x61/0x7d
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff81496c62>] io_schedule+0x44/0x6c
[<ffffffff8113b141>] sync_buffer+0x53/0x6b
[<ffffffff81497294>] __wait_on_bit_lock+0x55/0xb2
[<ffffffff810d2d1f>] ? find_get_page+0x64/0xa3
[<ffffffff8149736e>] out_of_line_wait_on_bit_lock+0x7d/0x9c
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81070c5a>] ? wake_bit_function+0x0/0x5a
[<ffffffff8113b380>] __lock_buffer+0x3d/0x53
[<ffffffff811b6eda>] lock_buffer+0x49/0x64
[<ffffffff811b7a15>] do_get_write_access+0x82/0x3f3
[<ffffffff811bbdb3>] ? journal_add_journal_head+0xce/0x162
[<ffffffff811b7dc0>] journal_get_write_access+0x3a/0x65
[<ffffffff8118c209>] __ext3_journal_get_write_access+0x34/0x74
[<ffffffff8117e464>] ext3_reserve_inode_write+0x50/0xaa
[<ffffffff8117e50d>] ext3_mark_inode_dirty+0x4f/0x80
[<ffffffff8117e6b8>] ext3_dirty_inode+0x79/0xa7
[<ffffffff81135095>] __mark_inode_dirty+0x45/0x190
[<ffffffff81129603>] file_update_time+0xc0/0x113
[<ffffffff810eb167>] do_wp_page+0x610/0x658
[<ffffffff8100bc21>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[<ffffffff810eccd9>] handle_mm_fault+0x6a2/0x72e
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff8149be99>] do_page_fault+0x226/0x24f
[<ffffffff81499965>] page_fault+0x25/0x30
INFO: task ns-slapd:1040 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ns-slapd D ffff88003e932024 0 1040 1
ffff88003bc119f8 0000000000000282 ffffffff8100e45c ffffc20000025410
00000000f1efb74c ffff88003bc119d8 ffffffff8100ee82 0000000000000004
ffff88003bc0b248 000000000000e2e8 ffff88003bc0b248 0000000000012d00
Call Trace:
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff8100ee6f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff811b841d>] start_this_handle+0x2d4/0x373
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff811b865d>] journal_start+0xb7/0x106
[<ffffffff81187903>] ext3_journal_start_sb+0x62/0x78
[<ffffffff8117d60b>] ext3_journal_start+0x28/0x3e
[<ffffffff8117e67d>] ext3_dirty_inode+0x3e
9:10 a.m.
Kevin Bowling wrote:
On 9/11/2009 12:43 PM, Noriko Hosoi wrote:
> On 09/10/2009 07:46 PM, Kevin Bowling wrote:
>> Hi,
>>
>> I have been running FDS/389 on a F11 xen DomU for several months. I
>> use it as the backend for UNIX username/passwords and also for
>> redMine (a Ruby on Rails bug tracker) for http://www.gnucapplus.org/.
>>
>> This VM would regularly lock up every week or so when 389 was still
>> called FDS. I've since upgraded to 389 by issuing 'yum upgrade' as
>> well as running the 'setup-...-.pl -u' script and now it barely goes
>> a day before crashing. When ldap crashes, the whole box basically
>> becomes unresponsive.
>>
>> I left the Xen hardware console open to see what was up and the only
>> thing I could conclude was that 389 was crashing (if I issued a
>> service start it came back to life). Doing anything like a top or
>> ls will completely kill the box. Likewise, the logs show nothing at
>> or before the time of crash. I suspected too few file descriptors
>> but changing that to a very high number had no impact.
>>
>> I was about to do a rip and replace with OpenLDAP which I use very
>> sucesessfully for our corporate systems but figured I ought to see
>> if anyone here can help or if I can submit any kind of meaningful
>> bug report first. I assume I will need to run 389's slapd without
>> daemonizing it and hope it spits something useful out to stderr.
>> Any advice here would be greatly appreciated, as would any success
>> stories of using 389 on F11.
> Hello Kevin,
>
> You specified the platform "F11 xen DomU". Did you have a chance to
> run the 389 server on any other platforms? I'm wondering if the
> crash is observed only on the specific platform or not. Is the
> server running on the 64-bit machine or 32-bit?
>
> If you start the server with "-d 1" option, the server will run as
> the trace mode. (E.g., /usr/lib[64]/dirsrv/slapd-YOURID/start-slapd
> -d 1)
>
> I'm afraid it might be a memory leak. When you restart the 389
> server, could you check the size of ns-slapd some time like every
> hour and see if the server size keeps growing or stops? Also, the
> server quits if it fails to write to the errors log. If it happens,
> it's logged in the system log. Does the messages file on the system
> happen to have some logs related to the 389 server?
>
> Thanks,
> --noriko
>>
>> I'm not subscribed to the list so please CC.
>>
>> Regards,
>>
>> Kevin Bowing
It was stable for 17 days while running with debug enabled to
console. I upgraded to the F11 2.6.30 kernel rebase, and now I get
some debugging info on the console. I'm taking a wild guess that it
is timing related. Where should I place a bug report?
Is it related to this -
https://bugzilla.redhat.com/show_bug.cgi?id=521637
Regards,
Kevin
[root@buildbox-a2 ~]# xm console 8
INFO: task kjournald:61 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff88003e932000 0 61 2
ffff88003e919d40 0000000000000246 ffffffff8100e45c 0000000000000000
000000001cee5db8 ffff88003e919d20 ffffffff8100ee82 0000000000000202
ffff88003e9c83a8 000000000000e2e8 ffff88003e9c83a8 0000000000012d00
Call Trace:
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff8100ee6f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff811b8b33>] journal_commit_transaction+0x13d/0xe42
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff810632bc>] ? try_to_del_timer_sync+0x69/0x87
[<ffffffff811bcdf7>] kjournald+0xfd/0x253
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff811bccfa>] ? kjournald+0x0/0x253
[<ffffffff81070709>] kthread+0x6d/0xae
[<ffffffff8101313a>] child_rip+0xa/0x20
[<ffffffff81012afd>] ? restore_args+0x0/0x30
[<ffffffff81013130>] ? child_rip+0x0/0x20
INFO: task ns-slapd:1034 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ns-slapd D ffffc20000000000 0 1034 1
ffff88003dd87908 0000000000000282 ffff88003dd87868 ffffffff8100ed0d
ffff88003dd86000 00000000e59205a0 ffff88003dd87888 ffffffff8107957a
ffff88003d4fe0e8 000000000000e2e8 ffff88003d4fe0e8 0000000000012d00
Call Trace:
[<ffffffff8100ed0d>] ? xen_clocksource_get_cycles+0x1c/0x32
[<ffffffff8107957a>] ? clocksource_read+0x22/0x38
[<ffffffff81074986>] ? ktime_get_ts+0x61/0x7d
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff81496c62>] io_schedule+0x44/0x6c
[<ffffffff8113b141>] sync_buffer+0x53/0x6b
[<ffffffff81497294>] __wait_on_bit_lock+0x55/0xb2
[<ffffffff810d2d1f>] ? find_get_page+0x64/0xa3
[<ffffffff8149736e>] out_of_line_wait_on_bit_lock+0x7d/0x9c
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81070c5a>] ? wake_bit_function+0x0/0x5a
[<ffffffff8113b380>] __lock_buffer+0x3d/0x53
[<ffffffff811b6eda>] lock_buffer+0x49/0x64
[<ffffffff811b7a15>] do_get_write_access+0x82/0x3f3
[<ffffffff811bbdb3>] ? journal_add_journal_head+0xce/0x162
[<ffffffff811b7dc0>] journal_get_write_access+0x3a/0x65
[<ffffffff8118c209>] __ext3_journal_get_write_access+0x34/0x74
[<ffffffff8117e464>] ext3_reserve_inode_write+0x50/0xaa
[<ffffffff8117e50d>] ext3_mark_inode_dirty+0x4f/0x80
[<ffffffff8117e6b8>] ext3_dirty_inode+0x79/0xa7
[<ffffffff81135095>] __mark_inode_dirty+0x45/0x190
[<ffffffff81129603>] file_update_time+0xc0/0x113
[<ffffffff810eb167>] do_wp_page+0x610/0x658
[<ffffffff81INFO: task kjournald:61 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff88003e932000 0 61 2
ffff88003e919d40 0000000000000246 ffffffff8100e45c 0000000000000000
000000001cee5db8 ffff88003e919d20 ffffffff8100ee82 0000000000000202
ffff88003e9c83a8 000000000000e2e8 ffff88003e9c83a8 0000000000012d00
Call Trace:
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff8100ee6f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff811b8b33>] journal_commit_transaction+0x13d/0xe42
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff810632bc>] ? try_to_del_timer_sync+0x69/0x87
[<ffffffff811bcdf7>] kjournald+0xfd/0x253
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff811bccfa>] ? kjournald+0x0/0x253
[<ffffffff81070709>] kthread+0x6d/0xae
[<ffffffff8101313a>] child_rip+0xa/0x20
[<ffffffff81012afd>] ? restore_args+0x0/0x30
[<ffffffff81013130>] ? child_rip+0x0/0x20
INFO: task ns-slapd:1034 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ns-slapd D ffffc20000000000 0 1034 1
ffff88003dd87908 0000000000000282 ffff88003dd87868 ffffffff8100ed0d
ffff88003dd86000 00000000e59205a0 ffff88003dd87888 ffffffff8107957a
ffff88003d4fe0e8 000000000000e2e8 ffff88003d4fe0e8 0000000000012d00
Call Trace:
[<ffffffff8100ed0d>] ? xen_clocksource_get_cycles+0x1c/0x32
[<ffffffff8107957a>] ? clocksource_read+0x22/0x38
[<ffffffff81074986>] ? ktime_get_ts+0x61/0x7d
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff81496c62>] io_schedule+0x44/0x6c
[<ffffffff8113b141>] sync_buffer+0x53/0x6b
[<ffffffff81497294>] __wait_on_bit_lock+0x55/0xb2
[<ffffffff810d2d1f>] ? find_get_page+0x64/0xa3
[<ffffffff8149736e>] out_of_line_wait_on_bit_lock+0x7d/0x9c
[<ffffffff8113b0ee>] ? sync_buffer+0x0/0x6b
[<ffffffff81070c5a>] ? wake_bit_function+0x0/0x5a
[<ffffffff8113b380>] __lock_buffer+0x3d/0x53
[<ffffffff811b6eda>] lock_buffer+0x49/0x64
[<ffffffff811b7a15>] do_get_write_access+0x82/0x3f3
[<ffffffff811bbdb3>] ? journal_add_journal_head+0xce/0x162
[<ffffffff811b7dc0>] journal_get_write_access+0x3a/0x65
[<ffffffff8118c209>] __ext3_journal_get_write_access+0x34/0x74
[<ffffffff8117e464>] ext3_reserve_inode_write+0x50/0xaa
[<ffffffff8117e50d>] ext3_mark_inode_dirty+0x4f/0x80
[<ffffffff8117e6b8>] ext3_dirty_inode+0x79/0xa7
[<ffffffff81135095>] __mark_inode_dirty+0x45/0x190
[<ffffffff81129603>] file_update_time+0xc0/0x113
[<ffffffff810eb167>] do_wp_page+0x610/0x658
[<ffffffff8100bc21>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[<ffffffff810eccd9>] handle_mm_fault+0x6a2/0x72e
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff8149be99>] do_page_fault+0x226/0x24f
[<ffffffff81499965>] page_fault+0x25/0x30
INFO: task ns-slapd:1040 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ns-slapd D ffff88003e932024 0 1040 1
ffff88003bc119f8 0000000000000282 ffffffff8100e45c ffffc20000025410
00000000f1efb74c ffff88003bc119d8 ffffffff8100ee82 0000000000000004
ffff88003bc0b248 000000000000e2e8 ffff88003bc0b248 0000000000012d00
Call Trace:
[<ffffffff8100e45c>] ? xen_force_evtchn_callback+0x20/0x36
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff8100ee6f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff814993de>] ? _spin_unlock_irqrestore+0x4e/0x64
[<ffffffff8100ee82>] ? check_events+0x12/0x20
[<ffffffff81496bf6>] schedule+0x21/0x49
[<ffffffff811b841d>] start_this_handle+0x2d4/0x373
[<ffffffff81070bfb>] ? autoremove_wake_function+0x0/0x5f
[<ffffffff811b865d>] journal_start+0xb7/0x106
[<ffffffff81187903>] ext3_journal_start_sb+0x62/0x78
[<ffffffff8117d60b>] ext3_journal_start+0x28/0x3e
[<ffffffff8117e67d>] ext3_dirty_inode+0x3e
--
389 users mailing list
389-users(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-directory-users
5319
days inactive
5339
days old
389-users@lists.fedoraproject.org
7 comments
3 participants
participants (3)
-
Kevin Bowling -
Noriko Hosoi -
Rich Megginson