Crashing
by Wendt, Trevor
Hello all,
Need some help with tuning and crash debugging. We're running Fedora-Directory/1.0.4 B2006.312.1539. The problem is on our "Dedicated Consumer" machine running on RHEL 5. We have over ~150,000 users authenticating against our FDS systems. System resources are not a problem (~.39 load, free memory, 92k swap)
For months, the system is solid without any issues then we seem to get a large spike in traffic and FDS crashes. I run Monit so the service is restarted automatically but I cannot figure out why the service keeps crashing.
FDS was setup and tuned based off: http://directory.fedoraproject.org/wiki/Performance_Tuning#Linux
I have reviewed http://directory.fedoraproject.org/wiki/FAQ#Debugging_Crashes as well, but some of that is over my head. I have turned buffering off and increased the logging level in the LDAP config.
Here is our "monitor" script output:
version: 1
dn: cn=monitor
objectClass: top
objectClass: extensibleObject
cn: monitor
version: Fedora-Directory/1.0.4 B2006.312.1539
threads: 30
currentconnections: 19
totalconnections: 11918
dtablesize: 8192
readwaiters: 0
opsinitiated: 43703
opscompleted: 43702
entriessent: 16086
bytessent: 2911011
currenttime: 20110805164243Z
starttime: 20110805114053Z
nbackends: 2
Here is our "Access Log Analyzer" summary for a 24 hour period:
---------------------------------------------------------------
Access Log Analyzer 6.0
Filename Total Lines Lines processed
---------------------------------------------------------------
/opt/fedora-ds/slapd/logs/access 298225 298231
----------- Access Log Output ------------
Restarts: 6
Total Connections: 39720
Peak Concurrent Connections: 84
Total Operations: 95471
Total Results: 95393
Overall Performance: 99.9%
Searches: 48215
Modifications: 167
Adds: 551
Deletes: 2
Mod RDNs: 0
6.x Stats
Persistent Searches: 0
Internal Operations: 0
Entry Operations: 0
Extended Operations: 845
Abandoned Requests: 0
Smart Referrals Received: 0
VLV Operations: 0
VLV Unindexed Searches: 0
SORT Operations: 0
SSL Connections: 0
Entire Search Base Queries: 0
Unindexed Searches: 6
FDs Taken: 39720
FDs Returned: 39657
Highest FD Taken: 93
Broken Pipes: 0
Connections Reset By Peer: 0
Resource Unavailable: 10872
- 10872 (T1) Idle Timeout Exceeded
Binds: 45691
Unbinds: 27987
LDAP v2 Binds: 15694
LDAP v3 Binds: 29997
SSL Client Binds: 0
Failed SSL Client Binds: 0
SASL Binds: 0
Directory Manager Binds: 0
Anonymous Binds: 16346
Other Binds: 29345
---------------------------------------------------------------
In FDS console:
-- Configuration > Performance tab: Size Limit: 2000, Time Limit: 3600, Idle Timeout: 60, Max file descriptors: 8192.
-- Configuration > Data > Database Link Settings > Connection Management: Max TCP Connections: 10, Bind timeout: 20, Max binds per connection: 20, Timeout before abandon: 10, Max LDAP Connections: 20, Max bind retries: 3, Max operations per connection: 5, connection life: 60.
We have talked about moving to the latest 389 Directory packages and I have a migration process tested out so it's a matter of getting the OK and time but I doubt the upgrade will solve our crashing problem. It seems to me we are hitting some limits that just haven't been accounted for yet and that is where I need help.
Any suggestions on how to proceed with stopping these crashes is welcomed! Thanks for reading.
Trevor
________________________________
This electronic message transmission contains information from Black Hills Corporation, its affiliate or subsidiary, which may be confidential or privileged. The information is intended to be for the use of the individual or entity named above. If you are not the intended recipient, be aware the disclosure, copying, distribution or use of the contents of this information is prohibited. If you received this electronic transmission in error, please reply to sender immediately; then delete this message without copying it or further reading.
12 years, 7 months
LDAPCon 2011 Call for Papers
by Peter Gietz
With the usual apologies.
The 3rd Edition of the International Conference on LDAP (LDAPCon
2011[1]) will be held on October, 10-11, 2011 in Heidelberg, Germany.
A Call For Papers[2] has been raised and the Program Committee asks you
to submit abstracts by July 8th.
The International Conference on LDAP is a technical forum for IT
professionals interested in LDAP and related topics like directory
servers, directory management applications, directory integration,
identity and access management, and meta directories.
It focuses on implementation and integration of LDAP servers and
LDAP-enabled client applications. The event will bring together vendors,
developers, active and prospective LDAP practitioners to share their
experiences about deployment strategies, service operations,
interoperability, discuss LDAP usage in new projects and learn about
upcoming trends and developments.
The 1st LDAPCon[3] was held in September 2007 in Germany, the 2nd
LDAPCon[4] was held in September 2009 in Portland, Oregon, USA
(Some pictures from LDAPCon 2007 [5] and a nice summary of LDAPCon 2009 [6])
So if you're involved with LDAP in interesting projects and you want to
share your experiences, please check the Call For Papers and submit a
proposal.
Best,
Peter
[1]: http://www.ldapcon.org
[2]: http://www.daasi.de/ldapcon2011/index.php?site=cfp
[3]: http://www.guug.de/veranstaltungen/ldapcon2007/index.html
[4]: http://www.symas.com/ldapcon2009
[5]: http://www.flickr.com/photos/ludovic_p/sets/72157601937159198/detail/
[6]: http://blogs.sun.com/Ludo/entry/ldapcon_2009_summary
--
_______________________________________________________________________
Peter Gietz (CEO)
DAASI International GmbH phone: +49 7071 407109-0
Europaplatz 3 Fax: +49 7071 407109-9
D-72074 Tübingen mail: peter.gietz(a)daasi.de
Germany Web: www.daasi.de
DAASI International GmbH, Tübingen
Geschäftsführer Peter Gietz, Amtsgericht Stuttgart HRB 382175
Directory Applications for Advanced Security and Information Management
_______________________________________________________________________
12 years, 7 months
xinetd app LDAP errors when LDAP server is down for non-LDAP user
by James Smallacombe
We're having a pretty severe issue of a server/client app that is running out of
xinetd generating nss_ldap errors when the primary LDAP server is down. The thing
is, the user that this application (nagios nrpe) runs as exists in every host's
/etc/passwd (and group) file and NOT in the Directory Server, just for this
reason. I am wondering if this is a pam issue, but I admit I do not know to what
extent that service users consult pam. Here is the error:
Aug 2 12:03:18 host01 xinetd[32012]: nss_ldap: failed to bind to LDAP
server ldap://ldap_1.domain/: Can't contact LDAP server
Aug 2 12:03:18 host01 xinetd[32012]: nss_ldap: reconnected to LDAP server
ldap://ldap_2.domain/
Aug 2 12:03:18 host01 nrpe[32012]: Error: Could not complete SSL handshake.5
Again /etc/xinetd.d/nrpe is configured to run this client as a user that exists in
local auth, not LDAP. Why would it need to contact the LDAP server at all? We do
not use LDAP for name resolution, that is all done in DNS and /etc/resolv.conf.
We ONLY use it for user authentication.
We used authconfig to set this up on the clients. I am wondering if the PAM stack
in /etc/pam.d/system-auth, which gets modified by authconfig for LDAP has anything
to do with it. The one thing that caught my eye was this:
account required pam_unix.so broken_shadow
account sufficient pam_succeed_if.so uid < 500 quiet
account [default=bad success=ok user_unknown=ignore] pam_ldap.so
account required pam_permit.so
The UID of the daemon user is ABOVE 500. Would changing it to one below 500 fix
the problem?
Thanks in advance!
12 years, 7 months
Delete object on Consumer
by Jim Tyrrell
Hi,
We have a setup with multiple masters which are replicating down to 389
Directory Server consumers via 2 hubs, but have a consistency issue.
It appears a few objects were deleted and re-added to the masters but
the object was not deleted from the 389 consumers. We now have 1
object on the masters and 2 objects on the consumers which causes
problems for the mail servers. If we delete the object from the master
we are still left with one object on the slaves. The slaves currently
have a few duplicate objects like this:
dn: cn=mx::10, cn=somedomain.co.uk, ou=dns, o=acmesystems.com
cn: mx::10
mailtransport: nexthop:[mailserver.ourdomain.com]
dnspreference: 10
dnstype: MX
dn:
nsuniqueid=7edfa581-1dd211b2-8014f995-55bd0000+cn=mx::10,cn=somedomain.co.uk,
ou=dns,o=acmesystems.com
cn: mx::10
mailtransport: nexthop:[mailserver.ourdomain.com]
dnspreference: 10
dnstype: MX
The object showing nsuniqueid is the valid one that exists on the
master. Is there a way to remove the 2nd object from the consumer
without re-initialising?
I have seen this before on a single consumer so we re-initialised it,
but its a much bigger problem to re-initialise all of the consumers. It
would be ideal if there is a way to manually delete an object direct on
a consumer?
Thanks.
Jim.
12 years, 8 months
Change name of server, admin-server no longer works
by Techie
Hello,
We were required to change the hostname of our LDAP server running
389-DS. Since that time the LDAP server runs fine but the admin server
does not authenticate login any longer, meaning i cannot log into the
admin server. What do I need to do to fix the admin server and change
all references from the old host name to the new host name.
Thanks
Jimmy
12 years, 8 months