We had to update our server from CentOS 6.7 to CentOS 6.8 due to security compliance. When doing so however, it caused 389 to be unstable for TLS/SSL port 636. It would be up for a minute or two, then fail with the following error when a server/script tried to connect. Non-TLS/SSL port 389 would work fine without any issues/errors. Before we patched, it would work without issues. Connection to port shows no issue with certificate.
[26/Jan/2017:01:02:39 -0500] conn=97 fd=64 slot=64 SSL connection from X.X.X.X to X.X.X.X [26/Jan/2017:01:02:39 -0500] conn=97 op=-1 fd=64 closed - Unspecified failure while processing SSL Client Key Exchange handshake.
From the client:
TLS: loaded CA certificate file /etc/pki/tls/certs/bundle.crt. TLS: certificate [CN=XXXXXX.com,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated] is valid TLS: error: tlsm_PR_Recv returned -1 - error 104:Connection reset by peer TLS: error: connect - force handshake failure: errno 104 - moznss error -5961 TLS: can't connect: TLS error -5961:TCP connection reset by peer. ldap_err2string ldap_start_tls: Connect error (-11) additional info: TLS error -5961:TCP connection reset by peer ldap_sasl_bind
Normal Connection:
[26/Jan/2017:05:29:35 -0500] conn=904 fd=65 slot=65 SSL connection from X.X.X.X to X.X.X.X [26/Jan/2017:05:29:35 -0500] conn=904 TLS1.2 256-bit AES
Current Version of 389:
389-adminutil-1.1.19-1.el6.x86_64 389-ds-base-libs-1.2.11.15-74.el6.x86_64 389-ds-console-doc-1.2.6-1.el6.noarch 389-admin-1.1.35-1.el6.x86_64 389-ds-console-1.2.6-1.el6.noarch 389-dsgw-1.1.11-1.el6.x86_64 389-ds-base-1.2.11.15-74.el6.x86_64 389-console-1.1.7-1.el6.noarch
NSS:
nss-3.21.0-8.el6.x86_64 nss-softokn-3.14.3-23.el6_7.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.i686 nss-softokn-freebl-3.14.3-23.el6_7.x86_64 nss-sysinit-3.21.0-8.el6.x86_64 nss-tools-3.21.0-8.el6.x86_64 nss-util-3.21.0-2.el6.x86_64
Port is open:
tcp 0 0 :::636 :::* LISTEN
Approx Strace:
getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}], 4, 250) = 1 ([{fd=8, revents=POLLIN}]) accept(8, {sa_family=AF_INET6, sin6_port=htons(52890), inet_pton(AF_INET6, "::ffff:X.X.X.X", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 36 fcntl(36, F_GETFL) = 0x2 (flags O_RDWR) fcntl(36, F_SETFL, O_RDWR|O_NONBLOCK) = 0 fcntl(36, F_DUPFD, 64) = 64 close(36) = 0 setsockopt(64, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0 setsockopt(64, SOL_TCP, TCP_NODELAY, [0], 4) = 0 getsockname(64, {sa_family=AF_INET6, sin6_port=htons(636), inet_pton(AF_INET6, "::ffff:X.X.X.X", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}, {fd=64, events=POLLIN}], 5, 250) = 1 ([{fd=64, revents=POLLIN}]) futex(0x16ee83c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x16ee838, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}], 4, 250) = 1 ([{fd=40, revents=POLLIN}]) read(40, "\0", 200) = 1 close(64) = 0 getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}], 4, 250) = 0 (Timeout)
This may happen if some LDAP clients are not up to date (what are they? Java clients?) A tcpdump could show the details of the ciphers negociated in the TLS or SSL handshake for the failing LDAP clients. Possible related article: https://access.redhat.com/solutions/2332231 Thanks, M.
On Thu, Jan 26, 2017 at 9:59 AM, John McKee johnnyboy24@fedoraproject.org wrote:
We had to update our server from CentOS 6.7 to CentOS 6.8 due to security compliance. When doing so however, it caused 389 to be unstable for TLS/SSL port 636. It would be up for a minute or two, then fail with the following error when a server/script tried to connect. Non-TLS/SSL port 389 would work fine without any issues/errors. Before we patched, it would work without issues. Connection to port shows no issue with certificate.
[26/Jan/2017:01:02:39 -0500] conn=97 fd=64 slot=64 SSL connection from X.X.X.X to X.X.X.X [26/Jan/2017:01:02:39 -0500] conn=97 op=-1 fd=64 closed - Unspecified failure while processing SSL Client Key Exchange handshake.
From the client:
TLS: loaded CA certificate file /etc/pki/tls/certs/bundle.crt. TLS: certificate [CN=XXXXXX.com,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated] is valid TLS: error: tlsm_PR_Recv returned -1 - error 104:Connection reset by peer TLS: error: connect - force handshake failure: errno 104 - moznss error -5961 TLS: can't connect: TLS error -5961:TCP connection reset by peer. ldap_err2string ldap_start_tls: Connect error (-11) additional info: TLS error -5961:TCP connection reset by peer ldap_sasl_bind
Normal Connection:
[26/Jan/2017:05:29:35 -0500] conn=904 fd=65 slot=65 SSL connection from X.X.X.X to X.X.X.X [26/Jan/2017:05:29:35 -0500] conn=904 TLS1.2 256-bit AES
Current Version of 389:
389-adminutil-1.1.19-1.el6.x86_64 389-ds-base-libs-1.2.11.15-74.el6.x86_64 389-ds-console-doc-1.2.6-1.el6.noarch 389-admin-1.1.35-1.el6.x86_64 389-ds-console-1.2.6-1.el6.noarch 389-dsgw-1.1.11-1.el6.x86_64 389-ds-base-1.2.11.15-74.el6.x86_64 389-console-1.1.7-1.el6.noarch
NSS:
nss-3.21.0-8.el6.x86_64 nss-softokn-3.14.3-23.el6_7.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.i686 nss-softokn-freebl-3.14.3-23.el6_7.x86_64 nss-sysinit-3.21.0-8.el6.x86_64 nss-tools-3.21.0-8.el6.x86_64 nss-util-3.21.0-2.el6.x86_64
Port is open:
tcp 0 0 :::636 :::* LISTEN
Approx Strace:
getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}], 4, 250) = 1 ([{fd=8, revents=POLLIN}]) accept(8, {sa_family=AF_INET6, sin6_port=htons(52890), inet_pton(AF_INET6, "::ffff:X.X.X.X", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 36 fcntl(36, F_GETFL) = 0x2 (flags O_RDWR) fcntl(36, F_SETFL, O_RDWR|O_NONBLOCK) = 0 fcntl(36, F_DUPFD, 64) = 64 close(36) = 0 setsockopt(64, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0 setsockopt(64, SOL_TCP, TCP_NODELAY, [0], 4) = 0 getsockname(64, {sa_family=AF_INET6, sin6_port=htons(636), inet_pton(AF_INET6, "::ffff:X.X.X.X", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}, {fd=64, events=POLLIN}], 5, 250) = 1 ([{fd=64, revents=POLLIN}]) futex(0x16ee83c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x16ee838, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}], 4, 250) = 1 ([{fd=40, revents=POLLIN}]) read(40, "\0", 200) = 1 close(64) = 0 getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}], 4, 250) = 0 (Timeout)
389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
On 1/26/2017 7:59 PM, John McKee wrote:
We had to update our server from CentOS 6.7 to CentOS 6.8 due to security compliance. When doing so however, it caused 389 to be unstable for TLS/SSL port 636. It would be up for a minute or two, then fail with the following error when a server/script tried to connect. Non-TLS/SSL port 389 would work fine without any issues/errors. Before we patched, it would work without issues. Connection to port shows no issue with certificate.
<cut>
Hello,
I had similar problem one year ago (the thread is here https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... )
Can you try this:
In order to verify if cause is the same, run this command to see if the daemon crashes:
openssl s_client -connect LDAPHOSTNAME:636 -cipher ECDHE-RSA-AES256-GCM-SHA384
If it crashes, put this line in /etc/sysconfig/dirsrv
export NSS_DISABLE_HW_GCM=1
After this restart the service and see if it will crash again by openssl client
Hope this helps,
@Marc Sauton
These test connections were made from localhost, and from a remote ldapsearch. Also, they were from nagios queries to verify that LDAP was up and running.
@Todor Petkov
Dirsrv does not crash, just the TLS portion does not accept connections any longer. However I forgot to mention that I did try that by setting it in the init.d script, and by trying to do it via command line before running it (/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-XXXX/ -d1). Neither of those options worked for me :/
What is the exact ldapsearch command you are using?
On 01/26/2017 12:59 PM, John McKee wrote:
We had to update our server from CentOS 6.7 to CentOS 6.8 due to security compliance. When doing so however, it caused 389 to be unstable for TLS/SSL port 636. It would be up for a minute or two, then fail with the following error when a server/script tried to connect. Non-TLS/SSL port 389 would work fine without any issues/errors. Before we patched, it would work without issues. Connection to port shows no issue with certificate.
[26/Jan/2017:01:02:39 -0500] conn=97 fd=64 slot=64 SSL connection from X.X.X.X to X.X.X.X [26/Jan/2017:01:02:39 -0500] conn=97 op=-1 fd=64 closed - Unspecified failure while processing SSL Client Key Exchange handshake.
From the client:
TLS: loaded CA certificate file /etc/pki/tls/certs/bundle.crt. TLS: certificate [CN=XXXXXX.com,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated] is valid TLS: error: tlsm_PR_Recv returned -1 - error 104:Connection reset by peer TLS: error: connect - force handshake failure: errno 104 - moznss error -5961 TLS: can't connect: TLS error -5961:TCP connection reset by peer. ldap_err2string ldap_start_tls: Connect error (-11) additional info: TLS error -5961:TCP connection reset by peer ldap_sasl_bind
Normal Connection:
[26/Jan/2017:05:29:35 -0500] conn=904 fd=65 slot=65 SSL connection from X.X.X.X to X.X.X.X [26/Jan/2017:05:29:35 -0500] conn=904 TLS1.2 256-bit AES
Current Version of 389:
389-adminutil-1.1.19-1.el6.x86_64 389-ds-base-libs-1.2.11.15-74.el6.x86_64 389-ds-console-doc-1.2.6-1.el6.noarch 389-admin-1.1.35-1.el6.x86_64 389-ds-console-1.2.6-1.el6.noarch 389-dsgw-1.1.11-1.el6.x86_64 389-ds-base-1.2.11.15-74.el6.x86_64 389-console-1.1.7-1.el6.noarch
NSS:
nss-3.21.0-8.el6.x86_64 nss-softokn-3.14.3-23.el6_7.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.i686 nss-softokn-freebl-3.14.3-23.el6_7.x86_64 nss-sysinit-3.21.0-8.el6.x86_64 nss-tools-3.21.0-8.el6.x86_64 nss-util-3.21.0-2.el6.x86_64
Port is open:
tcp 0 0 :::636 :::* LISTEN
Approx Strace:
getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}], 4, 250) = 1 ([{fd=8, revents=POLLIN}]) accept(8, {sa_family=AF_INET6, sin6_port=htons(52890), inet_pton(AF_INET6, "::ffff:X.X.X.X", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 36 fcntl(36, F_GETFL) = 0x2 (flags O_RDWR) fcntl(36, F_SETFL, O_RDWR|O_NONBLOCK) = 0 fcntl(36, F_DUPFD, 64) = 64 close(36) = 0 setsockopt(64, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0 setsockopt(64, SOL_TCP, TCP_NODELAY, [0], 4) = 0 getsockname(64, {sa_family=AF_INET6, sin6_port=htons(636), inet_pton(AF_INET6, "::ffff:X.X.X.X", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}, {fd=64, events=POLLIN}], 5, 250) = 1 ([{fd=64, revents=POLLIN}]) futex(0x16ee83c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x16ee838, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}], 4, 250) = 1 ([{fd=40, revents=POLLIN}]) read(40, "\0", 200) = 1 close(64) = 0 getpeername(8, 0x7ffe450d5980, [112]) = -1 ENOTCONN (Transport endpoint is not connected) poll([{fd=40, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=-1}], 4, 250) = 0 (Timeout)
389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
@Mark Reynolds
ldapsearch -Z -h localhost -x -b dc=XX,dc=XX,dc=com uid=XXXX -d1
Which gave the output mentioned above.
On 01/26/2017 03:16 PM, John McKee wrote:
@Mark Reynolds
ldapsearch -Z -h localhost -x -b dc=XX,dc=XX,dc=com uid=XXXX -d1
What about:
ldapsearch -ZZ -h localhost -x -b dc=XX,dc=XX,dc=com uid=XXXX -d1
And what about:
ldapsearch -H "ldaps://localhost:636" -x -b dc=XX,dc=XX,dc=com uid=XXXX -d1
Did you set the cert dir in /etc/openldap/ldap.conf
TLS_CACERTDIR=<to certificate directory>
example:
TLS_CACERTDIR=/etc/dirsrv/slapd-INSTANCE
Regards, Mark
Which gave the output mentioned above. _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
@Mark Reynolds Those commands would fail as well, even the replication appears to have issues and gets the same error.
Here is my /etc/openldap/ldap.conf:
# The distinguished name of the search base. base dc=XX,dc=XX,dc=com
URI ldaps://XX.XX.com TLS_CACERT /etc/pki/tls/certs/bundle.crt TLS_REQCERT demand
We do not have the TLS_CACERTDIR listed, however it always worked without it, and we have other slaves which are working fine without it (since its managed through puppet).
It would appear that the masters only seem to be affected by this issue. The slaves have no issues at the moment.
On 01/26/2017 05:25 PM, John McKee wrote:
@Mark Reynolds Those commands would fail as well, even the replication appears to have issues and gets the same error.
Here is my /etc/openldap/ldap.conf:
# The distinguished name of the search base. base dc=XX,dc=XX,dc=com
URI ldaps://XX.XX.com TLS_CACERT /etc/pki/tls/certs/bundle.crt TLS_REQCERT demand
We do not have the TLS_CACERTDIR listed, however it always worked without it, ]
But it's not working now, so you should follow my suggestion and see if it helps. I have to update ldap.conf to get it working for me. You "might" need to restart the server after making this change. If this doesn't help then I'm not sure what is wrong.
and we have other slaves which are working fine without it (since its managed through puppet).
It would appear that the masters only seem to be affected by this issue. The slaves have no issues at the moment.
389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
@Mark Reynolds
I had no choice but to revert to my snapshot to get production back up and running, but I will experiment and will post the outcome. Thank you for your help thus far.
the error mentioned at the beginning was: [26/Jan/2017:01:02:39 -0500] conn=97 op=-1 fd=64 closed - Unspecified failure while processing SSL Client Key Exchange handshake. And with "TLS_REQCERT demand", there is likely a failed SSL server certificate verification, which may be related to either a cert chain not fully trusted, or an incorrect cert usage, or incorrect validity dates, or a non matching subject DN, or an issuer in the chain with a signature's algorithm now considered weak. if you have the luxury to try gain, collect a tcpdump trace to see the details of the SSL handshake, and/or eventually try only as a test with "TLS_REQCERT = allow" or "TLS_REQCERT = never" Thanks, M.
On Fri, Jan 27, 2017 at 9:34 AM, John McKee johnnyboy24@fedoraproject.org wrote:
@Mark Reynolds
I had no choice but to revert to my snapshot to get production back up and running, but I will experiment and will post the outcome. Thank you for your help thus far. _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
389-users@lists.fedoraproject.org