Replication + SSLCLIENTAUTH failure: setup_ol_tls_conn - failed: unable to create new TLS context
Hi all,
I have tried to set up a replication agreement on a 389ds master to send updates to a 389ds slave. The master is configure to use client certs for authentication.
The 389ds master fails each time it attempts to contact the slave with the following message, and tcpdump shows no traffic flowing over the wire:
[30/Mar/2016:17:19:19 +0000] setup_ol_tls_conn - failed: unable to create new TLS context [30/Mar/2016:17:19:19 +0000] slapi_ldap_bind - Error: could not configure the server for cert auth - error -1 - make sure the server is correctly configured for SSL/TLS [30/Mar/2016:17:19:19 +0000] NSMMReplicationPlugin - agmt="cn=Agreement ldap.example.com" (ldap:636): Replication bind with EXTERNAL auth failed: LDAP error 0 (Success) ()
The server is correctly configured for SSL/TLS, and I am able to bind to both the master and the slave over SSL port 636.
The replication agreement looks as follows:
dn: cn=Agreement ldap.example.com,cn=replica,cn=dc\3Dexample,dc\3Dcom,cn=mapping tree,cn=config objectClass: nsds5replicationagreement objectClass: top cn: Agreement ldap.example.com description: Replication agreement to ldap.example.com nsDS5ReplicaBindDN: cn=Replication Manager,cn=config nsDS5ReplicaBindMethod: SSLCLIENTAUTH nsds5replicaChangesSentSinceStartup: nsDS5ReplicaHost: ldap.example.com nsds5replicaLastInitEnd: 0 nsds5replicaLastInitStart: 20160330162755Z nsds5replicaLastInitStatus: 255 Replication error acquiring replica: unknown error nsds5replicaLastUpdateEnd: 0 nsds5replicaLastUpdateStart: 0 nsds5replicaLastUpdateStatus: 255 Replication error acquiring replica: unkno wn error - Unable to acquire replica nsDS5ReplicaPort: 636 nsDS5ReplicaRoot: dc=example,dc=com nsDS5ReplicaTransportInfo: SSL nsds5replicaUpdateInProgress: FALSE
Is there anything I can do to coax a useful error message out of the master server? "LDAP error 0 (Success)” tells me this is a bug of some kind, as why would it fail saying success?
This is 389ds on Ubuntu 14.04:
ii 389-ds-base 1.3.2.16-0ubuntu1 amd64 389 Directory Server suite - server
Regards, Graham —
How does your Replication Manager on the slave server look like?
ldapsearch -x -h <slavehost> -p <slaveport> [...] -b "cn=Replication Manager,cn=config"
Also, could you share your certmap.conf on the slave server? And the Subject in the cert?
On 03/30/2016 10:30 AM, Graham Leggett wrote:
Hi all,
I have tried to set up a replication agreement on a 389ds master to send updates to a 389ds slave. The master is configure to use client certs for authentication.
The 389ds master fails each time it attempts to contact the slave with the following message, and tcpdump shows no traffic flowing over the wire:
[30/Mar/2016:17:19:19 +0000] setup_ol_tls_conn - failed: unable to create new TLS context [30/Mar/2016:17:19:19 +0000] slapi_ldap_bind - Error: could not configure the server for cert auth - error -1 - make sure the server is correctly configured for SSL/TLS [30/Mar/2016:17:19:19 +0000] NSMMReplicationPlugin - agmt="cn=Agreement ldap.example.com" (ldap:636): Replication bind with EXTERNAL auth failed: LDAP error 0 (Success) ()
The server is correctly configured for SSL/TLS, and I am able to bind to both the master and the slave over SSL port 636.
The replication agreement looks as follows:
dn: cn=Agreement ldap.example.com,cn=replica,cn=dc\3Dexample,dc\3Dcom,cn=mapping tree,cn=config objectClass: nsds5replicationagreement objectClass: top cn: Agreement ldap.example.com description: Replication agreement to ldap.example.com nsDS5ReplicaBindDN: cn=Replication Manager,cn=config nsDS5ReplicaBindMethod: SSLCLIENTAUTH nsds5replicaChangesSentSinceStartup: nsDS5ReplicaHost: ldap.example.com nsds5replicaLastInitEnd: 0 nsds5replicaLastInitStart: 20160330162755Z nsds5replicaLastInitStatus: 255 Replication error acquiring replica: unknown error nsds5replicaLastUpdateEnd: 0 nsds5replicaLastUpdateStart: 0 nsds5replicaLastUpdateStatus: 255 Replication error acquiring replica: unkno wn error - Unable to acquire replica nsDS5ReplicaPort: 636 nsDS5ReplicaRoot: dc=example,dc=com nsDS5ReplicaTransportInfo: SSL nsds5replicaUpdateInProgress: FALSE
Is there anything I can do to coax a useful error message out of the master server? "LDAP error 0 (Success)” tells me this is a bug of some kind, as why would it fail saying success?
This is 389ds on Ubuntu 14.04:
ii 389-ds-base 1.3.2.16-0ubuntu1 amd64 389 Directory Server suite - server
Regards, Graham — -- 389 users mailing list 389-users@%(host_name)s http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
On 30 Mar 2016, at 7:45 PM, Noriko Hosoi nhosoi@redhat.com wrote:
How does your Replication Manager on the slave server look like?
ldapsearch -x -h <slavehost> -p <slaveport> [...] -b "cn=Replication Manager,cn=config"
Also, could you share your certmap.conf on the slave server? And the Subject in the cert?
As I pointed out below, tcpdump shows no traffic flows at all.
As a result the config of the slave server won’t have any bearing on the problem - that server is never contacted.
Regards, Graham —
On 30 Mar 2016, at 7:30 PM, Graham Leggett minfrin@sharp.fm wrote:
I have tried to set up a replication agreement on a 389ds master to send updates to a 389ds slave. The master is configure to use client certs for authentication.
The 389ds master fails each time it attempts to contact the slave with the following message, and tcpdump shows no traffic flowing over the wire:
[30/Mar/2016:17:19:19 +0000] setup_ol_tls_conn - failed: unable to create new TLS context [30/Mar/2016:17:19:19 +0000] slapi_ldap_bind - Error: could not configure the server for cert auth - error -1 - make sure the server is correctly configured for SSL/TLS [30/Mar/2016:17:19:19 +0000] NSMMReplicationPlugin - agmt="cn=Agreement ldap.example.com" (ldap:636): Replication bind with EXTERNAL auth failed: LDAP error 0 (Success) ()
The code looks broken, raised a bug with theoretical patch here:
https://fedorahosted.org/389/ticket/48782
Regards, Graham —
On 31 Mar 2016, at 12:25 AM, Graham Leggett minfrin@sharp.fm wrote:
[30/Mar/2016:17:19:19 +0000] setup_ol_tls_conn - failed: unable to create new TLS context [30/Mar/2016:17:19:19 +0000] slapi_ldap_bind - Error: could not configure the server for cert auth - error -1 - make sure the server is correctly configured for SSL/TLS [30/Mar/2016:17:19:19 +0000] NSMMReplicationPlugin - agmt="cn=Agreement ldap.example.com" (ldap:636): Replication bind with EXTERNAL auth failed: LDAP error 0 (Success) ()
The code looks broken, raised a bug with theoretical patch here:
Much stepping through code later.
It turns out that on Ubuntu Trusty, 389ds the server is backed with NSS as the security library, however 389ds’s replication plugin is backed with gnutls.
The NSS nicknames for certfile, keyfile and cacertdir are passed into gnutls, which then fails here as follows:
/* OpenSSL builds the cert chain for us, but GnuTLS * expects it to be present in the certfile. If it's * not, we have to build it ourselves. So we have to * do some special checks here... */ rc = tlsg_getfile( lt->lt_keyfile, &buf ); if ( rc ) return -1;
397 in tls_g.c (gdb) print rc $5 = <optimized out> (gdb) print *lt $6 = { lt_certfile = 0x7f1f9801f3a0 "Internal (Software) Token:ldap.example.com", lt_keyfile = 0x7f1f980187b0 "Server-Key", lt_dhfile = 0x0, lt_cacertfile = 0x0, lt_cacertdir = 0x7f1f98013960 "/etc/dirsrv/slapd-hg", lt_ciphersuite = 0x0, lt_crlfile = 0x0, lt_randfile = 0x0, lt_protocol_min = 768} (gdb) print buf $7 = 0x0
The “return -1” above is the origin of the “-1” return code in the logged error message.
Running ldd against the ns-slapd binary shows this (snipped):
root@ldap.example.com:~/src/openldap-2.4.31# ldd /usr/sbin/ns-slapd libnss3.so => /usr/lib/x86_64-linux-gnu/libnss3.so (0x00007f0e14e60000) libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007f0e12def000)
It looks like no kind of replication could ever work on Ubuntu Trusty, as NSS parameters are passed to gnutls, which can only fail.
Has anyone seen this kind of thing before?
Regards, Graham —
On 03/30/2016 06:45 PM, Graham Leggett wrote:
On 31 Mar 2016, at 12:25 AM, Graham Leggett minfrin@sharp.fm wrote:
[30/Mar/2016:17:19:19 +0000] setup_ol_tls_conn - failed: unable to create new TLS context [30/Mar/2016:17:19:19 +0000] slapi_ldap_bind - Error: could not configure the server for cert auth - error -1 - make sure the server is correctly configured for SSL/TLS [30/Mar/2016:17:19:19 +0000] NSMMReplicationPlugin - agmt="cn=Agreement ldap.example.com" (ldap:636): Replication bind with EXTERNAL auth failed: LDAP error 0 (Success) ()
The code looks broken, raised a bug with theoretical patch here:
Much stepping through code later.
It turns out that on Ubuntu Trusty, 389ds the server is backed with NSS as the security library, however 389ds’s replication plugin is backed with gnutls.
The NSS nicknames for certfile, keyfile and cacertdir are passed into gnutls, which then fails here as follows:
/* OpenSSL builds the cert chain for us, but GnuTLS * expects it to be present in the certfile. If it's * not, we have to build it ourselves. So we have to * do some special checks here... */ rc = tlsg_getfile( lt->lt_keyfile, &buf ); if ( rc ) return -1;397 in tls_g.c (gdb) print rc $5 = <optimized out> (gdb) print *lt $6 = { lt_certfile = 0x7f1f9801f3a0 "Internal (Software) Token:ldap.example.com", lt_keyfile = 0x7f1f980187b0 "Server-Key", lt_dhfile = 0x0, lt_cacertfile = 0x0, lt_cacertdir = 0x7f1f98013960 "/etc/dirsrv/slapd-hg", lt_ciphersuite = 0x0, lt_crlfile = 0x0, lt_randfile = 0x0, lt_protocol_min = 768} (gdb) print buf $7 = 0x0
The “return -1” above is the origin of the “-1” return code in the logged error message.
Running ldd against the ns-slapd binary shows this (snipped):
root@ldap.example.com:~/src/openldap-2.4.31# ldd /usr/sbin/ns-slapd libnss3.so => /usr/lib/x86_64-linux-gnu/libnss3.so (0x00007f0e14e60000) libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007f0e12def000)
It looks like no kind of replication could ever work on Ubuntu Trusty, as NSS parameters are passed to gnutls, which can only fail.
Has anyone seen this kind of thing before?
https://fedorahosted.org/389/ticket/47536
It's not that "389ds’s replication plugin is backed with gnutls", it is that 389ds replication uses openldap, which uses gnutls on ubuntu, and 389ds, for the moment, assumes that openldap is backed by NSS.
Regards, Graham — -- 389 users mailing list 389-users@%(host_name)s http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
389-users@lists.fedoraproject.org
-
Graham Leggett -
Noriko Hosoi -
Rich Megginson