Revisiting this problem of replication and certificates. Thank you Marc Sauton for pointing out the 'dsconf' command to spill the ca-cert list.
The synopsis is: instance #1 can replicate with instance #3 when #3 has a GlobalSign cert, but not when #3 has a Let's Encrypt cert. Instance #2 has no such problem replicating.
(All instances are running 1.4.4.17 B2021.280.1354 on CentOS)
On instance #1 (and on instance #2), when we use dsconf to ask about the ca-certificate list, we get:
Certificate Name: GlobalSign Root R3 Subject DN: CN=GlobalSign,O=GlobalSign,OU=GlobalSign Root CA - R3 Issuer DN: CN=GlobalSign,O=GlobalSign,OU=GlobalSign Root CA - R3 Expires: 2029-03-18 10:00:00 Trust Flags: CT,,
Certificate Name: GlobalSign RSA Organization Validation CA - 2018 Subject DN: CN=GlobalSign RSA OV SSL CA 2018,O=GlobalSign nv-sa,C=BE Issuer DN: CN=GlobalSign,O=GlobalSign,OU=GlobalSign Root CA - R3 Expires: 2028-11-21 00:00:00 Trust Flags: CT,,
Certificate Name: Lets Encrypt Top Subject DN: CN=ISRG Root X1,O=Internet Security Research Group,C=US Issuer DN: CN=DST Root CA X3,O=Digital Signature Trust Co. Expires: 2024-09-30 18:14:03 Trust Flags: CT,,
Certificate Name: Lets Encrypt Intermediate Subject DN: CN=R3,O=Let's Encrypt,C=US Issuer DN: CN=ISRG Root X1,O=Internet Security Research Group,C=US Expires: 2025-09-15 16:00:00 Trust Flags: CT,,
On instance #3, when a GlobalSign cert is installed and port 636 is queried with openssl s_client:
CONNECTED(00000003) depth=2 OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign verify return:1 depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018 verify return:1 depth=0 C = US, ST = Alaska, L = Juneau, O = State of Alaska, CN = ancds10.state.ak.us verify return:1
Certificate chain 0 s:C = US, ST = Alaska, L = Juneau, O = State of Alaska, CN = ancds10.state.ak.us i:C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018 a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Apr 26 18:06:15 2023 GMT; NotAfter: May 27 18:06:14 2024 GMT 1 s:C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018 i:OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Nov 21 00:00:00 2018 GMT; NotAfter: Nov 21 00:00:00 2028 GMT 2 s:OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign i:OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Mar 18 10:00:00 2009 GMT; NotAfter: Mar 18 10:00:00 2029 GMT
Server certificate -----BEGIN CERTIFICATE----- MIIGijCCBXKgAwIBAgIMfqUEla3ttBPK/lvlMA0GCSqGSIb3DQEBCwUAMFAxCzAJ BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSYwJAYDVQQDEx1H bG9iYWxTaWduIFJTQSBPViBTU0wgQ0EgMjAxODAeFw0yMzA0MjYxODA2MTVaFw0y NDA1MjcxODA2MTRaMGcxCzAJBgNVBAYTAlVTMQ8wDQYDVQQIEwZBbGFza2ExDzAN BgNVBAcTBkp1bmVhdTEYMBYGA1UEChMPU3RhdGUgb2YgQWxhc2thMRwwGgYDVQQD ExNhbmNkczEwLnN0YXRlLmFrLnVzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA12s9pW28BNfMPMUdV54DFGg7EmJv7pcmYhnOvq0vqQ845tEillOHptUj muwVOgj8Dcl+PPiDHXeggwKdMA9253Pov7eVGzKfmNN1IwSmOZYKaNLKy1CNQS13 eD0Wov0+yq35CqRHWwsl8+7Og56IfPXSmmRQPp21VBC//qkcBezomtTaSSzeE1op 248cN8H0wjL0gsPdujzrJmr7xP1gNT4gZQVkNlEAo8hJ3IxvSPvJ+E24FJwMixVb ICUz/crjRXG9nSSudP/225GxaaG3QPOSzOIZD4sT+Pt7lxmQU1syTChd5SHVK1AS WW4I2z1j68/o/ujXiqeP4iiMysRzXQIDAQABo4IDSzCCA0cwDgYDVR0PAQH/BAQD AgWgMIGOBggrBgEFBQcBAQSBgTB/MEQGCCsGAQUFBzAChjhodHRwOi8vc2VjdXJl Lmdsb2JhbHNpZ24uY29tL2NhY2VydC9nc3JzYW92c3NsY2EyMDE4LmNydDA3Bggr BgEFBQcwAYYraHR0cDovL29jc3AuZ2xvYmFsc2lnbi5jb20vZ3Nyc2FvdnNzbGNh MjAxODBWBgNVHSAETzBNMEEGCSsGAQQBoDIBFDA0MDIGCCsGAQUFBwIBFiZodHRw czovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAIBgZngQwBAgIwCQYD VR0TBAIwADA/BgNVHR8EODA2MDSgMqAwhi5odHRwOi8vY3JsLmdsb2JhbHNpZ24u Y29tL2dzcnNhb3Zzc2xjYTIwMTguY3JsMB4GA1UdEQQXMBWCE2FuY2RzMTAuc3Rh dGUuYWsudXMwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB8GA1UdIwQY MBaAFPjvf/LNeGeo3m+PJI2I8YcDArPrMB0GA1UdDgQWBBQ7Sj3cl8+m/NHTW2Gs QN+x7P7iMTCCAX8GCisGAQQB1nkCBAIEggFvBIIBawFpAHYAc9meiRtMlnigIH1H neayxhzQUV5xGSqMa4AQesF3crUAAAGHvr64EQAABAMARzBFAiBQcBIdszriaxKs vUFlLbKEH4jRYQoKwWjWX7sKIJRn2AIhAJ54vIDKjHY+si3R6qKS/xKhHUWASU4l YKxu/GPB+Z22AHYA7s3QZNXbGs7FXLedtM0TojKHRny87N7DUUhZRnEftZsAAAGH vr65OAAABAMARzBFAiBFFZn7eXA4dcDwX4+DbuHqMgtFSqGWcb18/kNymrcXzAIh AI1RfNaDFrbkPE2l/xc/Rj9C6zI4BvlGpcyw7CI+qAB3AHcA2ra/az+1tiKfm8K7 XGvocJFxbLtRhIU0vaQ9MEjX+6sAAAGHvr64vwAABAMASDBGAiEAjTr0QpjbfuZw kH1C6/rvfI8vcuZsMijy3cByBjpiO4ACIQDYFzAcyjiG+8WA0KpM1roXLFZp/GXf 8hJP+yYVvKVtXzANBgkqhkiG9w0BAQsFAAOCAQEAld/5Th8eHwDUO273c0ISRWfI ts1j/AzyhbKhhKJI/CuALjB34jsynQoqDOS4LevMsVwftGcw0LYzTNDyKaKtUKb5 Uj5El1dUdhne2Je+5jKu6lOeCvM5HZg/kEBdb5JRsl1GQxvnxlgEMq+kfFaphUAj 3u+zVkJsdbMk6mUBy8+7+NZM7l5c1QiXbFMP/VGh3u3fgVOgcLUuKMZaHa9UbhUq mS6nOmmuVE9xNBgyCYfS2Fwrp+2j5zktFBzoxe+L8i37DVp83GKZQZxx5mOCC28J YN57RNh3T7i+nDsGRVoVp28b2SmDKTh20tOYMs89khe0npancuIl7rklo+BlSg== -----END CERTIFICATE----- subject=C = US, ST = Alaska, L = Juneau, O = State of Alaska, CN = ancds10.state.ak.us issuer=C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018
No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits
SSL handshake has read 4159 bytes and written 387 bytes Verification: OK
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok)
DONE
On instance #3 when a Let's Encrypt cert is installed and port 636 queried with openssl s_client:
CONNECTED(00000003) depth=2 OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign verify return:1 depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018 verify return:1 depth=0 C = US, ST = Alaska, L = Juneau, O = State of Alaska, CN = ancds10.state.ak.us verify return:1
Certificate chain 0 s:C = US, ST = Alaska, L = Juneau, O = State of Alaska, CN = ancds10.state.ak.us i:C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018 a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Apr 26 18:06:15 2023 GMT; NotAfter: May 27 18:06:14 2024 GMT 1 s:C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018 i:OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Nov 21 00:00:00 2018 GMT; NotAfter: Nov 21 00:00:00 2028 GMT 2 s:OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign i:OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Mar 18 10:00:00 2009 GMT; NotAfter: Mar 18 10:00:00 2029 GMT
Server certificate -----BEGIN CERTIFICATE----- MIIGijCCBXKgAwIBAgIMfqUEla3ttBPK/lvlMA0GCSqGSIb3DQEBCwUAMFAxCzAJ BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSYwJAYDVQQDEx1H bG9iYWxTaWduIFJTQSBPViBTU0wgQ0EgMjAxODAeFw0yMzA0MjYxODA2MTVaFw0y NDA1MjcxODA2MTRaMGcxCzAJBgNVBAYTAlVTMQ8wDQYDVQQIEwZBbGFza2ExDzAN BgNVBAcTBkp1bmVhdTEYMBYGA1UEChMPU3RhdGUgb2YgQWxhc2thMRwwGgYDVQQD ExNhbmNkczEwLnN0YXRlLmFrLnVzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA12s9pW28BNfMPMUdV54DFGg7EmJv7pcmYhnOvq0vqQ845tEillOHptUj muwVOgj8Dcl+PPiDHXeggwKdMA9253Pov7eVGzKfmNN1IwSmOZYKaNLKy1CNQS13 eD0Wov0+yq35CqRHWwsl8+7Og56IfPXSmmRQPp21VBC//qkcBezomtTaSSzeE1op 248cN8H0wjL0gsPdujzrJmr7xP1gNT4gZQVkNlEAo8hJ3IxvSPvJ+E24FJwMixVb ICUz/crjRXG9nSSudP/225GxaaG3QPOSzOIZD4sT+Pt7lxmQU1syTChd5SHVK1AS WW4I2z1j68/o/ujXiqeP4iiMysRzXQIDAQABo4IDSzCCA0cwDgYDVR0PAQH/BAQD AgWgMIGOBggrBgEFBQcBAQSBgTB/MEQGCCsGAQUFBzAChjhodHRwOi8vc2VjdXJl Lmdsb2JhbHNpZ24uY29tL2NhY2VydC9nc3JzYW92c3NsY2EyMDE4LmNydDA3Bggr BgEFBQcwAYYraHR0cDovL29jc3AuZ2xvYmFsc2lnbi5jb20vZ3Nyc2FvdnNzbGNh MjAxODBWBgNVHSAETzBNMEEGCSsGAQQBoDIBFDA0MDIGCCsGAQUFBwIBFiZodHRw czovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAIBgZngQwBAgIwCQYD VR0TBAIwADA/BgNVHR8EODA2MDSgMqAwhi5odHRwOi8vY3JsLmdsb2JhbHNpZ24u Y29tL2dzcnNhb3Zzc2xjYTIwMTguY3JsMB4GA1UdEQQXMBWCE2FuY2RzMTAuc3Rh dGUuYWsudXMwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB8GA1UdIwQY MBaAFPjvf/LNeGeo3m+PJI2I8YcDArPrMB0GA1UdDgQWBBQ7Sj3cl8+m/NHTW2Gs QN+x7P7iMTCCAX8GCisGAQQB1nkCBAIEggFvBIIBawFpAHYAc9meiRtMlnigIH1H neayxhzQUV5xGSqMa4AQesF3crUAAAGHvr64EQAABAMARzBFAiBQcBIdszriaxKs vUFlLbKEH4jRYQoKwWjWX7sKIJRn2AIhAJ54vIDKjHY+si3R6qKS/xKhHUWASU4l YKxu/GPB+Z22AHYA7s3QZNXbGs7FXLedtM0TojKHRny87N7DUUhZRnEftZsAAAGH vr65OAAABAMARzBFAiBFFZn7eXA4dcDwX4+DbuHqMgtFSqGWcb18/kNymrcXzAIh AI1RfNaDFrbkPE2l/xc/Rj9C6zI4BvlGpcyw7CI+qAB3AHcA2ra/az+1tiKfm8K7 XGvocJFxbLtRhIU0vaQ9MEjX+6sAAAGHvr64vwAABAMASDBGAiEAjTr0QpjbfuZw kH1C6/rvfI8vcuZsMijy3cByBjpiO4ACIQDYFzAcyjiG+8WA0KpM1roXLFZp/GXf 8hJP+yYVvKVtXzANBgkqhkiG9w0BAQsFAAOCAQEAld/5Th8eHwDUO273c0ISRWfI ts1j/AzyhbKhhKJI/CuALjB34jsynQoqDOS4LevMsVwftGcw0LYzTNDyKaKtUKb5 Uj5El1dUdhne2Je+5jKu6lOeCvM5HZg/kEBdb5JRsl1GQxvnxlgEMq+kfFaphUAj 3u+zVkJsdbMk6mUBy8+7+NZM7l5c1QiXbFMP/VGh3u3fgVOgcLUuKMZaHa9UbhUq mS6nOmmuVE9xNBgyCYfS2Fwrp+2j5zktFBzoxe+L8i37DVp83GKZQZxx5mOCC28J YN57RNh3T7i+nDsGRVoVp28b2SmDKTh20tOYMs89khe0npancuIl7rklo+BlSg== -----END CERTIFICATE----- subject=C = US, ST = Alaska, L = Juneau, O = State of Alaska, CN = ancds10.state.ak.us issuer=C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018
No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits
SSL handshake has read 4159 bytes and written 387 bytes Verification: OK
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok)
DONE
It looks to me like that certificate is signed by the certs included in the list of ca-certs shown with dsconf.
But when I try to establish a replication agreement (#1 as the supplier/#3 as the consumer), I get the following in the error log:
slapi_ldap_bind - Could not send bind request for id [cn=replication manager,cn=config] authentication mechanism [SIMPLE]: error -1 (Can't contact LDAP server), system error -5987 (Invalid function argument.), network error 0 (Unknown error, host "ancds10.state.ak.us:636")
NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=DS11-ancds10" (ancds10:636) - Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) (error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (unable to get issuer certificate))
That says to me that instance #1 is unable to validate the certificate being offered by instance #3
But if I define #2 as the supplier, the replication works just fine to #3 . . and dsconf on instance #2 shows exactly the same ca-certs list as on #1
I can't figure out what I'm missing.
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston@alaska.gov Department of Administration State of Alaska
On 4/26/2023 12:43 PM, John Thurston wrote:
I have two hosts with 389-Directory/1.4.4.17 B2021.280.1354 on CentOS Stream release 8 (4.18.0-448.el8.x86_64)
On a.state.ak.us, there is one instance defined (call this instance #1)
On b.state.ak.us, there are two instances defined (call them #2 and #3)
Instances #1 and #3 have GlobalSign certificates installed. Instance #2 currently has a Let's Encrypt certificate installed. All instances also have root and intermediate certs in their databases for GlobalSign, which are marked with Trust Flags "CT,,".
I can define instance #2 as a supplier, and define a replication agreement which populates #3. This works with both LDAPS and STARTTLS.
If I, instead, try to define the same replication agreement on instance #1, it fails with:
slapi_ldap_bind - Error: could not send startTLS request: error -11 (Connect error)
NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=DS11-1to3" (b:389) - Replication bind with SIMPLE auth failed: LDAP error -11 (Connect error) (error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (unable to get issuer certificate))
slapi_ldap_bind - Error: could not send startTLS request: error -11 (Connect error)
I am unable to figure out how instances #1 and #2 differ.
Instance #1 has long-established supplier-agreements (using both LDAPS and STARTTLS) with other instances of 389-Directory. So I know instance #1 can function correctly as a supplier. Instance #3 demonstrates it can be a consumer when supplied by instance #2. I can perform LDAPS and STARTTLS queries from a.state.ak.us to instance #3, so I know it is listening on the network and not blocked by a host-based firewall.
Any suggestions of where to look, or config-attributes to check, would be appreciated.