On May 30, 2020, the AddTrust CA expired as a CA. I'll get to the IPA issue after a bit of background in case everyone is not familiar. The external certs we're using are from InCommon and were cross signed by AddTrust and when we originally got the certs, the trust A path was below:
AddTrust Ext CA -> UserTrust CA (intermediate) -> InCommon CA (Intermediate) -> server_cert
The B path which should have worked was: UserTrust CA (Root) -> InCommon CA (Intermediate) -> server_cert
How OpenSSL is supposed to work is after path A expires, its supposed to use path B. Unfortunately for OpenSSL and OpenLDAP in CENTOS/RHEL 7 and older there is a bug and that does not happen and will not attempt path B. See bugzilla for more information: https://bugzilla.redhat.com/show_bug.cgi?id=1840767
The only way I could get them to walk to path B was to remove the AddTrust CA from all openssl certificate stores. Also, blacklisting doesn't work either as it just made the certs as self-signed.
Fortunately there is a path C that we can deploy and force that trust path: Comodo AAA Certificate (Root) -> UserTrust CA (Intermediate) -> InCommon CA (Intermediate) -> server_cert
This is also the cert bundle now provided by InCommon.
The main issue here is when openssl "builds" the extracted certificates, it adds in the CA's from both /etc/ipa/ca.crt and from katello-ca.crt. We've been able to update the katello and push out that as an RPM, we're having issues with the ca distributed by IDM.
====== actual issue with IPA ===============
Post May 30, we could no longer log into IPA. We'd attempted to follow the process for "updating" the certificate. That didn't work. We did an install as we did end up adding a new signed server certificate. That didn't update the Root or the Intermediate CAs. So I went in with a hammer and manually removed the offending AddTrust cert chain A from the following NSSDB files: /etc/httpd/alias /etc/pki/pki-tomcat/alias /etc/dirsrv/slapd-LIDS-VIRGINIA-EDU /etc/ipa/nssdb I also manually cleaned up the /etc/ipa/ca.crt and /etc/pki/ca-trust/source/ipa.p11-kit cert stores.
With the above, I was able to get IPA to restart and we could then log into the console and do all we needed to do. The issue now is that the command "ipa-certupdate" still pulls the old AddTrust cert path and I'm pretty sure its because its stored in 389ds.
ldapsearch -x -b dc=dom,dc=example,dc=com "(objectClass=ipaCertificate)" | grep Subject ipaCertSubject: CN=Certificate Authority,O=DOM.EXAMPLE.COM ipaCertSubject: CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arbor, ipaCertSubject: CN=USERTrust RSA Certification Authority,O=The USERTRUST Netwo ipaCertSubject: CN=AddTrust External CA Root,OU=AddTrust External TTP Network,
How do I update LDAP without things blowing up (oh we're 3 node clustered as well)? Or better yet, is was there a better way to replace certs?
Our main com.example.com CA is just fine. All the articles/info I could find was replacing that and not the external CA's.
CENTOS/RHEL8 does not have this problem btw. It's fixed in openssl 1.1.1.
Also, sorry for the followup, but I forgot to mention.
All services and communication seem to be working with the exception of the following:
1. The joining new servers to IPA as the downloads the bundle for path A still and puts in in /etc/ipa/ca.crt which will then fail on the API calls to IPA. 2. Executing an ipa-certupdate on any hosts fails. For the ipa-certupdate to even work, I have to manually clean up the ca.crt with only the path C CA certificates. Then it'll start to work and hit the api, but when it rewrites the /etc/ipa/ca.crt file and fail on the last steps.
I'm guessing the join and update are both getting the CA certs from API which is reaching into the LDAP db itself. If I can get those old CAs removed and new ones added, I'm hoping all will be fixed.
On 6/4/20 9:21 PM, Peter Lewis via FreeIPA-users wrote:
On May 30, 2020, the AddTrust CA expired as a CA. I'll get to the IPA issue after a bit of background in case everyone is not familiar. The external certs we're using are from InCommon and were cross signed by AddTrust and when we originally got the certs, the trust A path was below:
AddTrust Ext CA -> UserTrust CA (intermediate) -> InCommon CA (Intermediate) -> server_cert
The B path which should have worked was: UserTrust CA (Root) -> InCommon CA (Intermediate) -> server_cert
How OpenSSL is supposed to work is after path A expires, its supposed to use path B. Unfortunately for OpenSSL and OpenLDAP in CENTOS/RHEL 7 and older there is a bug and that does not happen and will not attempt path B. See bugzilla for more information: https://bugzilla.redhat.com/show_bug.cgi?id=1840767
The only way I could get them to walk to path B was to remove the AddTrust CA from all openssl certificate stores. Also, blacklisting doesn't work either as it just made the certs as self-signed.
Fortunately there is a path C that we can deploy and force that trust path: Comodo AAA Certificate (Root) -> UserTrust CA (Intermediate) -> InCommon CA (Intermediate) -> server_cert
This is also the cert bundle now provided by InCommon.
The main issue here is when openssl "builds" the extracted certificates, it adds in the CA's from both /etc/ipa/ca.crt and from katello-ca.crt. We've been able to update the katello and push out that as an RPM, we're having issues with the ca distributed by IDM.
====== actual issue with IPA ===============
Post May 30, we could no longer log into IPA. We'd attempted to follow the process for "updating" the certificate. That didn't work. We did an install as we did end up adding a new signed server certificate. That didn't update the Root or the Intermediate CAs. So I went in with a hammer and manually removed the offending AddTrust cert chain A from the following NSSDB files: /etc/httpd/alias /etc/pki/pki-tomcat/alias /etc/dirsrv/slapd-LIDS-VIRGINIA-EDU /etc/ipa/nssdb I also manually cleaned up the /etc/ipa/ca.crt and /etc/pki/ca-trust/source/ipa.p11-kit cert stores.
With the above, I was able to get IPA to restart and we could then log into the console and do all we needed to do. The issue now is that the command "ipa-certupdate" still pulls the old AddTrust cert path and I'm pretty sure its because its stored in 389ds.
ldapsearch -x -b dc=dom,dc=example,dc=com "(objectClass=ipaCertificate)" | grep Subject ipaCertSubject: CN=Certificate Authority,O=DOM.EXAMPLE.COM ipaCertSubject: CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arbor, ipaCertSubject: CN=USERTrust RSA Certification Authority,O=The USERTRUST Netwo ipaCertSubject: CN=AddTrust External CA Root,OU=AddTrust External TTP Network,
How do I update LDAP without things blowing up (oh we're 3 node clustered as well)? Or better yet, is was there a better way to replace certs?
Our main com.example.com CA is just fine. All the articles/info I could find was replacing that and not the external CA's.
CENTOS/RHEL8 does not have this problem btw. It's fixed in openssl 1.1.1.
Hi,
starting from IPA 4.8.5 [1], the command ipa-cacert-manage provides the "delete" subcommand in order to remove a CA cert from LDAP store.
If you are using an older release, the steps are manual and require removing the certificate from LDAP. The procedure has already been discussed in this mailing list, see for instance this archive [2].
HTH, flo
[1] https://www.freeipa.org/page/Releases/4.8.5 [2] https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Thanks for the reply. We are unfortunately still on 4.6.5, so they'll manually have to be removed.
thanks for the pointer to the other discussion, Pete
I'm putting this out there to help others if they need it, but be wary as the following caveats apply: 1. I am not an expert in FreeIPA. Make a backup or snapshot if possible. For nssdb stuff, you can just tar up those directories for a backup before munging the data in there. 2. I'm not 100% on order as I've been doing this repair over the last few days. 3. There could be extra steps that are unnecessary 4. I'm on 4.6.5 so no cool new CA tools available
Here are my final steps that worked: 1. On all IPA servers, add new wildcard cert ipa-server-certinstall -v -w -d certificate.key certificate_bundle_with_servercert.pem ipa-cacert-manage renew --external-cert-file certificate_bundle_with_servercert.cer --external-cert-file locate_ca.pem (I'm not sure this did anything) 2. On all IPA servers, clean up all nss cache files by hand (as to not delete the wrong cert) a. list the contents certutil -L -d /etc/httpd/alias /etc/pki/pki-tomcat/alias /etc/dirsrv/slapd-DOM-EXAMPLE-COM /etc/ipa/nssdb b. Delete all but the *.lids and the DOM.EXAMPLE.COM and any internal CA's like "Server-Cert cert-pki-ca" certutil -D -d /etc/httpd/alias -n "CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arbor,ST=MI,C=US" certutil -D -d /etc/pki/pki-tomcat/alias -n "CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US" etc 3. On all IPA servers, add in the 3 certs for the new path C chain. They're named locally as c1, c2 and c3.pem locally. Comodo, InCommon, Usertrust (I scripted it to take $1 as the nssdb path) to each of the above nssdb caches. certutil -A -d $1 -i c1.pem -n "C=GB, ST=Greater Manchester, L=Salford, O=Comodo CA Limited, CN=AAA Certificate Services" -t "C,," certutil -A -d $1 -i c2.pem -n "C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust RSA Certification Authority" -t "C,," certutil -A -d $1 -i c3.pem -n "C=US, ST=MI, L=Ann Arbor, O=Internet2, OU=InCommon, CN=InCommon RSA Server CA" -t "C,," 4. On all IPA servers, update the /etc/ipa/ca.crt with the chain by hand 5. On all IPA servers, restart IPA client a. ipasvc restart 6. On just one Server, now to remove the CA's from LDAP itself (it gets replicated to the other 389ds servers). a. Get the DN names: ldapsearch -x -o ldif-wrap=no -b dc=dom,dc=example,dc=com"(objectClass=ipaCertificate)" | grep dn: b. Run the ldapdelete command on each (3 different CA's in my case): ldapdelete "cn=CN\3DInCommon RSA Server CA\2COU\3DInCommon\2CO\3DInternet2\2CL\3DAnn Arbor\2CST\3DMI\2CC\3DUS,cn=certificates,cn=ipa,cn=etc,dc=dom,dc=example,dc=com" -D 'cn=directory manager' -W 7. On the same server as above, now to Add in the CA's again: a. ipa-cacert-manage -v install certificate_bundle_without_servercert.pem 8. On all IPA servers run: a. kinit admin (or whatever admin account you're using) b. ipa-certupdate 9. On all client's: a. manually update /etc/ipa/ca.crt with the new chain + local CA. Same step as we did with #5 above b. kinit admin c. ipa-certupdate Good luck.
Hi Peter, this is generally good info, and all the cleanups you mention below are worth doing.
I just want to mention that if someone is in a pinch and needs to prioritize operation that the only fixes that are really necessary are those that involve certificate chains sent from servers to clients.
Namely the changes to the NSSDBs (steps 1 to 3, plus restart of server's services to reread the new chains)
The changes to the certificate stores (like ca.crt or LDAP) (steps 4 and onwards) are definitely a good idea, but are not technically necessary once the TLS servers send the right set of non-expired chains to the clients.
So, definitely prioritize work on the servers, and leave clients last.
HTH, Simo.
On Fri, 2020-06-05 at 18:34 +0000, Peter Lewis via FreeIPA-users wrote:
I'm putting this out there to help others if they need it, but be wary as the following caveats apply:
- I am not an expert in FreeIPA. Make a backup or snapshot if possible. For nssdb stuff, you can just tar up those directories for a backup before munging the data in there.
- I'm not 100% on order as I've been doing this repair over the last few days.
- There could be extra steps that are unnecessary
- I'm on 4.6.5 so no cool new CA tools available
Here are my final steps that worked:
- On all IPA servers, add new wildcard cert ipa-server-certinstall -v -w -d certificate.key certificate_bundle_with_servercert.pem ipa-cacert-manage renew --external-cert-file certificate_bundle_with_servercert.cer --external-cert-file locate_ca.pem (I'm not sure this did anything)
2. On all IPA servers, clean up all nss cache files by hand (as to not delete the wrong cert) a. list the contents certutil -L -d /etc/httpd/alias /etc/pki/pki-tomcat/alias /etc/dirsrv/slapd-DOM-EXAMPLE-COM /etc/ipa/nssdb b. Delete all but the *.lids and the DOM.EXAMPLE.COM and any internal CA's like "Server-Cert cert-pki-ca" certutil -D -d /etc/httpd/alias -n "CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arbor,ST=MI,C=US" certutil -D -d /etc/pki/pki-tomcat/alias -n "CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US" etc 3. On all IPA servers, add in the 3 certs for the new path C chain. They're named locally as c1, c2 and c3.pem locally. Comodo, InCommon, Usertrust (I scripted it to take $1 as the nssdb path) to each of the above nssdb caches. certutil -A -d $1 -i c1.pem -n "C=GB, ST=Greater Manchester, L=Salford, O=Comodo CA Limited, CN=AAA Certificate Services" -t "C,," certutil -A -d $1 -i c2.pem -n "C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust RSA Certification Authority" -t "C,," certutil -A -d $1 -i c3.pem -n "C=US, ST=MI, L=Ann Arbor, O=Internet2, OU=InCommon, CN=InCommon RSA Server CA" -t "C,," 4. On all IPA servers, update the /etc/ipa/ca.crt with the chain by hand 5. On all IPA servers, restart IPA client a. ipasvc restart 6. On just one Server, now to remove the CA's from LDAP itself (it gets replicated to the other 389ds servers). a. Get the DN names: ldapsearch -x -o ldif-wrap=no -b dc=dom,dc=example,dc=com"(objectClass=ipaCertificate)" | grep dn: b. Run the ldapdelete command on each (3 different CA's in my case): ldapdelete "cn=CN\3DInCommon RSA Server CA\2COU\3DInCommon\2CO\3DInternet2\2CL\3DAnn Arbor\2CST\3DMI\2CC\3DUS,cn=certificates,cn=ipa,cn=etc,dc=dom,dc=example,dc=com" -D 'cn=directory manager' -W 7. On the same server as above, now to Add in the CA's again: a. ipa-cacert-manage -v install certificate_bundle_without_servercert.pem 8. On all IPA servers run: a. kinit admin (or whatever admin account you're using) b. ipa-certupdate 9. On all client's: a. manually update /etc/ipa/ca.crt with the new chain + local CA. Same step as we did with #5 above b. kinit admin c. ipa-certupdate Good luck.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Peter Lewis -
Simo Sorce