Went to renew an externally-signed IPA CA certificate that was valid through today, and discovered that FreeIPA had decided to renew it with a self-signed cert a month ago, and had since reissued all other subsystem certs against that self-signed CA. After running through the ipa-cacert-manage renew dance and ipa-certupdate, the system store now contains the following certs, in this order:
- old, now-expired IPA CA cert - old, soon-to-be-expired external CA root cert - self-signed IPA cert - new IPA CA cert - new external CA root cert
There's also a chicken-and-egg problem with trying to renew anything, in that all new requests are signed with the self-signed IPA CA instead of the new intermediate IPA CA.
How do I unravel this, and completely purge the self-signed cert from existence? Why did FreeIPA try to renew the intermediate CA cert on its own, and why did it succeed?
(This is FreeIPA 4.7.2 on Fedora 29, which I'm stuck with until the CA chains are sorted out -- upgrading is still a manual replica replacement process, since ipa-server-upgrade and friends *still* insist on verifying a CA lifetime of >2 years, inexplicable behavior reported years ago...)
-Rob
On 1/2/20 7:24 AM, Rob Foehl via FreeIPA-users wrote:
Went to renew an externally-signed IPA CA certificate that was valid through today, and discovered that FreeIPA had decided to renew it with a self-signed cert a month ago, and had since reissued all other subsystem certs against that self-signed CA.
That is surprising, maybe there was a tracking request that triggered the renewal to self-signed. Can you check now if the self-signed CA is tracked? (It should not)
After running through the ipa-cacert-manage renew dance and ipa-certupdate, the system store now contains the following certs, in this order:
- old, now-expired IPA CA cert
- old, soon-to-be-expired external CA root cert
- self-signed IPA cert
- new IPA CA cert
- new external CA root cert
There's also a chicken-and-egg problem with trying to renew anything, in that all new requests are signed with the self-signed IPA CA instead of the new intermediate IPA CA.
As far as I understand, the private key of IPA CA does not change even when the CA is renewed from self-signed to externally-signed (or the reverse), and this means that the same key is used to issue the IPA certs. In that case, there is no difference if a cert was signed with the old or the new CA cert.
What makes you think that the new requests are signed with the self-signed IPA CA?
Do you have any issue when you try to renew other certs?
flo
How do I unravel this, and completely purge the self-signed cert from existence? Why did FreeIPA try to renew the intermediate CA cert on its own, and why did it succeed?
(This is FreeIPA 4.7.2 on Fedora 29, which I'm stuck with until the CA chains are sorted out -- upgrading is still a manual replica replacement process, since ipa-server-upgrade and friends *still* insist on verifying a CA lifetime of >2 years, inexplicable behavior reported years ago...)
-Rob _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On Thu, 2 Jan 2020, Florence Blanc-Renaud wrote:
On 1/2/20 7:24 AM, Rob Foehl via FreeIPA-users wrote:
Went to renew an externally-signed IPA CA certificate that was valid through today, and discovered that FreeIPA had decided to renew it with a self-signed cert a month ago, and had since reissued all other subsystem certs against that self-signed CA.
That is surprising, maybe there was a tracking request that triggered the renewal to self-signed. Can you check now if the self-signed CA is tracked? (It should not)
There's one like this, which presumably was the case a month ago when the self-signed CA was generated:
Request ID '20190325054235': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: <external CA> subject: <IPA CA> expires: 2022-12-31 17:47:12 EST key usage: keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes
This system has never (intentionally) used a self-signed CA, initial install was with the older external CA. This system was (re)installed as a replica about a year ago, and last upgrade attempt was on 20190325.
There's also a chicken-and-egg problem with trying to renew anything, in that all new requests are signed with the self-signed IPA CA instead of the new intermediate IPA CA.
As far as I understand, the private key of IPA CA does not change even when the CA is renewed from self-signed to externally-signed (or the reverse), and this means that the same key is used to issue the IPA certs. In that case, there is no difference if a cert was signed with the old or the new CA cert.
The unchanging private key is its own issue, but this is the reason why I didn't notice this a month earlier and haven't had more trouble since.
What makes you think that the new requests are signed with the self-signed IPA CA?
Certificate lifetimes aren't bound to the new intermediate IPA CA.
Do you have any issue when you try to renew other certs?
I can't renew any of them against the proper CA, nor are correct chains returned to clients. The question remains: how do I get rid of the self-signed CA entirely?
-Rob
On Thu, 2 Jan 2020, Rob Foehl via FreeIPA-users wrote:
The question remains: how do I get rid of the self-signed CA entirely?
Best hint toward this I've managed to find thus far is in the comments on https://pagure.io/freeipa/issue/7283 , with got me as far as the cACertificate and ipaCertIssuerSerial entries corresponding to the extraneous self-signed cert... If I remove those and the cert from the NSSDBs, then what? Reissue all dependent certs in the IPA CA chain?
What do I do about the rest of the mess it's created, and/or preventing future problems? Bug ticket for the erroneous self-signed renewal?
-Rob
On 1/13/20 10:58 AM, Rob Foehl via FreeIPA-users wrote:
On Thu, 2 Jan 2020, Rob Foehl via FreeIPA-users wrote:
The question remains: how do I get rid of the self-signed CA entirely?
Hi Rob,
there is currently no easy way to do this, except using a lot of manual commands. For info, we already have a ticket to enhance ipa-cacert-manage with a subcommand allowing to remove a cert: https://pagure.io/freeipa/issue/8124
Best hint toward this I've managed to find thus far is in the comments on https://pagure.io/freeipa/issue/7283 , with got me as far as the cACertificate and ipaCertIssuerSerial entries corresponding to the extraneous self-signed cert... If I remove those and the cert from the NSSDBs, then what? Reissue all dependent certs in the IPA CA chain?
What do I do about the rest of the mess it's created, and/or preventing future problems? Bug ticket for the erroneous self-signed renewal?
If the issue is reproducible, then yes, we do need to fix the problem. Please open an issue at https://pagure.io/freeipa/new_issue
Thanks, flo
-Rob
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On Thu, 16 Jan 2020, Florence Blanc-Renaud wrote:
On 1/13/20 10:58 AM, Rob Foehl via FreeIPA-users wrote:
On Thu, 2 Jan 2020, Rob Foehl via FreeIPA-users wrote:
The question remains: how do I get rid of the self-signed CA entirely?
Hi Rob,
there is currently no easy way to do this, except using a lot of manual commands. For info, we already have a ticket to enhance ipa-cacert-manage with a subcommand allowing to remove a cert: https://pagure.io/freeipa/issue/8124
Okay, so... What's the process? What are the commands?
Is it going to be less pain than ripping out the entire IPA environment and starting over, only to run into this same mess in another two years?
I agree that there ought to be an easy way (e.g. ipa-cacert-manage delete) to do this, but at the moment, I need *any* way to do this -- been fighting with a crippled system for three weeks now.
Best hint toward this I've managed to find thus far is in the comments on https://pagure.io/freeipa/issue/7283 , with got me as far as the cACertificate and ipaCertIssuerSerial entries corresponding to the extraneous self-signed cert... If I remove those and the cert from the NSSDBs, then what? Reissue all dependent certs in the IPA CA chain?
What do I do about the rest of the mess it's created, and/or preventing future problems? Bug ticket for the erroneous self-signed renewal?
If the issue is reproducible, then yes, we do need to fix the problem. Please open an issue at https://pagure.io/freeipa/new_issue
It is, as confirmed by installing a new F31 test VM and going through the external CA ipa-server-install. https://pagure.io/freeipa/issue/8176
(In the process, I also discovered that the install fails with CA_UNREACHABLE errors unless the system's hostname is actually resolvable in the DNS -- despite ipa-server-install still insisting on mangling /etc/hosts unnecessarily, as reported in https://pagure.io/freeipa/issue/6984 years ago. Come on.)
-Rob
On Mon, Jan 13, 2020 at 04:58:05AM -0500, Rob Foehl via FreeIPA-users wrote:
On Thu, 2 Jan 2020, Rob Foehl via FreeIPA-users wrote:
The question remains: how do I get rid of the self-signed CA entirely?
Best hint toward this I've managed to find thus far is in the comments on https://pagure.io/freeipa/issue/7283 , with got me as far as the cACertificate and ipaCertIssuerSerial entries corresponding to the extraneous self-signed cert... If I remove those and the cert from the NSSDBs, then what? Reissue all dependent certs in the IPA CA chain?
If the IPA CA's key and subject did not change, then there is no need to reissue end-entity or other subordinate certificates. Only the IPA CA certificate needs to be renewed (from self-signed to externally signed) and distributed.
Cheers, Fraser
On Mon, 20 Jan 2020, Fraser Tweedale wrote:
On Mon, Jan 13, 2020 at 04:58:05AM -0500, Rob Foehl via FreeIPA-users wrote:
On Thu, 2 Jan 2020, Rob Foehl via FreeIPA-users wrote:
The question remains: how do I get rid of the self-signed CA entirely?
Best hint toward this I've managed to find thus far is in the comments on https://pagure.io/freeipa/issue/7283 , with got me as far as the cACertificate and ipaCertIssuerSerial entries corresponding to the extraneous self-signed cert... If I remove those and the cert from the NSSDBs, then what? Reissue all dependent certs in the IPA CA chain?
If the IPA CA's key and subject did not change, then there is no need to reissue end-entity or other subordinate certificates. Only the IPA CA certificate needs to be renewed (from self-signed to externally signed) and distributed.
I did that already. Newly (re)issued certificates do not have their expiration times bound to the externally-signed CA. Anything with copies of both CA certs (as fetched by ipa-certupdate, which in and of itself is a nightmare) feeds only the self-signed CA chain to clients, not the correct intermediate cert, breaking everything that only knows about the external root.
Any chance we could just stick to the question of how to completely purge the self-signed cert from existence?
-Rob
On 1/20/20 1:54 AM, Rob Foehl via FreeIPA-users wrote:
On Mon, 20 Jan 2020, Fraser Tweedale wrote:
On Mon, Jan 13, 2020 at 04:58:05AM -0500, Rob Foehl via FreeIPA-users wrote:
On Thu, 2 Jan 2020, Rob Foehl via FreeIPA-users wrote:
The question remains: how do I get rid of the self-signed CA entirely?
Best hint toward this I've managed to find thus far is in the comments on https://pagure.io/freeipa/issue/7283 , with got me as far as the cACertificate and ipaCertIssuerSerial entries corresponding to the extraneous self-signed cert... If I remove those and the cert from the NSSDBs, then what? Reissue all dependent certs in the IPA CA chain?
If the IPA CA's key and subject did not change, then there is no need to reissue end-entity or other subordinate certificates. Only the IPA CA certificate needs to be renewed (from self-signed to externally signed) and distributed.
I did that already. Newly (re)issued certificates do not have their expiration times bound to the externally-signed CA. Anything with copies of both CA certs (as fetched by ipa-certupdate, which in and of itself is a nightmare) feeds only the self-signed CA chain to clients, not the correct intermediate cert, breaking everything that only knows about the external root.
Any chance we could just stick to the question of how to completely purge the self-signed cert from existence?
Sure, you can follow a manual process to remove the self-signed cert: 1- use ldapmodify in order to remove the cert from the LDAP database. You need first to find the exact dn, and then the exact cACertificate;binary attribute to delete. It will be stored below cn=certificates,cn=ipa,cn=etc,$BASEDN.
2- on all the IPA servers, use "certutil -D -d </path/to/db> -n <nickname>" to remove the cert from the following databases: /etc/dirsrv/slapd-DOMAIN-COM /etc/httpd/alias /etc/pki/pki-tomcat/alias/ /etc/ipa/nssdb
3- on all IPA servers and clients, run ipa-certupdate, this command will remove the cert from /usr/share/ipa/html/ca.crt /var/kerberos/krb5kdc/cacert.pem /etc/ipa/ca.crt /var/lib/ipa-client/pki/kdc-ca-bundle.pem /var/lib/ipa-client/pki/ca-bundle.pem
But as Fraser pointed out, there is no need to re-issue the other certs. flo
-Rob
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Florence Blanc-Renaud via FreeIPA-users wrote:
On 1/20/20 1:54 AM, Rob Foehl via FreeIPA-users wrote:
On Mon, 20 Jan 2020, Fraser Tweedale wrote:
On Mon, Jan 13, 2020 at 04:58:05AM -0500, Rob Foehl via FreeIPA-users wrote:
On Thu, 2 Jan 2020, Rob Foehl via FreeIPA-users wrote:
The question remains: how do I get rid of the self-signed CA entirely?
Best hint toward this I've managed to find thus far is in the comments on https://pagure.io/freeipa/issue/7283 , with got me as far as the cACertificate and ipaCertIssuerSerial entries corresponding to the extraneous self-signed cert... If I remove those and the cert from the NSSDBs, then what? Reissue all dependent certs in the IPA CA chain?
If the IPA CA's key and subject did not change, then there is no need to reissue end-entity or other subordinate certificates. Only the IPA CA certificate needs to be renewed (from self-signed to externally signed) and distributed.
I did that already. Newly (re)issued certificates do not have their expiration times bound to the externally-signed CA. Anything with copies of both CA certs (as fetched by ipa-certupdate, which in and of itself is a nightmare) feeds only the self-signed CA chain to clients, not the correct intermediate cert, breaking everything that only knows about the external root.
Any chance we could just stick to the question of how to completely purge the self-signed cert from existence?
Sure, you can follow a manual process to remove the self-signed cert: 1- use ldapmodify in order to remove the cert from the LDAP database. You need first to find the exact dn, and then the exact cACertificate;binary attribute to delete. It will be stored below cn=certificates,cn=ipa,cn=etc,$BASEDN.
2- on all the IPA servers, use "certutil -D -d </path/to/db> -n <nickname>" to remove the cert from the following databases: /etc/dirsrv/slapd-DOMAIN-COM /etc/httpd/alias /etc/pki/pki-tomcat/alias/ /etc/ipa/nssdb
3- on all IPA servers and clients, run ipa-certupdate, this command will remove the cert from /usr/share/ipa/html/ca.crt /var/kerberos/krb5kdc/cacert.pem /etc/ipa/ca.crt /var/lib/ipa-client/pki/kdc-ca-bundle.pem /var/lib/ipa-client/pki/ca-bundle.pem
But as Fraser pointed out, there is no need to re-issue the other certs.
He wants to re-issue them because while they are validly signed by the right key they aren't bound by the dates of the CA apparently.
I suppose for each one you could resubmit the certmonger request to force a new cert to be issued. The re-issued cert should conform to the CA validity period.
rob
On Mon, 20 Jan 2020, Rob Crittenden wrote:
Florence Blanc-Renaud via FreeIPA-users wrote:
Sure, you can follow a manual process to remove the self-signed cert: 1- use ldapmodify in order to remove the cert from the LDAP database. You need first to find the exact dn, and then the exact cACertificate;binary attribute to delete. It will be stored below cn=certificates,cn=ipa,cn=etc,$BASEDN.
2- on all the IPA servers, use "certutil -D -d </path/to/db> -n <nickname>" to remove the cert from the following databases: /etc/dirsrv/slapd-DOMAIN-COM /etc/httpd/alias /etc/pki/pki-tomcat/alias/ /etc/ipa/nssdb
3- on all IPA servers and clients, run ipa-certupdate, this command will remove the cert from /usr/share/ipa/html/ca.crt /var/kerberos/krb5kdc/cacert.pem /etc/ipa/ca.crt /var/lib/ipa-client/pki/kdc-ca-bundle.pem /var/lib/ipa-client/pki/ca-bundle.pem
Thanks for this.
But as Fraser pointed out, there is no need to re-issue the other certs.
He wants to re-issue them because while they are validly signed by the right key they aren't bound by the dates of the CA apparently.
Correct -- the lifetimes are all hosed, and the installed/served certificate chains are wrong in numerous places. If nothing else, the reissued certs at least make it clear which is which. Ongoing issues:
- All of the certificates with the excessive lifetimes should probably also be removed or revoked after reissue, and I'd prefer to lose the pile of expired certs as well
- ipa-certupdate won't work on any client which still only knows about the now-expired IPA CA, which is more than half -- and the rest have all three chains. There are a few CentOS 6 boxes in the mix, so I guess I have to figure this out manually either way.
- ipa-certupdate only works at all when run as root with a valid Kerberos ticket for an admin user, which is difficult when the root account isn't allowed to login and/or spawn interactive shells
- All now-destroyed replicas need to be rebuilt/replaced, using that to move up to a current 4.8.x release, and then deal with all the ID / serial / etc. skew even though replicas are replaced with themselves
I'm still not convinced I wouldn't be better off just trashing what's left of it and starting over, especially considering that there's only a handful of users and that I'm going to need to break in to or replace nearly every host anyway...
-Rob
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Fraser Tweedale -
Rob Crittenden -
Rob Foehl