Roderick Johnstone via FreeIPA-users wrote: > Hi > > Our freeipa certificates need to be renewed due to passing their expiry > dates. > > While some certificates have renewed ok, the ipaCert and > auditSigningCert are renewing but the new certificates have the wrong > Subject. > > Environment is: > serverA (CRL, first, master) RHEL 7.3, ipa 4.4 > serverB (replica) RHEL 7.3, ipa 4.4 > serverC (replica) RHEL 7.4, ipa 4.5 > > Once there are renewed certificates with the wrong Subject present, > there are various problems with renewing the remaining certificates, > which I think might be related to the bad Subject: > > 1) When just ipaCert has the wrong subject no further renewals happen > > 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd > service will not start and no further renewals happen. > > I've been round the following loop many times on ServerA, our first master: > > 1) Restore good certificates from backup > 2) Put the clock back to a time when certificates are all valid > 3) Resubmit certificates for renewal > > Each time the ipaCert renews it has the same wrong Subject. The wrong > Subject includes the host name of one of our ipa client systems. > > Each time the auditSigningCert renews it has the same wrong Subject but > a different subject to the ipaCert. The wrong Subject in this case > includes the host name of a system which has never been an ipa client, > but might have been added and removed with ipa host-add and ipa host-del > for testing something, a while ago. > > As far as I can see, the "cert_subject" is set correctly in the file > /var/lib/certmonger/<request id> until the point at which the > certificate is actually renewed. > > I'd be very grateful for some pointers as to which configuration options > and logs to check through to resolve this problem on our production system. > > If its of any relevance we did change which server is the first master > some time ago. I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what the subject is.

Roderick Johnstone via FreeIPA-users wrote: > On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote: >> Roderick Johnstone via FreeIPA-users wrote: >>> Hi >>> >>> Our freeipa certificates need to be renewed due to passing their expiry >>> dates. >>> >>> While some certificates have renewed ok, the ipaCert and >>> auditSigningCert are renewing but the new certificates have the wrong >>> Subject. >>> >>> Environment is: >>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4 >>> serverB (replica) RHEL 7.3, ipa 4.4 >>> serverC (replica) RHEL 7.4, ipa 4.5 >>> >>> Once there are renewed certificates with the wrong Subject present, >>> there are various problems with renewing the remaining certificates, >>> which I think might be related to the bad Subject: >>> >>> 1) When just ipaCert has the wrong subject no further renewals happen >>> >>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd >>> service will not start and no further renewals happen. >>> >>> I've been round the following loop many times on ServerA, our first >>> master: >>> >>> 1) Restore good certificates from backup >>> 2) Put the clock back to a time when certificates are all valid >>> 3) Resubmit certificates for renewal >>> >>> Each time the ipaCert renews it has the same wrong Subject. The wrong >>> Subject includes the host name of one of our ipa client systems. >>> >>> Each time the auditSigningCert renews it has the same wrong Subject but >>> a different subject to the ipaCert. The wrong Subject in this case >>> includes the host name of a system which has never been an ipa client, >>> but might have been added and removed with ipa host-add and ipa host-del >>> for testing something, a while ago. >>> >>> As far as I can see, the "cert_subject" is set correctly in the file >>> /var/lib/certmonger/<request id> until the point at which the >>> certificate is actually renewed. >>> >>> I'd be very grateful for some pointers as to which configuration options >>> and logs to check through to resolve this problem on our production >>> system. >>> >>> If its of any relevance we did change which server is the first master >>> some time ago. >> >> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what >> the subject is. > > I'm not seeing any obvious CSR fields in the > /etc/pki/pki-tomcat/ca/CS.cfg file. foo.bar.certreq=

> The CSR in the certmonger requests file for the auditSigningCert seems > to be showing with the correct Subject. This is different from the bad > subject showing in the requests file field: > cert_subject= The value of cert_subject comes from the issued certificate. > and the Subject which is showing in the 'getcert list' output (which is > the same as that in the cert_subject= field.> > I'm not quite sure what this all means. It is displayed from the data within the tracked certmonger request.

certmonger logs to syslog so you can check there or you can stop the process and run it manually with: certmonger -n -d 9 2>&1 | tee certmonger.log That will provide a lot of debugging output that may show what is going on.

On 16/01/2018 12:14, Roderick Johnstone via FreeIPA-users wrote: Hi Rob This is all on my first master server. I put the clock back to when the certificates that O restore form backup are all valid. I restored the databases in /etc/httpd/alias and /etc/pki/pki-tomcat/alias from the last good backup I had. I also restored the CS.cfg file from backup. I updated the trusts in /etc/pki/pki-tomcat/alias for caSigningCert cert-pki-ca to match what is in section 5 of: https://access.redhat.com/solutions/643753 This was previously: caSigningCert cert-pki-ca                                    CTu,u,u for some reason. I stopped the certmonger service and run the certmonger command you gave to start verbose logging. I was able to start all the ipa services after running: pki-server subsystem-enable ca (this seems to become disabled when the tomcatd service cannot start. I ran getcert resubmit -i <requestid> for the expiring certificates. The first one I tried (ocspSigningCert) renewed but gets an odd Subject. It includes the hostname of one of my replica servers. The other certificates have not renewed. As you said, there is a large amount of info in the verbose certmonger debug logs, but it is not immediately obvious to me what has gone wrong, except that there are some instances of: Internal error Would you be prepared to have a look at the log file off-list (3.3MB file, uncompressed) to see if it means more to you.

On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote: > Roderick Johnstone via FreeIPA-users wrote: >> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote: >>> Roderick Johnstone via FreeIPA-users wrote: >>>> Hi >>>> >>>> Our freeipa certificates need to be renewed due to passing their >>>> expiry >>>> dates. >>>> >>>> While some certificates have renewed ok, the ipaCert and >>>> auditSigningCert are renewing but the new certificates have the wrong >>>> Subject. >>>> >>>> Environment is: >>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4 >>>> serverB (replica) RHEL 7.3, ipa 4.4 >>>> serverC (replica) RHEL 7.4, ipa 4.5 >>>> >>>> Once there are renewed certificates with the wrong Subject present, >>>> there are various problems with renewing the remaining certificates, >>>> which I think might be related to the bad Subject: >>>> >>>> 1) When just ipaCert has the wrong subject no further renewals happen >>>> >>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd >>>> service will not start and no further renewals happen. >>>> >>>> I've been round the following loop many times on ServerA, our first >>>> master: >>>> >>>> 1) Restore good certificates from backup >>>> 2) Put the clock back to a time when certificates are all valid >>>> 3) Resubmit certificates for renewal >>>> >>>> Each time the ipaCert renews it has the same wrong Subject. The wrong >>>> Subject includes the host name of one of our ipa client systems. >>>> >>>> Each time the auditSigningCert renews it has the same wrong Subject >>>> but >>>> a different subject to the ipaCert. The wrong Subject in this case >>>> includes the host name of a system which has never been an ipa client, >>>> but might have been added and removed with ipa host-add and ipa >>>> host-del >>>> for testing something, a while ago. >>>> >>>> As far as I can see, the "cert_subject" is set correctly in the file >>>> /var/lib/certmonger/<request id> until the point at which the >>>> certificate is actually renewed. >>>> >>>> I'd be very grateful for some pointers as to which configuration >>>> options >>>> and logs to check through to resolve this problem on our production >>>> system. >>>> >>>> If its of any relevance we did change which server is the first master >>>> some time ago. >>> >>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what >>> the subject is. >> >> I'm not seeing any obvious CSR fields in the >> /etc/pki/pki-tomcat/ca/CS.cfg file. > > foo.bar.certreq= > >> The CSR in the certmonger requests file for the auditSigningCert seems >> to be showing with the correct Subject. This is different from the bad >> subject showing in the requests file field: >> cert_subject= > > The value of cert_subject comes from the issued certificate. > >> and the Subject which is showing in the 'getcert list' output (which is >> the same as that in the cert_subject= field.> >> I'm not quite sure what this all means. > > It is displayed from the data within the tracked certmonger request. > > certmonger logs to syslog so you can check there or you can stop the > process and run it manually with: certmonger -n -d 9 2>&1 | tee > certmonger.log > > That will provide a lot of debugging output that may show what is > going on. I've restored certificate databases from backup and put the clock back to a time when certificates are valid and renewed the ocspSigining certificate with: getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i 20161124081302 (I've previously tried without the -N with similar results) What I am seeing in the certmonger logs is: 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert cert-pki-ca'. 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert cert-pki-ca' to public key. 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert cert-pki-ca". 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401. 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert cert-pki-ca'. 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert cert-pki-ca' to public key. 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert cert-pki-ca". 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert cert-pki-ca'. 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert cert-pki-ca' to public key. 2017-10-23 00:05:39 [581] Found a certificate with the same nickname but different subject, removing certificate "ocspSigningCert cert-pki-ca" with subject "CN=OCSP Subsystem,O=<REALM>". 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca". 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert cert-pki-ca". 2017-10-23 00:05:39 [48576] Adding hook "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"" (0). 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" issued by CA and saved. I now have a date valid ocspSigningCertificate, but with the wrong subject, and a broken certificate system which will no longer start. ipactl status ... pki-tomcatd Service: STOPPED I can't renew other expired certificates. I also note that there is now no key for ocspSigningCertificate as shown by: certutil -K -d /etc/pki/pki-tomcat/alias I wonder if this is because the certificate subject changed? There was a key before the certificate renewed. The ca debug logs are showing: [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname: 'ocspSigningCert cert-pki-ca' with serial number: 268370108 [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to x509CertImpl [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: Certificate object not found Certificate object not found     at com.netscape.ca.SigningUnit.init(SigningUnit.java:184) Any help in repairing my broken ipa system will be much appreciated.

On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote: > Roderick Johnstone via FreeIPA-users wrote: >> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote: >>> Roderick Johnstone via FreeIPA-users wrote: >>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote: >>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>> Hi >>>>>> >>>>>> Our freeipa certificates need to be renewed due to passing their >>>>>> expiry >>>>>> dates. >>>>>> >>>>>> While some certificates have renewed ok, the ipaCert and >>>>>> auditSigningCert are renewing but the new certificates have the >>>>>> wrong >>>>>> Subject. >>>>>> >>>>>> Environment is: >>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4 >>>>>> serverB (replica) RHEL 7.3, ipa 4.4 >>>>>> serverC (replica) RHEL 7.4, ipa 4.5 >>>>>> >>>>>> Once there are renewed certificates with the wrong Subject present, >>>>>> there are various problems with renewing the remaining certificates, >>>>>> which I think might be related to the bad Subject: >>>>>> >>>>>> 1) When just ipaCert has the wrong subject no further renewals >>>>>> happen >>>>>> >>>>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd >>>>>> service will not start and no further renewals happen. >>>>>> >>>>>> I've been round the following loop many times on ServerA, our first >>>>>> master: >>>>>> >>>>>> 1) Restore good certificates from backup >>>>>> 2) Put the clock back to a time when certificates are all valid >>>>>> 3) Resubmit certificates for renewal >>>>>> >>>>>> Each time the ipaCert renews it has the same wrong Subject. The >>>>>> wrong >>>>>> Subject includes the host name of one of our ipa client systems. >>>>>> >>>>>> Each time the auditSigningCert renews it has the same wrong Subject >>>>>> but >>>>>> a different subject to the ipaCert. The wrong Subject in this case >>>>>> includes the host name of a system which has never been an ipa >>>>>> client, >>>>>> but might have been added and removed with ipa host-add and ipa >>>>>> host-del >>>>>> for testing something, a while ago. >>>>>> >>>>>> As far as I can see, the "cert_subject" is set correctly in the file >>>>>> /var/lib/certmonger/<request id> until the point at which the >>>>>> certificate is actually renewed. >>>>>> >>>>>> I'd be very grateful for some pointers as to which configuration >>>>>> options >>>>>> and logs to check through to resolve this problem on our production >>>>>> system. >>>>>> >>>>>> If its of any relevance we did change which server is the first >>>>>> master >>>>>> some time ago. >>>>> >>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what >>>>> the subject is. >>>> >>>> I'm not seeing any obvious CSR fields in the >>>> /etc/pki/pki-tomcat/ca/CS.cfg file. >>> >>> foo.bar.certreq= >>> >>>> The CSR in the certmonger requests file for the auditSigningCert seems >>>> to be showing with the correct Subject. This is different from the bad >>>> subject showing in the requests file field: >>>> cert_subject= >>> >>> The value of cert_subject comes from the issued certificate. >>> >>>> and the Subject which is showing in the 'getcert list' output >>>> (which is >>>> the same as that in the cert_subject= field.> >>>> I'm not quite sure what this all means. >>> >>> It is displayed from the data within the tracked certmonger request. >>> >>> certmonger logs to syslog so you can check there or you can stop the >>> process and run it manually with: certmonger -n -d 9 2>&1 | tee >>> certmonger.log >>> >>> That will provide a lot of debugging output that may show what is >>> going on. >> >> I've restored certificate databases from backup and put the clock back >> to a time when certificates are valid and renewed the ocspSigining >> certificate with: >> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i 20161124081302 >> >> (I've previously tried without the -N with similar results) >> >> What I am seeing in the certmonger logs is: >> >> >> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert cert-pki-ca'. >> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert >> cert-pki-ca' to public key. >> 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert >> cert-pki-ca". >> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert >> cert-pki-ca" in token "NSS Certificate DB" in database >> "/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401. >> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert cert-pki-ca'. >> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert >> cert-pki-ca' to public key. >> 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert >> cert-pki-ca". >> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert cert-pki-ca'. >> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert >> cert-pki-ca' to public key. >> 2017-10-23 00:05:39 [581] Found a certificate with the same nickname but >> different subject, removing certificate "ocspSigningCert cert-pki-ca" >> with subject "CN=OCSP Subsystem,O=<REALM>". >> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert >> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca". >> 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert >> cert-pki-ca". >> 2017-10-23 00:05:39 [48576] Adding hook >> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert >> cert-pki-ca"" (0). >> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert >> cert-pki-ca" in token "NSS Certificate DB" in database >> "/etc/pki/pki-tomcat/alias" issued by CA and saved. >> >> I now have a date valid ocspSigningCertificate, but with the wrong >> subject, and a broken certificate system which will no longer start. >> >> ipactl status >> ... >> pki-tomcatd Service: STOPPED >> >> I can't renew other expired certificates. >> >> I also note that there is now no key for ocspSigningCertificate as shown >> by: >> certutil -K -d /etc/pki/pki-tomcat/alias >> >> I wonder if this is because the certificate subject changed? There was a >> key before the certificate renewed. >> >> The ca debug logs are showing: >> >> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname: >> 'ocspSigningCert cert-pki-ca' with serial number: 268370108 >> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to x509CertImpl >> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: Certificate >> object not found >> Certificate object not found >>      at com.netscape.ca.SigningUnit.init(SigningUnit.java:184) >> >> Any help in repairing my broken ipa system will be much appreciated. > > Sorry for the delay. I think my previous reading of the certmonger > csrgen code was wrong. > > IIRC in your certmonger entry the cert_subject has the hostname value > right? Do you also have cert_subject_der? > > You can decode that by: > > 1. Running a hex-to-bin, something hacky like this in python: > > import binascii > > hex_string = "<hex value>" > > binary_string = binascii.unhexlify(hex_string) > > fd = open("out", "wb") > fd.write(binary_string) > fd.close() > > 2. Run: openssl asn1parse -in out -inform der > > I'm going to assume this also has the hostname encoded in it. Hi Again Yes, correct. The cert_subject_der does have the bad CN with the hostname encoded in it. > > Can you try this: > > 1. Make a backup of the request file (just in case) > 2. Remove > cert_subject_der > 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM> > 3. Try the renewal again I ran though all of this, checking that the request file was still what we wanted after restarting certmonger with verbose logging, restoring all the databases, fixing permissions, checking keys etc., and ran the getcert resubmit. It still generates a certificate with the bad CN including a hostname. Then the pki-tomcat server fails to start again. Other things I have checked are: 1) The csr= field in the (modified) request file has the correct subject. 2) The dogtag ca debug log file is showing: processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017 processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK ... RenewalSubmitter: renewal original profileId=caIPAserviceCert RenewalSubmitter: for renewal, original authenticator raCertAuth found CertProcessor: No authenticator credentials required processRenewal: set original Inputs into profile Context RenewalSubmitter: setInputsIntoContext() getting input name= cert_request_type RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10 RenewalSubmitter: setInputsIntoContext() getting input name= cert_request RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex encoded csr> ... Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM> The hex encoded csr in the last line decodes to have the bad subject where the hostname is one of our other ipa servers. The certificate always getthe same hostname in the bad subject cn. Do you have any other ideas of things to check to find out what's generating the bad subject?

On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote: > Roderick Johnstone via FreeIPA-users wrote: >> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote: >>> Roderick Johnstone via FreeIPA-users wrote: >>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote: >>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote: >>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>> Hi >>>>>>>> >>>>>>>> Our freeipa certificates need to be renewed due to passing their >>>>>>>> expiry >>>>>>>> dates. >>>>>>>> >>>>>>>> While some certificates have renewed ok, the ipaCert and >>>>>>>> auditSigningCert are renewing but the new certificates have the >>>>>>>> wrong >>>>>>>> Subject. >>>>>>>> >>>>>>>> Environment is: >>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4 >>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4 >>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5 >>>>>>>> >>>>>>>> Once there are renewed certificates with the wrong Subject >>>>>>>> present, >>>>>>>> there are various problems with renewing the remaining >>>>>>>> certificates, >>>>>>>> which I think might be related to the bad Subject: >>>>>>>> >>>>>>>> 1) When just ipaCert has the wrong subject no further renewals >>>>>>>> happen >>>>>>>> >>>>>>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd >>>>>>>> service will not start and no further renewals happen. >>>>>>>> >>>>>>>> I've been round the following loop many times on ServerA, our >>>>>>>> first >>>>>>>> master: >>>>>>>> >>>>>>>> 1) Restore good certificates from backup >>>>>>>> 2) Put the clock back to a time when certificates are all valid >>>>>>>> 3) Resubmit certificates for renewal >>>>>>>> >>>>>>>> Each time the ipaCert renews it has the same wrong Subject. The >>>>>>>> wrong >>>>>>>> Subject includes the host name of one of our ipa client systems. >>>>>>>> >>>>>>>> Each time the auditSigningCert renews it has the same wrong >>>>>>>> Subject >>>>>>>> but >>>>>>>> a different subject to the ipaCert. The wrong Subject in this case >>>>>>>> includes the host name of a system which has never been an ipa >>>>>>>> client, >>>>>>>> but might have been added and removed with ipa host-add and ipa >>>>>>>> host-del >>>>>>>> for testing something, a while ago. >>>>>>>> >>>>>>>> As far as I can see, the "cert_subject" is set correctly in the >>>>>>>> file >>>>>>>> /var/lib/certmonger/<request id> until the point at which the >>>>>>>> certificate is actually renewed. >>>>>>>> >>>>>>>> I'd be very grateful for some pointers as to which configuration >>>>>>>> options >>>>>>>> and logs to check through to resolve this problem on our >>>>>>>> production >>>>>>>> system. >>>>>>>> >>>>>>>> If its of any relevance we did change which server is the first >>>>>>>> master >>>>>>>> some time ago. >>>>>>> >>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see >>>>>>> what >>>>>>> the subject is. >>>>>> >>>>>> I'm not seeing any obvious CSR fields in the >>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file. >>>>> >>>>> foo.bar.certreq= >>>>> >>>>>> The CSR in the certmonger requests file for the auditSigningCert >>>>>> seems >>>>>> to be showing with the correct Subject. This is different from >>>>>> the bad >>>>>> subject showing in the requests file field: >>>>>> cert_subject= >>>>> >>>>> The value of cert_subject comes from the issued certificate. >>>>> >>>>>> and the Subject which is showing in the 'getcert list' output >>>>>> (which is >>>>>> the same as that in the cert_subject= field.> >>>>>> I'm not quite sure what this all means. >>>>> >>>>> It is displayed from the data within the tracked certmonger request. >>>>> >>>>> certmonger logs to syslog so you can check there or you can stop the >>>>> process and run it manually with: certmonger -n -d 9 2>&1 | tee >>>>> certmonger.log >>>>> >>>>> That will provide a lot of debugging output that may show what is >>>>> going on. >>>> >>>> I've restored certificate databases from backup and put the clock back >>>> to a time when certificates are valid and renewed the ocspSigining >>>> certificate with: >>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i 20161124081302 >>>> >>>> (I've previously tried without the -N with similar results) >>>> >>>> What I am seeing in the certmonger logs is: >>>> >>>> >>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert >>>> cert-pki-ca'. >>>> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert >>>> cert-pki-ca' to public key. >>>> 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert >>>> cert-pki-ca". >>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert >>>> cert-pki-ca" in token "NSS Certificate DB" in database >>>> "/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401. >>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert >>>> cert-pki-ca'. >>>> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert >>>> cert-pki-ca' to public key. >>>> 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert >>>> cert-pki-ca". >>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert >>>> cert-pki-ca'. >>>> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert >>>> cert-pki-ca' to public key. >>>> 2017-10-23 00:05:39 [581] Found a certificate with the same >>>> nickname but >>>> different subject, removing certificate "ocspSigningCert cert-pki-ca" >>>> with subject "CN=OCSP Subsystem,O=<REALM>". >>>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert >>>> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca". >>>> 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert >>>> cert-pki-ca". >>>> 2017-10-23 00:05:39 [48576] Adding hook >>>> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert >>>> cert-pki-ca"" (0). >>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert >>>> cert-pki-ca" in token "NSS Certificate DB" in database >>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved. >>>> >>>> I now have a date valid ocspSigningCertificate, but with the wrong >>>> subject, and a broken certificate system which will no longer start. >>>> >>>> ipactl status >>>> ... >>>> pki-tomcatd Service: STOPPED >>>> >>>> I can't renew other expired certificates. >>>> >>>> I also note that there is now no key for ocspSigningCertificate as >>>> shown >>>> by: >>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>> >>>> I wonder if this is because the certificate subject changed? There >>>> was a >>>> key before the certificate renewed. >>>> >>>> The ca debug logs are showing: >>>> >>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname: >>>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108 >>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to >>>> x509CertImpl >>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: >>>> Certificate >>>> object not found >>>> Certificate object not found >>>>       at com.netscape.ca.SigningUnit.init(SigningUnit.java:184) >>>> >>>> Any help in repairing my broken ipa system will be much appreciated. >>> >>> Sorry for the delay. I think my previous reading of the certmonger >>> csrgen code was wrong. >>> >>> IIRC in your certmonger entry the cert_subject has the hostname value >>> right? Do you also have cert_subject_der? >>> >>> You can decode that by: >>> >>> 1. Running a hex-to-bin, something hacky like this in python: >>> >>> import binascii >>> >>> hex_string = "<hex value>" >>> >>> binary_string = binascii.unhexlify(hex_string) >>> >>> fd = open("out", "wb") >>> fd.write(binary_string) >>> fd.close() >>> >>> 2. Run: openssl asn1parse -in out -inform der >>> >>> I'm going to assume this also has the hostname encoded in it. >> >> Hi Again >> >> Yes, correct. The cert_subject_der does have the bad CN with the >> hostname encoded in it. >> >>> >>> Can you try this: >>> >>> 1. Make a backup of the request file (just in case) > 2. Remove >>> cert_subject_der >>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM> >>> 3. Try the renewal again >> >> I ran though all of this, checking that the request file was still what >> we wanted after restarting certmonger with verbose logging, restoring >> all the databases, fixing permissions, checking keys etc., and ran the >> getcert resubmit. >> >> It still generates a certificate with the bad CN including a hostname. >> >> Then the pki-tomcat server fails to start again. >> >> Other things I have checked are: >> 1) The csr= field in the (modified) request file has the correct >> subject. >> >> 2) The dogtag ca debug log file is showing: >> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017 >> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK >> ... >> RenewalSubmitter: renewal original profileId=caIPAserviceCert >> RenewalSubmitter: for renewal, original authenticator raCertAuth found >> CertProcessor: No authenticator credentials required >> processRenewal: set original Inputs into profile Context >> RenewalSubmitter: setInputsIntoContext() getting input name= >> cert_request_type >> RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10 >> RenewalSubmitter: setInputsIntoContext() getting input name= >> cert_request >> RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex >> encoded csr> >> ... >> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM> >> >> The hex encoded csr in the last line decodes to have the bad subject >> where the hostname is one of our other ipa servers. >> >> The certificate always getthe same hostname in the bad subject cn. >> >> Do you have any other ideas of things to check to find out what's >> generating the bad subject? > > The order certmonger uses for the subject is: > > 1. cert_subject_der > 2. If there is no DER, try to encode cert_subject > 3. If that fails try again a different way > 4. If it fails yet again use CN=localhost > > It is baffling that it would pick a hostname much less one that > certmonger shouldn't even know about. Indeed, and if I try to renew other certificates for some of them it chooses other host names that should not be known about. Each certificate seems to get the same bad hostname each time I try to renew. > > I assume the CSR found in the dogtag logs matches the csr value in the > certmonger request? No, its different. The issued certificate matches the csr seen in dogtag which makes sense. But the csr in the dogtag logs has the bad subject. The csr in the request directory file has the good subject. > > Can you share the certmonger request file privately? Yes, certainly. Thanks for your continued help on this.

On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote: > Roderick Johnstone via FreeIPA-users wrote: >> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote: >>> Roderick Johnstone via FreeIPA-users wrote: >>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote: >>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote: >>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote: >>>>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> Our freeipa certificates need to be renewed due to passing their >>>>>>>>>> expiry >>>>>>>>>> dates. >>>>>>>>>> >>>>>>>>>> While some certificates have renewed ok, the ipaCert and >>>>>>>>>> auditSigningCert are renewing but the new certificates have the >>>>>>>>>> wrong >>>>>>>>>> Subject. >>>>>>>>>> >>>>>>>>>> Environment is: >>>>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4 >>>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4 >>>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5 >>>>>>>>>> >>>>>>>>>> Once there are renewed certificates with the wrong Subject >>>>>>>>>> present, >>>>>>>>>> there are various problems with renewing the remaining >>>>>>>>>> certificates, >>>>>>>>>> which I think might be related to the bad Subject: >>>>>>>>>> >>>>>>>>>> 1) When just ipaCert has the wrong subject no further renewals >>>>>>>>>> happen >>>>>>>>>> >>>>>>>>>> 2) When auditSigningCert has the wrong subject the ipa >>>>>>>>>> pki-tomcatd >>>>>>>>>> service will not start and no further renewals happen. >>>>>>>>>> >>>>>>>>>> I've been round the following loop many times on ServerA, our >>>>>>>>>> first >>>>>>>>>> master: >>>>>>>>>> >>>>>>>>>> 1) Restore good certificates from backup >>>>>>>>>> 2) Put the clock back to a time when certificates are all valid >>>>>>>>>> 3) Resubmit certificates for renewal >>>>>>>>>> >>>>>>>>>> Each time the ipaCert renews it has the same wrong Subject. The >>>>>>>>>> wrong >>>>>>>>>> Subject includes the host name of one of our ipa client systems. >>>>>>>>>> >>>>>>>>>> Each time the auditSigningCert renews it has the same wrong >>>>>>>>>> Subject >>>>>>>>>> but >>>>>>>>>> a different subject to the ipaCert. The wrong Subject in this >>>>>>>>>> case >>>>>>>>>> includes the host name of a system which has never been an ipa >>>>>>>>>> client, >>>>>>>>>> but might have been added and removed with ipa host-add and ipa >>>>>>>>>> host-del >>>>>>>>>> for testing something, a while ago. >>>>>>>>>> >>>>>>>>>> As far as I can see, the "cert_subject" is set correctly in the >>>>>>>>>> file >>>>>>>>>> /var/lib/certmonger/<request id> until the point at which the >>>>>>>>>> certificate is actually renewed. >>>>>>>>>> >>>>>>>>>> I'd be very grateful for some pointers as to which configuration >>>>>>>>>> options >>>>>>>>>> and logs to check through to resolve this problem on our >>>>>>>>>> production >>>>>>>>>> system. >>>>>>>>>> >>>>>>>>>> If its of any relevance we did change which server is the first >>>>>>>>>> master >>>>>>>>>> some time ago. >>>>>>>>> >>>>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see >>>>>>>>> what >>>>>>>>> the subject is. >>>>>>>> >>>>>>>> I'm not seeing any obvious CSR fields in the >>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file. >>>>>>> >>>>>>> foo.bar.certreq= >>>>>>> >>>>>>>> The CSR in the certmonger requests file for the auditSigningCert >>>>>>>> seems >>>>>>>> to be showing with the correct Subject. This is different from >>>>>>>> the bad >>>>>>>> subject showing in the requests file field: >>>>>>>> cert_subject= >>>>>>> >>>>>>> The value of cert_subject comes from the issued certificate. >>>>>>> >>>>>>>> and the Subject which is showing in the 'getcert list' output >>>>>>>> (which is >>>>>>>> the same as that in the cert_subject= field.> >>>>>>>> I'm not quite sure what this all means. >>>>>>> >>>>>>> It is displayed from the data within the tracked certmonger >>>>>>> request. >>>>>>> >>>>>>> certmonger logs to syslog so you can check there or you can stop >>>>>>> the >>>>>>> process and run it manually with: certmonger -n -d 9 2>&1 | tee >>>>>>> certmonger.log >>>>>>> >>>>>>> That will provide a lot of debugging output that may show what is >>>>>>> going on. >>>>>> >>>>>> I've restored certificate databases from backup and put the clock >>>>>> back >>>>>> to a time when certificates are valid and renewed the ocspSigining >>>>>> certificate with: >>>>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i 20161124081302 >>>>>> >>>>>> (I've previously tried without the -N with similar results) >>>>>> >>>>>> What I am seeing in the certmonger logs is: >>>>>> >>>>>> >>>>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert >>>>>> cert-pki-ca'. >>>>>> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert >>>>>> cert-pki-ca' to public key. >>>>>> 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert >>>>>> cert-pki-ca". >>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert >>>>>> cert-pki-ca" in token "NSS Certificate DB" in database >>>>>> "/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401. >>>>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert >>>>>> cert-pki-ca'. >>>>>> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert >>>>>> cert-pki-ca' to public key. >>>>>> 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert >>>>>> cert-pki-ca". >>>>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert >>>>>> cert-pki-ca'. >>>>>> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert >>>>>> cert-pki-ca' to public key. >>>>>> 2017-10-23 00:05:39 [581] Found a certificate with the same >>>>>> nickname but >>>>>> different subject, removing certificate "ocspSigningCert >>>>>> cert-pki-ca" >>>>>> with subject "CN=OCSP Subsystem,O=<REALM>". >>>>>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert >>>>>> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca". >>>>>> 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert >>>>>> cert-pki-ca". >>>>>> 2017-10-23 00:05:39 [48576] Adding hook >>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert >>>>>> cert-pki-ca"" (0). >>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert >>>>>> cert-pki-ca" in token "NSS Certificate DB" in database >>>>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved. >>>>>> >>>>>> I now have a date valid ocspSigningCertificate, but with the wrong >>>>>> subject, and a broken certificate system which will no longer start. >>>>>> >>>>>> ipactl status >>>>>> ... >>>>>> pki-tomcatd Service: STOPPED >>>>>> >>>>>> I can't renew other expired certificates. >>>>>> >>>>>> I also note that there is now no key for ocspSigningCertificate as >>>>>> shown >>>>>> by: >>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>> >>>>>> I wonder if this is because the certificate subject changed? There >>>>>> was a >>>>>> key before the certificate renewed. >>>>>> >>>>>> The ca debug logs are showing: >>>>>> >>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by >>>>>> nickname: >>>>>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108 >>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to >>>>>> x509CertImpl >>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: >>>>>> Certificate >>>>>> object not found >>>>>> Certificate object not found >>>>>>        at com.netscape.ca.SigningUnit.init(SigningUnit.java:184) >>>>>> >>>>>> Any help in repairing my broken ipa system will be much appreciated. >>>>> >>>>> Sorry for the delay. I think my previous reading of the certmonger >>>>> csrgen code was wrong. >>>>> >>>>> IIRC in your certmonger entry the cert_subject has the hostname value >>>>> right? Do you also have cert_subject_der? >>>>> >>>>> You can decode that by: >>>>> >>>>> 1. Running a hex-to-bin, something hacky like this in python: >>>>> >>>>> import binascii >>>>> >>>>> hex_string = "<hex value>" >>>>> >>>>> binary_string = binascii.unhexlify(hex_string) >>>>> >>>>> fd = open("out", "wb") >>>>> fd.write(binary_string) >>>>> fd.close() >>>>> >>>>> 2. Run: openssl asn1parse -in out -inform der >>>>> >>>>> I'm going to assume this also has the hostname encoded in it. >>>> >>>> Hi Again >>>> >>>> Yes, correct. The cert_subject_der does have the bad CN with the >>>> hostname encoded in it. >>>> >>>>> >>>>> Can you try this: >>>>> >>>>> 1. Make a backup of the request file (just in case) > 2. Remove >>>>> cert_subject_der >>>>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM> >>>>> 3. Try the renewal again >>>> >>>> I ran though all of this, checking that the request file was still >>>> what >>>> we wanted after restarting certmonger with verbose logging, restoring >>>> all the databases, fixing permissions, checking keys etc., and ran the >>>> getcert resubmit. >>>> >>>> It still generates a certificate with the bad CN including a hostname. >>>> >>>> Then the pki-tomcat server fails to start again. >>>> >>>> Other things I have checked are: >>>> 1) The csr= field in the (modified) request file has the correct >>>> subject. >>>> >>>> 2) The dogtag ca debug log file is showing: >>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017 >>>> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK >>>> ... >>>> RenewalSubmitter: renewal original profileId=caIPAserviceCert >>>> RenewalSubmitter: for renewal, original authenticator raCertAuth found >>>> CertProcessor: No authenticator credentials required >>>> processRenewal: set original Inputs into profile Context >>>> RenewalSubmitter: setInputsIntoContext() getting input name= >>>> cert_request_type >>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10 >>>> RenewalSubmitter: setInputsIntoContext() getting input name= >>>> cert_request >>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex >>>> encoded csr> >>>> ... >>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM> >>>> >>>> The hex encoded csr in the last line decodes to have the bad subject >>>> where the hostname is one of our other ipa servers. >>>> >>>> The certificate always getthe same hostname in the bad subject cn. >>>> >>>> Do you have any other ideas of things to check to find out what's >>>> generating the bad subject? >>> >>> The order certmonger uses for the subject is: >>> >>> 1. cert_subject_der >>> 2. If there is no DER, try to encode cert_subject >>> 3. If that fails try again a different way >>> 4. If it fails yet again use CN=localhost >>> >>> It is baffling that it would pick a hostname much less one that >>> certmonger shouldn't even know about. >> >> Indeed, and if I try to renew other certificates for some of them it >> chooses other host names that should not be known about. Each >> certificate seems to get the same bad hostname each time I try to renew. >> >>> >>> I assume the CSR found in the dogtag logs matches the csr value in the >>> certmonger request? >> >> No, its different. The issued certificate matches the csr seen in dogtag >> which makes sense. But the csr in the dogtag logs has the bad subject. >> The csr in the request directory file has the good subject. >> >>> >>> Can you share the certmonger request file privately? >> >> Yes, certainly. >> >> Thanks for your continued help on this. > > Let's try this. We'll drop the current request and create a new one. > > Stop tomcat, restore the old cert database, start tomcat, then: > > # getcert stop-tracking -i <request id> > # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n > "ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P <pin> -B > /usr/libexec/ipa/certmonger/stop_pkicad -C > '/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"' > > Then try resubmitting the request. > Hi Rob When you say 'stop tomcat', I'm just doing an ipactl stop. That is stopiing the ipa pki-tomcat service. Is that good enough or are there other services that need stopping too? I tried the stop-tracking/start-tracking. Exactly the same result as before. Same hostname appears in the subject CN field of the new certificate and then the pki-tomcat service won't start. I've also been back and redone the manual submits changing the point at which I copied in the edited request as I wondered if there was some caching of csr's going on in certmonger and it was remembering a previous request it had read on startup before I replaced the request file. But nothing seems to stop dogtag issuing a certificate with the Subject CN being that host name. Is it possible that dogtag is somehow overriding the Subject its being explicitly given? FWIW the CSR in the dogtag debug log is always identical, but I guess thats expected if the CSR information is always the same. I'm not sure if its possible to increase the verbosity on the dogtag side. Can you advise please? I've been assuming that its the bad subject that is preventing the pki-tomcat from starting after the new certificate is issued. Does that make sense to you? I wonder where we can go from here? There must be a good reason for whats happening!

On 25/01/2018 13:43, Rob Crittenden via FreeIPA-users wrote: > Roderick Johnstone via FreeIPA-users wrote: >> On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote: >>> Roderick Johnstone via FreeIPA-users wrote: >>>> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote: >>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote: >>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote: >>>>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote: >>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>>>>>> Hi >>>>>>>>>>>> >>>>>>>>>>>> Our freeipa certificates need to be renewed due to passing >>>>>>>>>>>> their >>>>>>>>>>>> expiry >>>>>>>>>>>> dates. >>>>>>>>>>>> >>>>>>>>>>>> While some certificates have renewed ok, the ipaCert and >>>>>>>>>>>> auditSigningCert are renewing but the new certificates have >>>>>>>>>>>> the >>>>>>>>>>>> wrong >>>>>>>>>>>> Subject. >>>>>>>>>>>> >>>>>>>>>>>> Environment is: >>>>>>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4 >>>>>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4 >>>>>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5 >>>>>>>>>>>> >>>>>>>>>>>> Once there are renewed certificates with the wrong Subject >>>>>>>>>>>> present, >>>>>>>>>>>> there are various problems with renewing the remaining >>>>>>>>>>>> certificates, >>>>>>>>>>>> which I think might be related to the bad Subject: >>>>>>>>>>>> >>>>>>>>>>>> 1) When just ipaCert has the wrong subject no further renewals >>>>>>>>>>>> happen >>>>>>>>>>>> >>>>>>>>>>>> 2) When auditSigningCert has the wrong subject the ipa >>>>>>>>>>>> pki-tomcatd >>>>>>>>>>>> service will not start and no further renewals happen. >>>>>>>>>>>> >>>>>>>>>>>> I've been round the following loop many times on ServerA, our >>>>>>>>>>>> first >>>>>>>>>>>> master: >>>>>>>>>>>> >>>>>>>>>>>> 1) Restore good certificates from backup >>>>>>>>>>>> 2) Put the clock back to a time when certificates are all >>>>>>>>>>>> valid >>>>>>>>>>>> 3) Resubmit certificates for renewal >>>>>>>>>>>> >>>>>>>>>>>> Each time the ipaCert renews it has the same wrong Subject. >>>>>>>>>>>> The >>>>>>>>>>>> wrong >>>>>>>>>>>> Subject includes the host name of one of our ipa client >>>>>>>>>>>> systems. >>>>>>>>>>>> >>>>>>>>>>>> Each time the auditSigningCert renews it has the same wrong >>>>>>>>>>>> Subject >>>>>>>>>>>> but >>>>>>>>>>>> a different subject to the ipaCert. The wrong Subject in this >>>>>>>>>>>> case >>>>>>>>>>>> includes the host name of a system which has never been an ipa >>>>>>>>>>>> client, >>>>>>>>>>>> but might have been added and removed with ipa host-add and >>>>>>>>>>>> ipa >>>>>>>>>>>> host-del >>>>>>>>>>>> for testing something, a while ago. >>>>>>>>>>>> >>>>>>>>>>>> As far as I can see, the "cert_subject" is set correctly in >>>>>>>>>>>> the >>>>>>>>>>>> file >>>>>>>>>>>> /var/lib/certmonger/<request id> until the point at which the >>>>>>>>>>>> certificate is actually renewed. >>>>>>>>>>>> >>>>>>>>>>>> I'd be very grateful for some pointers as to which >>>>>>>>>>>> configuration >>>>>>>>>>>> options >>>>>>>>>>>> and logs to check through to resolve this problem on our >>>>>>>>>>>> production >>>>>>>>>>>> system. >>>>>>>>>>>> >>>>>>>>>>>> If its of any relevance we did change which server is the >>>>>>>>>>>> first >>>>>>>>>>>> master >>>>>>>>>>>> some time ago. >>>>>>>>>>> >>>>>>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to >>>>>>>>>>> see >>>>>>>>>>> what >>>>>>>>>>> the subject is. >>>>>>>>>> >>>>>>>>>> I'm not seeing any obvious CSR fields in the >>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file. >>>>>>>>> >>>>>>>>> foo.bar.certreq= >>>>>>>>> >>>>>>>>>> The CSR in the certmonger requests file for the auditSigningCert >>>>>>>>>> seems >>>>>>>>>> to be showing with the correct Subject. This is different from >>>>>>>>>> the bad >>>>>>>>>> subject showing in the requests file field: >>>>>>>>>> cert_subject= >>>>>>>>> >>>>>>>>> The value of cert_subject comes from the issued certificate. >>>>>>>>> >>>>>>>>>> and the Subject which is showing in the 'getcert list' output >>>>>>>>>> (which is >>>>>>>>>> the same as that in the cert_subject= field.> >>>>>>>>>> I'm not quite sure what this all means. >>>>>>>>> >>>>>>>>> It is displayed from the data within the tracked certmonger >>>>>>>>> request. >>>>>>>>> >>>>>>>>> certmonger logs to syslog so you can check there or you can stop >>>>>>>>> the >>>>>>>>> process and run it manually with: certmonger -n -d 9 2>&1 | tee >>>>>>>>> certmonger.log >>>>>>>>> >>>>>>>>> That will provide a lot of debugging output that may show what is >>>>>>>>> going on. >>>>>>>> >>>>>>>> I've restored certificate databases from backup and put the clock >>>>>>>> back >>>>>>>> to a time when certificates are valid and renewed the ocspSigining >>>>>>>> certificate with: >>>>>>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i >>>>>>>> 20161124081302 >>>>>>>> >>>>>>>> (I've previously tried without the -N with similar results) >>>>>>>> >>>>>>>> What I am seeing in the certmonger logs is: >>>>>>>> >>>>>>>> >>>>>>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert >>>>>>>> cert-pki-ca'. >>>>>>>> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert >>>>>>>> cert-pki-ca' to public key. >>>>>>>> 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert >>>>>>>> cert-pki-ca". >>>>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert >>>>>>>> cert-pki-ca" in token "NSS Certificate DB" in database >>>>>>>> "/etc/pki/pki-tomcat/alias" will not be valid after >>>>>>>> 20171025122401. >>>>>>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert >>>>>>>> cert-pki-ca'. >>>>>>>> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert >>>>>>>> cert-pki-ca' to public key. >>>>>>>> 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert >>>>>>>> cert-pki-ca". >>>>>>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert >>>>>>>> cert-pki-ca'. >>>>>>>> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert >>>>>>>> cert-pki-ca' to public key. >>>>>>>> 2017-10-23 00:05:39 [581] Found a certificate with the same >>>>>>>> nickname but >>>>>>>> different subject, removing certificate "ocspSigningCert >>>>>>>> cert-pki-ca" >>>>>>>> with subject "CN=OCSP Subsystem,O=<REALM>". >>>>>>>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert >>>>>>>> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca". >>>>>>>> 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert >>>>>>>> cert-pki-ca". >>>>>>>> 2017-10-23 00:05:39 [48576] Adding hook >>>>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert >>>>>>>> cert-pki-ca"" (0). >>>>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert >>>>>>>> cert-pki-ca" in token "NSS Certificate DB" in database >>>>>>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved. >>>>>>>> >>>>>>>> I now have a date valid ocspSigningCertificate, but with the wrong >>>>>>>> subject, and a broken certificate system which will no longer >>>>>>>> start. >>>>>>>> >>>>>>>> ipactl status >>>>>>>> ... >>>>>>>> pki-tomcatd Service: STOPPED >>>>>>>> >>>>>>>> I can't renew other expired certificates. >>>>>>>> >>>>>>>> I also note that there is now no key for ocspSigningCertificate as >>>>>>>> shown >>>>>>>> by: >>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>> >>>>>>>> I wonder if this is because the certificate subject changed? There >>>>>>>> was a >>>>>>>> key before the certificate renewed. >>>>>>>> >>>>>>>> The ca debug logs are showing: >>>>>>>> >>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by >>>>>>>> nickname: >>>>>>>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108 >>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to >>>>>>>> x509CertImpl >>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: >>>>>>>> Certificate >>>>>>>> object not found >>>>>>>> Certificate object not found >>>>>>>>         at com.netscape.ca.SigningUnit.init(SigningUnit.java:184) >>>>>>>> >>>>>>>> Any help in repairing my broken ipa system will be much >>>>>>>> appreciated. >>>>>>> >>>>>>> Sorry for the delay. I think my previous reading of the certmonger >>>>>>> csrgen code was wrong. >>>>>>> >>>>>>> IIRC in your certmonger entry the cert_subject has the hostname >>>>>>> value >>>>>>> right? Do you also have cert_subject_der? >>>>>>> >>>>>>> You can decode that by: >>>>>>> >>>>>>> 1. Running a hex-to-bin, something hacky like this in python: >>>>>>> >>>>>>> import binascii >>>>>>> >>>>>>> hex_string = "<hex value>" >>>>>>> >>>>>>> binary_string = binascii.unhexlify(hex_string) >>>>>>> >>>>>>> fd = open("out", "wb") >>>>>>> fd.write(binary_string) >>>>>>> fd.close() >>>>>>> >>>>>>> 2. Run: openssl asn1parse -in out -inform der >>>>>>> >>>>>>> I'm going to assume this also has the hostname encoded in it. >>>>>> >>>>>> Hi Again >>>>>> >>>>>> Yes, correct. The cert_subject_der does have the bad CN with the >>>>>> hostname encoded in it. >>>>>> >>>>>>> >>>>>>> Can you try this: >>>>>>> >>>>>>> 1. Make a backup of the request file (just in case) > 2. Remove >>>>>>> cert_subject_der >>>>>>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM> >>>>>>> 3. Try the renewal again >>>>>> >>>>>> I ran though all of this, checking that the request file was still >>>>>> what >>>>>> we wanted after restarting certmonger with verbose logging, >>>>>> restoring >>>>>> all the databases, fixing permissions, checking keys etc., and >>>>>> ran the >>>>>> getcert resubmit. >>>>>> >>>>>> It still generates a certificate with the bad CN including a >>>>>> hostname. >>>>>> >>>>>> Then the pki-tomcat server fails to start again. >>>>>> >>>>>> Other things I have checked are: >>>>>> 1) The csr= field in the (modified) request file has the correct >>>>>> subject. >>>>>> >>>>>> 2) The dogtag ca debug log file is showing: >>>>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017 >>>>>> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK >>>>>> ... >>>>>> RenewalSubmitter: renewal original profileId=caIPAserviceCert >>>>>> RenewalSubmitter: for renewal, original authenticator raCertAuth >>>>>> found >>>>>> CertProcessor: No authenticator credentials required >>>>>> processRenewal: set original Inputs into profile Context >>>>>> RenewalSubmitter: setInputsIntoContext() getting input name= >>>>>> cert_request_type >>>>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10 >>>>>> RenewalSubmitter: setInputsIntoContext() getting input name= >>>>>> cert_request >>>>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex >>>>>> encoded csr> >>>>>> ... >>>>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM> >>>>>> >>>>>> The hex encoded csr in the last line decodes to have the bad subject >>>>>> where the hostname is one of our other ipa servers. >>>>>> >>>>>> The certificate always getthe same hostname in the bad subject cn. >>>>>> >>>>>> Do you have any other ideas of things to check to find out what's >>>>>> generating the bad subject? >>>>> >>>>> The order certmonger uses for the subject is: >>>>> >>>>> 1. cert_subject_der >>>>> 2. If there is no DER, try to encode cert_subject >>>>> 3. If that fails try again a different way >>>>> 4. If it fails yet again use CN=localhost >>>>> >>>>> It is baffling that it would pick a hostname much less one that >>>>> certmonger shouldn't even know about. >>>> >>>> Indeed, and if I try to renew other certificates for some of them it >>>> chooses other host names that should not be known about. Each >>>> certificate seems to get the same bad hostname each time I try to >>>> renew. >>>> >>>>> >>>>> I assume the CSR found in the dogtag logs matches the csr value in >>>>> the >>>>> certmonger request? >>>> >>>> No, its different. The issued certificate matches the csr seen in >>>> dogtag >>>> which makes sense. But the csr in the dogtag logs has the bad subject. >>>> The csr in the request directory file has the good subject. >>>> >>>>> >>>>> Can you share the certmonger request file privately? >>>> >>>> Yes, certainly. >>>> >>>> Thanks for your continued help on this. >>> >>> Let's try this. We'll drop the current request and create a new one. >>> >>> Stop tomcat, restore the old cert database, start tomcat, then: >>> >>> # getcert stop-tracking -i <request id> >>> # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n >>> "ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P <pin> -B >>> /usr/libexec/ipa/certmonger/stop_pkicad -C >>> '/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert >>> cert-pki-ca"' >>> >>> Then try resubmitting the request. >>> >> Hi Rob >> >> When you say 'stop tomcat', I'm just doing an ipactl stop. That is >> stopiing the ipa pki-tomcat service. Is that good enough or are there >> other services that need stopping too? >> >> I tried the stop-tracking/start-tracking. Exactly the same result as >> before. Same hostname appears in the subject CN field of the new >> certificate and then the pki-tomcat service won't start. >> >> I've also been back and redone the manual submits changing the point at >> which I copied in the edited request as I wondered if there was some >> caching of csr's going on in certmonger and it was remembering a >> previous request it had read on startup before I replaced the request >> file. >> >> But nothing seems to stop dogtag issuing a certificate with the Subject >> CN being that host name. >> >> Is it possible that dogtag is somehow overriding the Subject its being >> explicitly given? >> >> FWIW the CSR in the dogtag debug log is always identical, but I guess >> thats expected if the CSR information is always the same. >> >> I'm not sure if its possible to increase the verbosity on the dogtag >> side. Can you advise please? >> >> I've been assuming that its the bad subject that is preventing the >> pki-tomcat from starting after the new certificate is issued. Does that >> make sense to you? >> >> I wonder where we can go from here? There must be a good reason for >> whats happening! > > Let's step back and gather some more information. > > Can you restore the files again and send me the output of: > > # getcert list > > #  certutil -L -d /etc/pki/pki-tomcat/alias/ -n "ocspSigningCert > cert-pki-ca" Hi Rob You should have the output you requested by private email now. > > And the exact commands you are using to do all this, including > re-issuing the cert Thats a great idea. Maybe I'm just doing something in the wrong order. > > Using ipactl is fine. You just don't want to touch the NSS databases > while the service is running. Similarly you probably don't want to mess > with certmonger requests while it is running as it won't see the changes > until it restarts. Here is the procedure I'm using to renew the certificates. One of the tricky things is to not have the certificate system working when certmonger is started so that it doesn't renew a certificate on startup that might not be one I have changed the request file for (other certificates have similar problems with bad subject CN and stop the certificate system.) I've found it convenient to keep reinitializing the log files because they get very confusing when the clock keeps getting set back. The request number has changed since we did the stop-tracking / start-tracking. Roderick ipactl stop # Set clock back: systemctl stop ntpd timedatectl set-time "2017-10-23 00:00:00" date #Stop certificate renewals the systemd way systemctl stop certmonger # Sort out certmonger log mv /var/log/certmonger.log /var/log/certmonger.log-$(date +%Y%m%d:%H:%M) # ****** Copy in the fixed certmonger request \cp /root/20161124081302.ocsp_backup /var/lib/certmonger/requests/20161124081302 # In another window certmonger -n -d 9 2>&1 | tee /var/log/certmonger.log # In the original window # Sort out dogtag log mv /var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/debug-$(date +%Y%m%d:%H:%M).log touch /var/log/pki/pki-tomcat/ca/debug chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/debug mv /var/log/pki/pki-tomcat/ca/selftests.log /var/log/pki/pki-tomcat/ca/selftests.log-$(date +%Y%m%d:%H:%M) touch /var/log/pki/pki-tomcat/ca/selftests.log chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/selftests.log \cp /var/tmp/rmj/etc/httpd/alias/cert8.db /etc/httpd/alias/cert8.db \cp /var/tmp/rmj/etc/httpd/alias/key3.db /etc/httpd/alias/key3.db \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/cert8.db /etc/pki/pki-tomcat/alias/cert8.db \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/key3.db /etc/pki/pki-tomcat/alias/key3.db \cp /var/tmp/rmj/etc/pki/pki-tomcat/ca/CS.cfg /etc/pki/pki-tomcat/ca/CS.cfg # Fix up certificate permissions and check all is ok certutil -M -n "caSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -t CTu,Cu,Cu certutil -L -d /etc/pki/pki-tomcat/alias certutil -K -d /etc/pki/pki-tomcat/alias <pin> # Fix dogtag so it starts pki-server subsystem-enable ca > # Start IdM services ipactl start ipactl status #Resubmit certificate ocsp date;getcert resubmit  -i 20161124081302

Roderick Johnstone via FreeIPA-users wrote: > On 25/01/2018 16:56, Roderick Johnstone via FreeIPA-users wrote: >> On 25/01/2018 13:43, Rob Crittenden via FreeIPA-users wrote: >>> Roderick Johnstone via FreeIPA-users wrote: >>>> On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote: >>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote: >>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote: >>>>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote: >>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote: >>>>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>>>>>>>> Hi >>>>>>>>>>>>>> >>>>>>>>>>>>>> Our freeipa certificates need to be renewed due to passing >>>>>>>>>>>>>> their >>>>>>>>>>>>>> expiry >>>>>>>>>>>>>> dates. >>>>>>>>>>>>>> >>>>>>>>>>>>>> While some certificates have renewed ok, the ipaCert and >>>>>>>>>>>>>> auditSigningCert are renewing but the new certificates >>>>>>>>>>>>>> have the >>>>>>>>>>>>>> wrong >>>>>>>>>>>>>> Subject. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Environment is: >>>>>>>>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4 >>>>>>>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4 >>>>>>>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Once there are renewed certificates with the wrong Subject >>>>>>>>>>>>>> present, >>>>>>>>>>>>>> there are various problems with renewing the remaining >>>>>>>>>>>>>> certificates, >>>>>>>>>>>>>> which I think might be related to the bad Subject: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) When just ipaCert has the wrong subject no further >>>>>>>>>>>>>> renewals >>>>>>>>>>>>>> happen >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) When auditSigningCert has the wrong subject the ipa >>>>>>>>>>>>>> pki-tomcatd >>>>>>>>>>>>>> service will not start and no further renewals happen. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I've been round the following loop many times on ServerA, our >>>>>>>>>>>>>> first >>>>>>>>>>>>>> master: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) Restore good certificates from backup >>>>>>>>>>>>>> 2) Put the clock back to a time when certificates are all >>>>>>>>>>>>>> valid >>>>>>>>>>>>>> 3) Resubmit certificates for renewal >>>>>>>>>>>>>> >>>>>>>>>>>>>> Each time the ipaCert renews it has the same wrong >>>>>>>>>>>>>> Subject. The >>>>>>>>>>>>>> wrong >>>>>>>>>>>>>> Subject includes the host name of one of our ipa client >>>>>>>>>>>>>> systems. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Each time the auditSigningCert renews it has the same wrong >>>>>>>>>>>>>> Subject >>>>>>>>>>>>>> but >>>>>>>>>>>>>> a different subject to the ipaCert. The wrong Subject in this >>>>>>>>>>>>>> case >>>>>>>>>>>>>> includes the host name of a system which has never been an >>>>>>>>>>>>>> ipa >>>>>>>>>>>>>> client, >>>>>>>>>>>>>> but might have been added and removed with ipa host-add >>>>>>>>>>>>>> and ipa >>>>>>>>>>>>>> host-del >>>>>>>>>>>>>> for testing something, a while ago. >>>>>>>>>>>>>> >>>>>>>>>>>>>> As far as I can see, the "cert_subject" is set correctly >>>>>>>>>>>>>> in the >>>>>>>>>>>>>> file >>>>>>>>>>>>>> /var/lib/certmonger/<request id> until the point at which the >>>>>>>>>>>>>> certificate is actually renewed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'd be very grateful for some pointers as to which >>>>>>>>>>>>>> configuration >>>>>>>>>>>>>> options >>>>>>>>>>>>>> and logs to check through to resolve this problem on our >>>>>>>>>>>>>> production >>>>>>>>>>>>>> system. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If its of any relevance we did change which server is the >>>>>>>>>>>>>> first >>>>>>>>>>>>>> master >>>>>>>>>>>>>> some time ago. >>>>>>>>>>>>> >>>>>>>>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger >>>>>>>>>>>>> to see >>>>>>>>>>>>> what >>>>>>>>>>>>> the subject is. >>>>>>>>>>>> >>>>>>>>>>>> I'm not seeing any obvious CSR fields in the >>>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file. >>>>>>>>>>> >>>>>>>>>>> foo.bar.certreq= >>>>>>>>>>> >>>>>>>>>>>> The CSR in the certmonger requests file for the >>>>>>>>>>>> auditSigningCert >>>>>>>>>>>> seems >>>>>>>>>>>> to be showing with the correct Subject. This is different from >>>>>>>>>>>> the bad >>>>>>>>>>>> subject showing in the requests file field: >>>>>>>>>>>> cert_subject= >>>>>>>>>>> >>>>>>>>>>> The value of cert_subject comes from the issued certificate. >>>>>>>>>>> >>>>>>>>>>>> and the Subject which is showing in the 'getcert list' output >>>>>>>>>>>> (which is >>>>>>>>>>>> the same as that in the cert_subject= field.> >>>>>>>>>>>> I'm not quite sure what this all means. >>>>>>>>>>> >>>>>>>>>>> It is displayed from the data within the tracked certmonger >>>>>>>>>>> request. >>>>>>>>>>> >>>>>>>>>>> certmonger logs to syslog so you can check there or you can stop >>>>>>>>>>> the >>>>>>>>>>> process and run it manually with: certmonger -n -d 9 2>&1 | tee >>>>>>>>>>> certmonger.log >>>>>>>>>>> >>>>>>>>>>> That will provide a lot of debugging output that may show >>>>>>>>>>> what is >>>>>>>>>>> going on. >>>>>>>>>> >>>>>>>>>> I've restored certificate databases from backup and put the clock >>>>>>>>>> back >>>>>>>>>> to a time when certificates are valid and renewed the >>>>>>>>>> ocspSigining >>>>>>>>>> certificate with: >>>>>>>>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i >>>>>>>>>> 20161124081302 >>>>>>>>>> >>>>>>>>>> (I've previously tried without the -N with similar results) >>>>>>>>>> >>>>>>>>>> What I am seeing in the certmonger logs is: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert >>>>>>>>>> cert-pki-ca'. >>>>>>>>>> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert >>>>>>>>>> cert-pki-ca' to public key. >>>>>>>>>> 2017-10-23 00:05:28 [439] Located the certificate >>>>>>>>>> "ocspSigningCert >>>>>>>>>> cert-pki-ca". >>>>>>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert >>>>>>>>>> cert-pki-ca" in token "NSS Certificate DB" in database >>>>>>>>>> "/etc/pki/pki-tomcat/alias" will not be valid after >>>>>>>>>> 20171025122401. >>>>>>>>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert >>>>>>>>>> cert-pki-ca'. >>>>>>>>>> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert >>>>>>>>>> cert-pki-ca' to public key. >>>>>>>>>> 2017-10-23 00:05:28 [443] Located the certificate >>>>>>>>>> "ocspSigningCert >>>>>>>>>> cert-pki-ca". >>>>>>>>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert >>>>>>>>>> cert-pki-ca'. >>>>>>>>>> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert >>>>>>>>>> cert-pki-ca' to public key. >>>>>>>>>> 2017-10-23 00:05:39 [581] Found a certificate with the same >>>>>>>>>> nickname but >>>>>>>>>> different subject, removing certificate "ocspSigningCert >>>>>>>>>> cert-pki-ca" >>>>>>>>>> with subject "CN=OCSP Subsystem,O=<REALM>". >>>>>>>>>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert >>>>>>>>>> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca". >>>>>>>>>> 2017-10-23 00:05:39 [583] Located the certificate >>>>>>>>>> "ocspSigningCert >>>>>>>>>> cert-pki-ca". >>>>>>>>>> 2017-10-23 00:05:39 [48576] Adding hook >>>>>>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert >>>>>>>>>> cert-pki-ca"" (0). >>>>>>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert >>>>>>>>>> cert-pki-ca" in token "NSS Certificate DB" in database >>>>>>>>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved. >>>>>>>>>> >>>>>>>>>> I now have a date valid ocspSigningCertificate, but with the >>>>>>>>>> wrong >>>>>>>>>> subject, and a broken certificate system which will no longer >>>>>>>>>> start. >>>>>>>>>> >>>>>>>>>> ipactl status >>>>>>>>>> ... >>>>>>>>>> pki-tomcatd Service: STOPPED >>>>>>>>>> >>>>>>>>>> I can't renew other expired certificates. >>>>>>>>>> >>>>>>>>>> I also note that there is now no key for >>>>>>>>>> ocspSigningCertificate as >>>>>>>>>> shown >>>>>>>>>> by: >>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>>>> >>>>>>>>>> I wonder if this is because the certificate subject changed? >>>>>>>>>> There >>>>>>>>>> was a >>>>>>>>>> key before the certificate renewed. >>>>>>>>>> >>>>>>>>>> The ca debug logs are showing: >>>>>>>>>> >>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by >>>>>>>>>> nickname: >>>>>>>>>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108 >>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to >>>>>>>>>> x509CertImpl >>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: >>>>>>>>>> Certificate >>>>>>>>>> object not found >>>>>>>>>> Certificate object not found >>>>>>>>>>         at com.netscape.ca.SigningUnit.init(SigningUnit.java:184) >>>>>>>>>> >>>>>>>>>> Any help in repairing my broken ipa system will be much >>>>>>>>>> appreciated. >>>>>>>>> >>>>>>>>> Sorry for the delay. I think my previous reading of the certmonger >>>>>>>>> csrgen code was wrong. >>>>>>>>> >>>>>>>>> IIRC in your certmonger entry the cert_subject has the hostname >>>>>>>>> value >>>>>>>>> right? Do you also have cert_subject_der? >>>>>>>>> >>>>>>>>> You can decode that by: >>>>>>>>> >>>>>>>>> 1. Running a hex-to-bin, something hacky like this in python: >>>>>>>>> >>>>>>>>> import binascii >>>>>>>>> >>>>>>>>> hex_string = "<hex value>" >>>>>>>>> >>>>>>>>> binary_string = binascii.unhexlify(hex_string) >>>>>>>>> >>>>>>>>> fd = open("out", "wb") >>>>>>>>> fd.write(binary_string) >>>>>>>>> fd.close() >>>>>>>>> >>>>>>>>> 2. Run: openssl asn1parse -in out -inform der >>>>>>>>> >>>>>>>>> I'm going to assume this also has the hostname encoded in it. >>>>>>>> >>>>>>>> Hi Again >>>>>>>> >>>>>>>> Yes, correct. The cert_subject_der does have the bad CN with the >>>>>>>> hostname encoded in it. >>>>>>>> >>>>>>>>> >>>>>>>>> Can you try this: >>>>>>>>> >>>>>>>>> 1. Make a backup of the request file (just in case) > 2. Remove >>>>>>>>> cert_subject_der >>>>>>>>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM> >>>>>>>>> 3. Try the renewal again >>>>>>>> >>>>>>>> I ran though all of this, checking that the request file was still >>>>>>>> what >>>>>>>> we wanted after restarting certmonger with verbose logging, >>>>>>>> restoring >>>>>>>> all the databases, fixing permissions, checking keys etc., and >>>>>>>> ran the >>>>>>>> getcert resubmit. >>>>>>>> >>>>>>>> It still generates a certificate with the bad CN including a >>>>>>>> hostname. >>>>>>>> >>>>>>>> Then the pki-tomcat server fails to start again. >>>>>>>> >>>>>>>> Other things I have checked are: >>>>>>>> 1) The csr= field in the (modified) request file has the correct >>>>>>>> subject. >>>>>>>> >>>>>>>> 2) The dogtag ca debug log file is showing: >>>>>>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017 >>>>>>>> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK >>>>>>>> ... >>>>>>>> RenewalSubmitter: renewal original profileId=caIPAserviceCert >>>>>>>> RenewalSubmitter: for renewal, original authenticator raCertAuth >>>>>>>> found >>>>>>>> CertProcessor: No authenticator credentials required >>>>>>>> processRenewal: set original Inputs into profile Context >>>>>>>> RenewalSubmitter: setInputsIntoContext() getting input name= >>>>>>>> cert_request_type >>>>>>>> RenewalSubmitter: setInputsIntoContext() setting value in >>>>>>>> ctx:pkcs10 >>>>>>>> RenewalSubmitter: setInputsIntoContext() getting input name= >>>>>>>> cert_request >>>>>>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex >>>>>>>> encoded csr> >>>>>>>> ... >>>>>>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM> >>>>>>>> >>>>>>>> The hex encoded csr in the last line decodes to have the bad >>>>>>>> subject >>>>>>>> where the hostname is one of our other ipa servers. >>>>>>>> >>>>>>>> The certificate always getthe same hostname in the bad subject cn. >>>>>>>> >>>>>>>> Do you have any other ideas of things to check to find out what's >>>>>>>> generating the bad subject? >>>>>>> >>>>>>> The order certmonger uses for the subject is: >>>>>>> >>>>>>> 1. cert_subject_der >>>>>>> 2. If there is no DER, try to encode cert_subject >>>>>>> 3. If that fails try again a different way >>>>>>> 4. If it fails yet again use CN=localhost >>>>>>> >>>>>>> It is baffling that it would pick a hostname much less one that >>>>>>> certmonger shouldn't even know about. >>>>>> >>>>>> Indeed, and if I try to renew other certificates for some of them it >>>>>> chooses other host names that should not be known about. Each >>>>>> certificate seems to get the same bad hostname each time I try to >>>>>> renew. >>>>>> >>>>>>> >>>>>>> I assume the CSR found in the dogtag logs matches the csr value >>>>>>> in the >>>>>>> certmonger request? >>>>>> >>>>>> No, its different. The issued certificate matches the csr seen in >>>>>> dogtag >>>>>> which makes sense. But the csr in the dogtag logs has the bad >>>>>> subject. >>>>>> The csr in the request directory file has the good subject. >>>>>> >>>>>>> >>>>>>> Can you share the certmonger request file privately? >>>>>> >>>>>> Yes, certainly. >>>>>> >>>>>> Thanks for your continued help on this. >>>>> >>>>> Let's try this. We'll drop the current request and create a new one. >>>>> >>>>> Stop tomcat, restore the old cert database, start tomcat, then: >>>>> >>>>> # getcert stop-tracking -i <request id> >>>>> # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n >>>>> "ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P <pin> -B >>>>> /usr/libexec/ipa/certmonger/stop_pkicad -C >>>>> '/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert >>>>> cert-pki-ca"' >>>>> >>>>> Then try resubmitting the request. >>>>> >>>> Hi Rob >>>> >>>> When you say 'stop tomcat', I'm just doing an ipactl stop. That is >>>> stopiing the ipa pki-tomcat service. Is that good enough or are there >>>> other services that need stopping too? >>>> >>>> I tried the stop-tracking/start-tracking. Exactly the same result as >>>> before. Same hostname appears in the subject CN field of the new >>>> certificate and then the pki-tomcat service won't start. >>>> >>>> I've also been back and redone the manual submits changing the point at >>>> which I copied in the edited request as I wondered if there was some >>>> caching of csr's going on in certmonger and it was remembering a >>>> previous request it had read on startup before I replaced the >>>> request file. >>>> >>>> But nothing seems to stop dogtag issuing a certificate with the Subject >>>> CN being that host name. >>>> >>>> Is it possible that dogtag is somehow overriding the Subject its being >>>> explicitly given? >>>> >>>> FWIW the CSR in the dogtag debug log is always identical, but I guess >>>> thats expected if the CSR information is always the same. >>>> >>>> I'm not sure if its possible to increase the verbosity on the dogtag >>>> side. Can you advise please? >>>> >>>> I've been assuming that its the bad subject that is preventing the >>>> pki-tomcat from starting after the new certificate is issued. Does that >>>> make sense to you? >>>> >>>> I wonder where we can go from here? There must be a good reason for >>>> whats happening! >>> >>> Let's step back and gather some more information. >>> >>> Can you restore the files again and send me the output of: >>> >>> # getcert list >>> >>> #  certutil -L -d /etc/pki/pki-tomcat/alias/ -n "ocspSigningCert >>> cert-pki-ca" >> >> Hi Rob >> >> You should have the output you requested by private email now. >> >>> >>> And the exact commands you are using to do all this, including >>> re-issuing the cert >> >> Thats a great idea. Maybe I'm just doing something in the wrong order. >> >>> >>> Using ipactl is fine. You just don't want to touch the NSS databases >>> while the service is running. Similarly you probably don't want to mess >>> with certmonger requests while it is running as it won't see the changes >>> until it restarts. >> Here is the procedure I'm using to renew the certificates. One of the >> tricky things is to not have the certificate system working when >> certmonger is started so that it doesn't renew a certificate on >> startup that might not be one I have changed the request file for >> (other certificates have similar problems with bad subject CN and stop >> the certificate system.) >> >> I've found it convenient to keep reinitializing the log files because >> they get very confusing when the clock keeps getting set back. >> >> The request number has changed since we did the stop-tracking / >> start-tracking. >> >> Roderick >> >> ipactl stop >> >> # Set clock back: >> systemctl stop ntpd >> timedatectl set-time "2017-10-23 00:00:00" >> date >> >> #Stop certificate renewals the systemd way >> systemctl stop certmonger >> >> # Sort out certmonger log >> mv /var/log/certmonger.log /var/log/certmonger.log-$(date +%Y%m%d:%H:%M) >> >> # ****** Copy in the fixed certmonger request >> \cp /root/20161124081302.ocsp_backup >> /var/lib/certmonger/requests/20161124081302 >> >> >> # In another window >> certmonger -n -d 9 2>&1 | tee /var/log/certmonger.log >> >> # In the original window >> # Sort out dogtag log >> mv /var/log/pki/pki-tomcat/ca/debug >> /var/log/pki/pki-tomcat/ca/debug-$(date +%Y%m%d:%H:%M).log >> touch /var/log/pki/pki-tomcat/ca/debug >> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/debug >> >> mv /var/log/pki/pki-tomcat/ca/selftests.log >> /var/log/pki/pki-tomcat/ca/selftests.log-$(date +%Y%m%d:%H:%M) >> touch /var/log/pki/pki-tomcat/ca/selftests.log >> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/selftests.log >> >> \cp /var/tmp/rmj/etc/httpd/alias/cert8.db /etc/httpd/alias/cert8.db >> \cp /var/tmp/rmj/etc/httpd/alias/key3.db /etc/httpd/alias/key3.db >> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/cert8.db >> /etc/pki/pki-tomcat/alias/cert8.db >> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/key3.db >> /etc/pki/pki-tomcat/alias/key3.db >> >> \cp /var/tmp/rmj/etc/pki/pki-tomcat/ca/CS.cfg >> /etc/pki/pki-tomcat/ca/CS.cfg >> >> # Fix up certificate permissions and check all is ok >> certutil -M -n "caSigningCert cert-pki-ca" -d >> /etc/pki/pki-tomcat/alias -t CTu,Cu,Cu >> certutil -L -d /etc/pki/pki-tomcat/alias >> certutil -K -d /etc/pki/pki-tomcat/alias >> <pin> >> >> # Fix dogtag so it starts >> pki-server subsystem-enable ca > >> # Start IdM services >> >> ipactl start >> ipactl status >> >> #Resubmit certificate ocsp >> date;getcert resubmit  -i 20161124081302 > > Hi Rob > > Does it look like there is anything I should be doing differently in > this command sequence to try to get the certificate renewed correctly? > > I'm starting to wonder whether I should look at alternative strategies > for recovering the freeipa servers if the certificates won't renew > correctly. Do you have any ideas on this please? > This looks reasonable. Have you tried stop-tracking and start-tracking I suggested?

I'm still baffled how the name(s) are getting mixed up.

Something else you might want to try would be to move all the other requests out of the way in case there is some sort of memory corruption causing the hostname to get in there somehow.

Roderick Johnstone wrote: > On 31/01/2018 20:36, Rob Crittenden via FreeIPA-users wrote: >> Roderick Johnstone via FreeIPA-users wrote: >>> On 25/01/2018 16:56, Roderick Johnstone via FreeIPA-users wrote: >>>> On 25/01/2018 13:43, Rob Crittenden via FreeIPA-users wrote: >>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>> On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote: >>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote: >>>>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote: >>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>>>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote: >>>>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>>>>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote: >>>>>>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote: >>>>>>>>>>>>>>>> Hi >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Our freeipa certificates need to be renewed due to passing >>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>> expiry >>>>>>>>>>>>>>>> dates. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> While some certificates have renewed ok, the ipaCert and >>>>>>>>>>>>>>>> auditSigningCert are renewing but the new certificates >>>>>>>>>>>>>>>> have the >>>>>>>>>>>>>>>> wrong >>>>>>>>>>>>>>>> Subject. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Environment is: >>>>>>>>>>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4 >>>>>>>>>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4 >>>>>>>>>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Once there are renewed certificates with the wrong Subject >>>>>>>>>>>>>>>> present, >>>>>>>>>>>>>>>> there are various problems with renewing the remaining >>>>>>>>>>>>>>>> certificates, >>>>>>>>>>>>>>>> which I think might be related to the bad Subject: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1) When just ipaCert has the wrong subject no further >>>>>>>>>>>>>>>> renewals >>>>>>>>>>>>>>>> happen >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2) When auditSigningCert has the wrong subject the ipa >>>>>>>>>>>>>>>> pki-tomcatd >>>>>>>>>>>>>>>> service will not start and no further renewals happen. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I've been round the following loop many times on >>>>>>>>>>>>>>>> ServerA, our >>>>>>>>>>>>>>>> first >>>>>>>>>>>>>>>> master: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1) Restore good certificates from backup >>>>>>>>>>>>>>>> 2) Put the clock back to a time when certificates are all >>>>>>>>>>>>>>>> valid >>>>>>>>>>>>>>>> 3) Resubmit certificates for renewal >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Each time the ipaCert renews it has the same wrong >>>>>>>>>>>>>>>> Subject. The >>>>>>>>>>>>>>>> wrong >>>>>>>>>>>>>>>> Subject includes the host name of one of our ipa client >>>>>>>>>>>>>>>> systems. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Each time the auditSigningCert renews it has the same wrong >>>>>>>>>>>>>>>> Subject >>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>> a different subject to the ipaCert. The wrong Subject in >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>> case >>>>>>>>>>>>>>>> includes the host name of a system which has never been an >>>>>>>>>>>>>>>> ipa >>>>>>>>>>>>>>>> client, >>>>>>>>>>>>>>>> but might have been added and removed with ipa host-add >>>>>>>>>>>>>>>> and ipa >>>>>>>>>>>>>>>> host-del >>>>>>>>>>>>>>>> for testing something, a while ago. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As far as I can see, the "cert_subject" is set correctly >>>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>>> file >>>>>>>>>>>>>>>> /var/lib/certmonger/<request id> until the point at >>>>>>>>>>>>>>>> which the >>>>>>>>>>>>>>>> certificate is actually renewed. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'd be very grateful for some pointers as to which >>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>> options >>>>>>>>>>>>>>>> and logs to check through to resolve this problem on our >>>>>>>>>>>>>>>> production >>>>>>>>>>>>>>>> system. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If its of any relevance we did change which server is the >>>>>>>>>>>>>>>> first >>>>>>>>>>>>>>>> master >>>>>>>>>>>>>>>> some time ago. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger >>>>>>>>>>>>>>> to see >>>>>>>>>>>>>>> what >>>>>>>>>>>>>>> the subject is. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm not seeing any obvious CSR fields in the >>>>>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file. >>>>>>>>>>>>> >>>>>>>>>>>>> foo.bar.certreq= >>>>>>>>>>>>> >>>>>>>>>>>>>> The CSR in the certmonger requests file for the >>>>>>>>>>>>>> auditSigningCert >>>>>>>>>>>>>> seems >>>>>>>>>>>>>> to be showing with the correct Subject. This is different >>>>>>>>>>>>>> from >>>>>>>>>>>>>> the bad >>>>>>>>>>>>>> subject showing in the requests file field: >>>>>>>>>>>>>> cert_subject= >>>>>>>>>>>>> >>>>>>>>>>>>> The value of cert_subject comes from the issued certificate. >>>>>>>>>>>>> >>>>>>>>>>>>>> and the Subject which is showing in the 'getcert list' output >>>>>>>>>>>>>> (which is >>>>>>>>>>>>>> the same as that in the cert_subject= field.> >>>>>>>>>>>>>> I'm not quite sure what this all means. >>>>>>>>>>>>> >>>>>>>>>>>>> It is displayed from the data within the tracked certmonger >>>>>>>>>>>>> request. >>>>>>>>>>>>> >>>>>>>>>>>>> certmonger logs to syslog so you can check there or you can >>>>>>>>>>>>> stop >>>>>>>>>>>>> the >>>>>>>>>>>>> process and run it manually with: certmonger -n -d 9 2>&1 | >>>>>>>>>>>>> tee >>>>>>>>>>>>> certmonger.log >>>>>>>>>>>>> >>>>>>>>>>>>> That will provide a lot of debugging output that may show >>>>>>>>>>>>> what is >>>>>>>>>>>>> going on. >>>>>>>>>>>> >>>>>>>>>>>> I've restored certificate databases from backup and put the >>>>>>>>>>>> clock >>>>>>>>>>>> back >>>>>>>>>>>> to a time when certificates are valid and renewed the >>>>>>>>>>>> ocspSigining >>>>>>>>>>>> certificate with: >>>>>>>>>>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i >>>>>>>>>>>> 20161124081302 >>>>>>>>>>>> >>>>>>>>>>>> (I've previously tried without the -N with similar results) >>>>>>>>>>>> >>>>>>>>>>>> What I am seeing in the certmonger logs is: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert >>>>>>>>>>>> cert-pki-ca'. >>>>>>>>>>>> 2017-10-23 00:05:28 [438] Converted private key >>>>>>>>>>>> 'ocspSigningCert >>>>>>>>>>>> cert-pki-ca' to public key. >>>>>>>>>>>> 2017-10-23 00:05:28 [439] Located the certificate >>>>>>>>>>>> "ocspSigningCert >>>>>>>>>>>> cert-pki-ca". >>>>>>>>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named >>>>>>>>>>>> "ocspSigningCert >>>>>>>>>>>> cert-pki-ca" in token "NSS Certificate DB" in database >>>>>>>>>>>> "/etc/pki/pki-tomcat/alias" will not be valid after >>>>>>>>>>>> 20171025122401. >>>>>>>>>>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert >>>>>>>>>>>> cert-pki-ca'. >>>>>>>>>>>> 2017-10-23 00:05:28 [442] Converted private key >>>>>>>>>>>> 'ocspSigningCert >>>>>>>>>>>> cert-pki-ca' to public key. >>>>>>>>>>>> 2017-10-23 00:05:28 [443] Located the certificate >>>>>>>>>>>> "ocspSigningCert >>>>>>>>>>>> cert-pki-ca". >>>>>>>>>>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert >>>>>>>>>>>> cert-pki-ca'. >>>>>>>>>>>> 2017-10-23 00:05:28 [444] Converted private key >>>>>>>>>>>> 'ocspSigningCert >>>>>>>>>>>> cert-pki-ca' to public key. >>>>>>>>>>>> 2017-10-23 00:05:39 [581] Found a certificate with the same >>>>>>>>>>>> nickname but >>>>>>>>>>>> different subject, removing certificate "ocspSigningCert >>>>>>>>>>>> cert-pki-ca" >>>>>>>>>>>> with subject "CN=OCSP Subsystem,O=<REALM>". >>>>>>>>>>>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert >>>>>>>>>>>> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca". >>>>>>>>>>>> 2017-10-23 00:05:39 [583] Located the certificate >>>>>>>>>>>> "ocspSigningCert >>>>>>>>>>>> cert-pki-ca". >>>>>>>>>>>> 2017-10-23 00:05:39 [48576] Adding hook >>>>>>>>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert >>>>>>>>>>>> cert-pki-ca"" (0). >>>>>>>>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named >>>>>>>>>>>> "ocspSigningCert >>>>>>>>>>>> cert-pki-ca" in token "NSS Certificate DB" in database >>>>>>>>>>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved. >>>>>>>>>>>> >>>>>>>>>>>> I now have a date valid ocspSigningCertificate, but with the >>>>>>>>>>>> wrong >>>>>>>>>>>> subject, and a broken certificate system which will no longer >>>>>>>>>>>> start. >>>>>>>>>>>> >>>>>>>>>>>> ipactl status >>>>>>>>>>>> ... >>>>>>>>>>>> pki-tomcatd Service: STOPPED >>>>>>>>>>>> >>>>>>>>>>>> I can't renew other expired certificates. >>>>>>>>>>>> >>>>>>>>>>>> I also note that there is now no key for >>>>>>>>>>>> ocspSigningCertificate as >>>>>>>>>>>> shown >>>>>>>>>>>> by: >>>>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>>>>>> >>>>>>>>>>>> I wonder if this is because the certificate subject changed? >>>>>>>>>>>> There >>>>>>>>>>>> was a >>>>>>>>>>>> key before the certificate renewed. >>>>>>>>>>>> >>>>>>>>>>>> The ca debug logs are showing: >>>>>>>>>>>> >>>>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by >>>>>>>>>>>> nickname: >>>>>>>>>>>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108 >>>>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to >>>>>>>>>>>> x509CertImpl >>>>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: >>>>>>>>>>>> Certificate >>>>>>>>>>>> object not found >>>>>>>>>>>> Certificate object not found >>>>>>>>>>>>          at >>>>>>>>>>>> com.netscape.ca.SigningUnit.init(SigningUnit.java:184) >>>>>>>>>>>> >>>>>>>>>>>> Any help in repairing my broken ipa system will be much >>>>>>>>>>>> appreciated. >>>>>>>>>>> >>>>>>>>>>> Sorry for the delay. I think my previous reading of the >>>>>>>>>>> certmonger >>>>>>>>>>> csrgen code was wrong. >>>>>>>>>>> >>>>>>>>>>> IIRC in your certmonger entry the cert_subject has the hostname >>>>>>>>>>> value >>>>>>>>>>> right? Do you also have cert_subject_der? >>>>>>>>>>> >>>>>>>>>>> You can decode that by: >>>>>>>>>>> >>>>>>>>>>> 1. Running a hex-to-bin, something hacky like this in python: >>>>>>>>>>> >>>>>>>>>>> import binascii >>>>>>>>>>> >>>>>>>>>>> hex_string = "<hex value>" >>>>>>>>>>> >>>>>>>>>>> binary_string = binascii.unhexlify(hex_string) >>>>>>>>>>> >>>>>>>>>>> fd = open("out", "wb") >>>>>>>>>>> fd.write(binary_string) >>>>>>>>>>> fd.close() >>>>>>>>>>> >>>>>>>>>>> 2. Run: openssl asn1parse -in out -inform der >>>>>>>>>>> >>>>>>>>>>> I'm going to assume this also has the hostname encoded in it. >>>>>>>>>> >>>>>>>>>> Hi Again >>>>>>>>>> >>>>>>>>>> Yes, correct. The cert_subject_der does have the bad CN with the >>>>>>>>>> hostname encoded in it. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Can you try this: >>>>>>>>>>> >>>>>>>>>>> 1. Make a backup of the request file (just in case) > 2. Remove >>>>>>>>>>> cert_subject_der >>>>>>>>>>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM> >>>>>>>>>>> 3. Try the renewal again >>>>>>>>>> >>>>>>>>>> I ran though all of this, checking that the request file was >>>>>>>>>> still >>>>>>>>>> what >>>>>>>>>> we wanted after restarting certmonger with verbose logging, >>>>>>>>>> restoring >>>>>>>>>> all the databases, fixing permissions, checking keys etc., and >>>>>>>>>> ran the >>>>>>>>>> getcert resubmit. >>>>>>>>>> >>>>>>>>>> It still generates a certificate with the bad CN including a >>>>>>>>>> hostname. >>>>>>>>>> >>>>>>>>>> Then the pki-tomcat server fails to start again. >>>>>>>>>> >>>>>>>>>> Other things I have checked are: >>>>>>>>>> 1) The csr= field in the (modified) request file has the correct >>>>>>>>>> subject. >>>>>>>>>> >>>>>>>>>> 2) The dogtag ca debug log file is showing: >>>>>>>>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017 >>>>>>>>>> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK >>>>>>>>>> ... >>>>>>>>>> RenewalSubmitter: renewal original profileId=caIPAserviceCert >>>>>>>>>> RenewalSubmitter: for renewal, original authenticator raCertAuth >>>>>>>>>> found >>>>>>>>>> CertProcessor: No authenticator credentials required >>>>>>>>>> processRenewal: set original Inputs into profile Context >>>>>>>>>> RenewalSubmitter: setInputsIntoContext() getting input name= >>>>>>>>>> cert_request_type >>>>>>>>>> RenewalSubmitter: setInputsIntoContext() setting value in >>>>>>>>>> ctx:pkcs10 >>>>>>>>>> RenewalSubmitter: setInputsIntoContext() getting input name= >>>>>>>>>> cert_request >>>>>>>>>> RenewalSubmitter: setInputsIntoContext() setting value in >>>>>>>>>> ctx:<hex >>>>>>>>>> encoded csr> >>>>>>>>>> ... >>>>>>>>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM> >>>>>>>>>> >>>>>>>>>> The hex encoded csr in the last line decodes to have the bad >>>>>>>>>> subject >>>>>>>>>> where the hostname is one of our other ipa servers. >>>>>>>>>> >>>>>>>>>> The certificate always getthe same hostname in the bad subject >>>>>>>>>> cn. >>>>>>>>>> >>>>>>>>>> Do you have any other ideas of things to check to find out what's >>>>>>>>>> generating the bad subject? >>>>>>>>> >>>>>>>>> The order certmonger uses for the subject is: >>>>>>>>> >>>>>>>>> 1. cert_subject_der >>>>>>>>> 2. If there is no DER, try to encode cert_subject >>>>>>>>> 3. If that fails try again a different way >>>>>>>>> 4. If it fails yet again use CN=localhost >>>>>>>>> >>>>>>>>> It is baffling that it would pick a hostname much less one that >>>>>>>>> certmonger shouldn't even know about. >>>>>>>> >>>>>>>> Indeed, and if I try to renew other certificates for some of >>>>>>>> them it >>>>>>>> chooses other host names that should not be known about. Each >>>>>>>> certificate seems to get the same bad hostname each time I try to >>>>>>>> renew. >>>>>>>> >>>>>>>>> >>>>>>>>> I assume the CSR found in the dogtag logs matches the csr value >>>>>>>>> in the >>>>>>>>> certmonger request? >>>>>>>> >>>>>>>> No, its different. The issued certificate matches the csr seen in >>>>>>>> dogtag >>>>>>>> which makes sense. But the csr in the dogtag logs has the bad >>>>>>>> subject. >>>>>>>> The csr in the request directory file has the good subject. >>>>>>>> >>>>>>>>> >>>>>>>>> Can you share the certmonger request file privately? >>>>>>>> >>>>>>>> Yes, certainly. >>>>>>>> >>>>>>>> Thanks for your continued help on this. >>>>>>> >>>>>>> Let's try this. We'll drop the current request and create a new one. >>>>>>> >>>>>>> Stop tomcat, restore the old cert database, start tomcat, then: >>>>>>> >>>>>>> # getcert stop-tracking -i <request id> >>>>>>> # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n >>>>>>> "ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P >>>>>>> <pin> -B >>>>>>> /usr/libexec/ipa/certmonger/stop_pkicad -C >>>>>>> '/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert >>>>>>> cert-pki-ca"' >>>>>>> >>>>>>> Then try resubmitting the request. >>>>>>> >>>>>> Hi Rob >>>>>> >>>>>> When you say 'stop tomcat', I'm just doing an ipactl stop. That is >>>>>> stopiing the ipa pki-tomcat service. Is that good enough or are there >>>>>> other services that need stopping too? >>>>>> >>>>>> I tried the stop-tracking/start-tracking. Exactly the same result as >>>>>> before. Same hostname appears in the subject CN field of the new >>>>>> certificate and then the pki-tomcat service won't start. >>>>>> >>>>>> I've also been back and redone the manual submits changing the >>>>>> point at >>>>>> which I copied in the edited request as I wondered if there was some >>>>>> caching of csr's going on in certmonger and it was remembering a >>>>>> previous request it had read on startup before I replaced the >>>>>> request file. >>>>>> >>>>>> But nothing seems to stop dogtag issuing a certificate with the >>>>>> Subject >>>>>> CN being that host name. >>>>>> >>>>>> Is it possible that dogtag is somehow overriding the Subject its >>>>>> being >>>>>> explicitly given? >>>>>> >>>>>> FWIW the CSR in the dogtag debug log is always identical, but I guess >>>>>> thats expected if the CSR information is always the same. >>>>>> >>>>>> I'm not sure if its possible to increase the verbosity on the dogtag >>>>>> side. Can you advise please? >>>>>> >>>>>> I've been assuming that its the bad subject that is preventing the >>>>>> pki-tomcat from starting after the new certificate is issued. Does >>>>>> that >>>>>> make sense to you? >>>>>> >>>>>> I wonder where we can go from here? There must be a good reason for >>>>>> whats happening! >>>>> >>>>> Let's step back and gather some more information. >>>>> >>>>> Can you restore the files again and send me the output of: >>>>> >>>>> # getcert list >>>>> >>>>> #  certutil -L -d /etc/pki/pki-tomcat/alias/ -n "ocspSigningCert >>>>> cert-pki-ca" >>>> >>>> Hi Rob >>>> >>>> You should have the output you requested by private email now. >>>> >>>>> >>>>> And the exact commands you are using to do all this, including >>>>> re-issuing the cert >>>> >>>> Thats a great idea. Maybe I'm just doing something in the wrong order. >>>> >>>>> >>>>> Using ipactl is fine. You just don't want to touch the NSS databases >>>>> while the service is running. Similarly you probably don't want to >>>>> mess >>>>> with certmonger requests while it is running as it won't see the >>>>> changes >>>>> until it restarts. >>>> Here is the procedure I'm using to renew the certificates. One of the >>>> tricky things is to not have the certificate system working when >>>> certmonger is started so that it doesn't renew a certificate on >>>> startup that might not be one I have changed the request file for >>>> (other certificates have similar problems with bad subject CN and stop >>>> the certificate system.) >>>> >>>> I've found it convenient to keep reinitializing the log files because >>>> they get very confusing when the clock keeps getting set back. >>>> >>>> The request number has changed since we did the stop-tracking / >>>> start-tracking. >>>> >>>> Roderick >>>> >>>> ipactl stop >>>> >>>> # Set clock back: >>>> systemctl stop ntpd >>>> timedatectl set-time "2017-10-23 00:00:00" >>>> date >>>> >>>> #Stop certificate renewals the systemd way >>>> systemctl stop certmonger >>>> >>>> # Sort out certmonger log >>>> mv /var/log/certmonger.log /var/log/certmonger.log-$(date >>>> +%Y%m%d:%H:%M) >>>> >>>> # ****** Copy in the fixed certmonger request >>>> \cp /root/20161124081302.ocsp_backup >>>> /var/lib/certmonger/requests/20161124081302 >>>> >>>> >>>> # In another window >>>> certmonger -n -d 9 2>&1 | tee /var/log/certmonger.log >>>> >>>> # In the original window >>>> # Sort out dogtag log >>>> mv /var/log/pki/pki-tomcat/ca/debug >>>> /var/log/pki/pki-tomcat/ca/debug-$(date +%Y%m%d:%H:%M).log >>>> touch /var/log/pki/pki-tomcat/ca/debug >>>> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/debug >>>> >>>> mv /var/log/pki/pki-tomcat/ca/selftests.log >>>> /var/log/pki/pki-tomcat/ca/selftests.log-$(date +%Y%m%d:%H:%M) >>>> touch /var/log/pki/pki-tomcat/ca/selftests.log >>>> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/selftests.log >>>> >>>> \cp /var/tmp/rmj/etc/httpd/alias/cert8.db /etc/httpd/alias/cert8.db >>>> \cp /var/tmp/rmj/etc/httpd/alias/key3.db /etc/httpd/alias/key3.db >>>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/cert8.db >>>> /etc/pki/pki-tomcat/alias/cert8.db >>>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/key3.db >>>> /etc/pki/pki-tomcat/alias/key3.db >>>> >>>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/ca/CS.cfg >>>> /etc/pki/pki-tomcat/ca/CS.cfg >>>> >>>> # Fix up certificate permissions and check all is ok >>>> certutil -M -n "caSigningCert cert-pki-ca" -d >>>> /etc/pki/pki-tomcat/alias -t CTu,Cu,Cu >>>> certutil -L -d /etc/pki/pki-tomcat/alias >>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>> <pin> >>>> >>>> # Fix dogtag so it starts >>>> pki-server subsystem-enable ca > >>>> # Start IdM services >>>> >>>> ipactl start >>>> ipactl status >>>> >>>> #Resubmit certificate ocsp >>>> date;getcert resubmit  -i 20161124081302 >>> >>> Hi Rob >>> >>> Does it look like there is anything I should be doing differently in >>> this command sequence to try to get the certificate renewed correctly? >>> >>> I'm starting to wonder whether I should look at alternative strategies >>> for recovering the freeipa servers if the certificates won't renew >>> correctly. Do you have any ideas on this please? >>> >> >> This looks reasonable. Have you tried stop-tracking and start-tracking I >> suggested? > > Yes,I tried the stop-tracking / start-tracking. Same result, bad Subject > CN. > >> >> I'm still baffled how the name(s) are getting mixed up. > > Indeed, something is going round changing the subject. In the dogtag > debug log is: > [23/Oct/2017:00:13:52][http-bio-8080-exec-9]: processRenewal: > origNotAfter =Wed Oct 25 13:24:01 BST 2017 > [23/Oct/2017:00:13:52][http-bio-8080-exec-9]: processRenewal: orig subj > dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK > > and then very shortly afterwards there is a csr listed that decodes to > have the bad Subject CN. > > The listed csr doesn't match any I have been able to find on the system, > so something must be going round and changing it. > >> >> Something else you might want to try would be to move all the other >> requests out of the way in case there is some sort of memory corruption >> causing the hostname to get in there somehow. > > Thanks for the suggestion which I have now tried. Same result I'm afraid. > > I've now been battling this issue since the beginning of November. > > I'm wondering whether, just to get our IPA system fully operational > again, whether we might be able to add the correct CN into a Subject > Alternative name? > > Does this sound like a possible way to at least get all the IPA services > started properly? > > Do you think it would cause further problems down the line? > > Would you be able to advise how to get the certificates to have the > correct CN in the Subject Alternative Name. > > Maybe there are some other tricks we could use just to get going again. > > Can you advise please? > I'm really stumped here. Do you have the CA installed on any other masters? You might try setting it as the renewal master and trying the renewals there instead.