On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: > Hi, > > I seem to be facing a similar issue with one of my KRAs. My KRA > certificates were, for some reason, not automatically renewed when > they expired last month. Using `ipa-cert-fix` correctly fixed them on > _one_ host. On the other, they seem to be stuck in the renewal state > and `ipa-cert-fix` claims there's nothing to do: > > ``` > Request ID '20191031183458': >          status: MONITORING >          ca-error: Server at > "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: > Missing credential: sessionID >          stuck: no >          key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-kra',token='NSS Certificate DB',pin set >          certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-kra',token='NSS Certificate DB' >          CA: dogtag-ipa-ca-renew-agent >          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> >          subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >          expires: 2020-06-27 01:54:34 EDT >          key usage: digitalSignature,nonRepudiation >          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-kra" >          track: yes >          auto-renew: yes > Request ID '20191031183459': >          status: MONITORING >          ca-error: Server at > "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: > Missing credential: sessionID >          stuck: no >          key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS > Certificate DB',pin set >          certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS > Certificate DB' >          CA: dogtag-ipa-ca-renew-agent >          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> >          subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> >          expires: 2020-06-27 01:54:30 EDT >          key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >          eku: id-kp-clientAuth >          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "transportCert cert-pki-kra" >          track: yes >          auto-renew: yes > Request ID '20191031183500': >          status: MONITORING >          ca-error: Server at > "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: > Missing credential: sessionID >          stuck: no >          key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > cert-pki-kra',token='NSS Certificate DB',pin set >          certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > cert-pki-kra',token='NSS Certificate DB' >          CA: dogtag-ipa-ca-renew-agent >          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> >          subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> >          expires: 2020-06-27 01:54:32 EDT >          key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >          eku: id-kp-clientAuth >          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "storageCert cert-pki-kra" >          track: yes >          auto-renew: yes > ``` > > Here are the sequence of events that seem to have led to this: > > 1. Install FreeIPA Master many years ago and continue to upgrade it > from time to time. > 2. Install FreeIPA Replica a few years after and continue to upgrade > it from time to time. > 3. Allow the certificates to expire on both nodes. > 4. Attempt to patch the replica via `yum upgrade` on the second node. > 5. Notice after reboot that `pki-tomcatd` is having trouble and > discover certificate issues. > 5. Issue `ipa-cert-fix`, reboot again, and notice that things are > working. Try and create a key in the vault. > 6. Attempt to patch the master via `yum upgrade` on the first node. > 7. Notice after reboot that everything seems to be ok. Try and create > a key in the vault. > 8. Notice a few days later that renewal seems to be broken on the > first node. > > At this point `ipa-cert-fix` just shows that everything is fine. If I > run it with -v, and then check the "storageCert cert-pki-kra" > certificate with `openssl x509 -text -in`, I'm shown: Hi, just double-checking, but did you run ipa-cert-fix on the replica that was repaired in step 5? If that's the case, it's normal that ipa-cert-fix does not see any issue as it's running only locally and does not attempt to repair remote nodes. You will need to login to the node with expired certs and run ipa-cert-fix there.

HTH, flo > >          Validity >              Not Before: Jun 29 00:52:33 2020 GMT >              Not After : Jun 19 00:52:33 2022 GMT > > On the second known, `getcert list` shows correct expirations for > those certificates: > > Request ID '20191206005909': >          status: MONITORING >          stuck: no >          key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > cert-pki-kra',token='NSS Certificate DB',pin set >          certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > cert-pki-kra',token='NSS Certificate DB' >          CA: dogtag-ipa-ca-renew-agent >          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> >          subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> >          expires: 2022-06-18 20:52:33 EDT >          key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >          eku: id-kp-clientAuth >          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "storageCert cert-pki-kra" >          track: yes >          auto-renew: yes > > It seems like _something_, perhaps `ipa-cert-fix` somehow renewed > these certificates but...outside of certmonger? Is this some other > version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The > certificates are not in CA_WORKING though, they're in MONITORING. > > What can I do to get myself out of this state as it seems like I'm in > a "this could explode at any moment" situation? > > This is on Fedora 30 with IP version: > > Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 > 07:59:16 PM EDT. > Installed Packages > Name         : certmonger > Version      : 0.79.9 > Release      : 1.fc30 > Architecture : x86_64 > Size         : 3.4 M > Source       : certmonger-0.79.9-1.fc30.src.rpm > Repository   : @System >  From repo    : updates > > .. snip .. > > Name         : freeipa-server > Version      : 4.8.3 > Release      : 1.fc30 > Architecture : x86_64 > Size         : 1.3 M > Source       : freeipa-4.8.3-1.fc30.src.rpm > Repository   : @System >  From repo    : updates > > .. snip .. > > Thanks! > >      > Ilya Kogan > w:    github.com/ikogan <http://github.com/ikogan> e: > ikogan(a)mythicnet.org <mailto:ikogan@mythicnet.org> > <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to > freeipa-users-leave(a)lists.fedorahosted.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > _______________________________________________ FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...

Hi, Thanks for the help so far! I've actually run `ipa-cert-fix` on both nodes, it says everything is ok on both nodes. When I run it with verbose mode, it spits out the command it's running and the certificate it got, for example: ``` ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] ``` If I then take that cert and ask what `openssl x509 -text -noout` thinks about it, it tells me that it's valid from 2020-06-29 to 2022-06-29. Strangely, though, when I ask `getcert list`, it shows that the certificate: ``` certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' ``` expires on 2020-06-27. It's almost as if this node's certificate has _already_ been renewed but certmonger (I think) doesn't know about it, which might be why it's having trouble renewing it.

Here's what the two nodes say about replication: From node one: ``` ipa-two.mydomain.org <http://ipa-two.mydomain.org>   last update status: Error (0) Replica acquired successfully: Incremental update succeeded   last update ended: 2020-07-06 17:46:17+00:00 ``` From node two: ``` ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org>   last update status: Error (0) Replica acquired successfully: Incremental update succeeded   last update ended: 2020-07-06 17:46:17+00:00 ``` I suppose this might be a good time to mention that this is a simple two node multi-master setup. Finally, I'm not sure if I'm doing this correctly, but to make absolutely sure about which node is the renewal master, I ran this on both nodes: ``` ldapsearch -H ldap://ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn ldapsearch -H ldap://ipa-two.gaea.mythicnet.org <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn ``` The result for both is: ``` dn: cn=CA,cn=ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomai... ``` So it looks like the renewal master is the one having this problem. Ilya Kogan w: github.com/ikogan <http://github.com/ikogan> e: ikogan(a)mythicnet.org <mailto:ikogan@mythicnet.org> <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten(a)redhat.com <mailto:rcritten@redhat.com>> wrote: Florence Blanc-Renaud via FreeIPA-users wrote: > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: >> Hi, >> >> I seem to be facing a similar issue with one of my KRAs. My KRA >> certificates were, for some reason, not automatically renewed when >> they expired last month. Using `ipa-cert-fix` correctly fixed them on >> _one_ host. On the other, they seem to be stuck in the renewal state >> and `ipa-cert-fix` claims there's nothing to do: >> >> ``` >> Request ID '20191031183458': >>          status: MONITORING >>          ca-error: Server at >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: >> Missing credential: sessionID >>          stuck: no >>          key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >> cert-pki-kra',token='NSS Certificate DB',pin set >>          certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >> cert-pki-kra',token='NSS Certificate DB' >>          CA: dogtag-ipa-ca-renew-agent >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >>          subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> >>          expires: 2020-06-27 01:54:34 EDT >>          key usage: digitalSignature,nonRepudiation >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >>          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-kra" >>          track: yes >>          auto-renew: yes >> Request ID '20191031183459': >>          status: MONITORING >>          ca-error: Server at >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: >> Missing credential: sessionID >>          stuck: no >>          key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS >> Certificate DB',pin set >>          certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS >> Certificate DB' >>          CA: dogtag-ipa-ca-renew-agent >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >>          subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >>          expires: 2020-06-27 01:54:30 EDT >>          key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>          eku: id-kp-clientAuth >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >>          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "transportCert cert-pki-kra" >>          track: yes >>          auto-renew: yes >> Request ID '20191031183500': >>          status: MONITORING >>          ca-error: Server at >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: >> Missing credential: sessionID >>          stuck: no >>          key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB',pin set >>          certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB' >>          CA: dogtag-ipa-ca-renew-agent >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >>          subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >>          expires: 2020-06-27 01:54:32 EDT >>          key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>          eku: id-kp-clientAuth >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >>          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "storageCert cert-pki-kra" >>          track: yes >>          auto-renew: yes >> ``` >> >> Here are the sequence of events that seem to have led to this: >> >> 1. Install FreeIPA Master many years ago and continue to upgrade it >> from time to time. >> 2. Install FreeIPA Replica a few years after and continue to upgrade >> it from time to time. >> 3. Allow the certificates to expire on both nodes. >> 4. Attempt to patch the replica via `yum upgrade` on the second node. >> 5. Notice after reboot that `pki-tomcatd` is having trouble and >> discover certificate issues. >> 5. Issue `ipa-cert-fix`, reboot again, and notice that things are >> working. Try and create a key in the vault. >> 6. Attempt to patch the master via `yum upgrade` on the first node. >> 7. Notice after reboot that everything seems to be ok. Try and create >> a key in the vault. >> 8. Notice a few days later that renewal seems to be broken on the >> first node. >> >> At this point `ipa-cert-fix` just shows that everything is fine. If I >> run it with -v, and then check the "storageCert cert-pki-kra" >> certificate with `openssl x509 -text -in`, I'm shown: > > Hi, > just double-checking, but did you run ipa-cert-fix on the replica that > was repaired in step 5? If that's the case, it's normal that > ipa-cert-fix does not see any issue as it's running only locally and > does not attempt to repair remote nodes. > > You will need to login to the node with expired certs and run > ipa-cert-fix there. I'd also look to see which one is the renewal master. That is the one that should renew the cert. I'm too curious why the renewal raised an error (as if it actually tried to renew) rather than either go into CA_WORKING or pick up the updated cert. I'd also make sure that replication is working. On each master: # ipa-csreplica-manage list -v `hostname` rob > > HTH, > flo > >> >>          Validity >>              Not Before: Jun 29 00:52:33 2020 GMT >>              Not After : Jun 19 00:52:33 2022 GMT >> >> On the second known, `getcert list` shows correct expirations for >> those certificates: >> >> Request ID '20191206005909': >>          status: MONITORING >>          stuck: no >>          key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB',pin set >>          certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB' >>          CA: dogtag-ipa-ca-renew-agent >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >>          subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >>          expires: 2022-06-18 20:52:33 EDT >>          key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>          eku: id-kp-clientAuth >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >>          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "storageCert cert-pki-kra" >>          track: yes >>          auto-renew: yes >> >> It seems like _something_, perhaps `ipa-cert-fix` somehow renewed >> these certificates but...outside of certmonger? Is this some other >> version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The >> certificates are not in CA_WORKING though, they're in MONITORING. >> >> What can I do to get myself out of this state as it seems like I'm in >> a "this could explode at any moment" situation? >> >> This is on Fedora 30 with IP version: >> >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 >> 07:59:16 PM EDT. >> Installed Packages >> Name         : certmonger >> Version      : 0.79.9 >> Release      : 1.fc30 >> Architecture : x86_64 >> Size         : 3.4 M >> Source       : certmonger-0.79.9-1.fc30.src.rpm >> Repository   : @System >>  From repo    : updates >> >> .. snip .. >> >> Name         : freeipa-server >> Version      : 4.8.3 >> Release      : 1.fc30 >> Architecture : x86_64 >> Size         : 1.3 M >> Source       : freeipa-4.8.3-1.fc30.src.rpm >> Repository   : @System >>  From repo    : updates >> >> .. snip .. >> >> Thanks! >> >> >> Ilya Kogan >> w: github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> e: >> ikogan(a)mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> >> <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> >> >> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> >> To unsubscribe send an email to >> freeipa-users-leave(a)lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... >> >> > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > _______________________________________________ FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...

Wow ok, that was easy. `getcert list` now reports correct expiration dates for those certificates and they're all in MONITORING. It still has that ca-error field although it's no longer trying to renew. Is that going to be an issue or is it just going to try again when it's time to renew and succeed?

On Wed, Jul 8, 2020, 9:43 AM Florence Blanc-Renaud <flo(a)redhat.com <mailto:flo@redhat.com>> wrote: On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote: > Hi, > > Thanks for the help so far! I've actually run `ipa-cert-fix` on both > nodes, it says everything is ok on both nodes. When I run it with > verbose mode, it spits out the command it's running and the certificate > it got, for example: > >     ``` >     ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', >     'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert >     cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] >     ``` > > > If I then take that cert and ask what `openssl x509 -text -noout` thinks > about it, it tells me that it's valid from 2020-06-29 to 2022-06-29. > Strangely, though, when I ask `getcert list`, it shows that the certificate: > >     ``` >     certificate: >     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS >     Certificate DB' >     ``` > > > expires on 2020-06-27. It's almost as if this node's certificate has > _already_ been renewed but certmonger (I think) doesn't know about it, > which might be why it's having trouble renewing it. > Hi, you may want to restart certmonger to force it re-reading the certificate information: # sudo systemctl restart certmonger flo > Here's what the two nodes say about replication: > >  From node one: > >     ``` >     ipa-two.mydomain.org <http://ipa-two.mydomain.org> <http://ipa-two.mydomain.org> >        last update status: Error (0) Replica acquired successfully: >     Incremental update succeeded >        last update ended: 2020-07-06 17:46:17+00:00 >     ``` > > >  From node two: > >     ``` >     ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> <http://ipa-one.gaea.mythicnet.org> >        last update status: Error (0) Replica acquired successfully: >     Incremental update succeeded >        last update ended: 2020-07-06 17:46:17+00:00 >     ``` > > > I suppose this might be a good time to mention that this is a simple two > node multi-master setup. Finally, I'm not sure if I'm doing this > correctly, but to make absolutely sure about which node is the renewal > master, I ran this on both nodes: > >     ``` >     ldapsearch -H ldap://ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> >     <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b >     'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' >     '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >     ldapsearch -H ldap://ipa-two.gaea.mythicnet.org <http://ipa-two.gaea.mythicnet.org> >     <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b >     'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' >     '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >     ``` > > > The result for both is: > >     ``` >     dn: cn=CA,cn=ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> >     <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org >     ``` > > > So it looks like the renewal master is the one having this problem. > >        > Ilya Kogan > w:    github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> e: ikogan(a)mythicnet.org <mailto:ikogan@mythicnet.org> > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> > <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> > > > > On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten(a)redhat.com <mailto:rcritten@redhat.com> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>> wrote: > >     Florence Blanc-Renaud via FreeIPA-users wrote: >      > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: >      >> Hi, >      >> >      >> I seem to be facing a similar issue with one of my KRAs. My KRA >      >> certificates were, for some reason, not automatically renewed when >      >> they expired last month. Using `ipa-cert-fix` correctly fixed >     them on >      >> _one_ host. On the other, they seem to be stuck in the renewal state >      >> and `ipa-cert-fix` claims there's nothing to do: >      >> >      >> ``` >      >> Request ID '20191031183458': >      >>          status: MONITORING >      >>          ca-error: Server at >      >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: >      >> Missing credential: sessionID >      >>          stuck: no >      >>          key pair storage: >      >> >     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >      >> cert-pki-kra',token='NSS Certificate DB',pin set >      >>          certificate: >      >> >     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >      >> cert-pki-kra',token='NSS Certificate DB' >      >>          CA: dogtag-ipa-ca-renew-agent >      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >      >> <http://MYDOMAIN.ORG> >      >>          subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> >      >>          expires: 2020-06-27 01:54:34 EDT >      >>          key usage: digitalSignature,nonRepudiation >      >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >      >>          post-save command: >     /usr/libexec/ipa/certmonger/renew_ca_cert >      >> "auditSigningCert cert-pki-kra" >      >>          track: yes >      >>          auto-renew: yes >      >> Request ID '20191031183459': >      >>          status: MONITORING >      >>          ca-error: Server at >      >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: >      >> Missing credential: sessionID >      >>          stuck: no >      >>          key pair storage: >      >> >     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert >     cert-pki-kra',token='NSS >      >> Certificate DB',pin set >      >>          certificate: >      >> >     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert >     cert-pki-kra',token='NSS >      >> Certificate DB' >      >>          CA: dogtag-ipa-ca-renew-agent >      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >      >> <http://MYDOMAIN.ORG> >      >>          subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >      >> <http://MYDOMAIN.ORG> >      >>          expires: 2020-06-27 01:54:30 EDT >      >>          key usage: >      >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >      >>          eku: id-kp-clientAuth >      >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >      >>          post-save command: >     /usr/libexec/ipa/certmonger/renew_ca_cert >      >> "transportCert cert-pki-kra" >      >>          track: yes >      >>          auto-renew: yes >      >> Request ID '20191031183500': >      >>          status: MONITORING >      >>          ca-error: Server at >      >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: >      >> Missing credential: sessionID >      >>          stuck: no >      >>          key pair storage: >      >> >     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >      >> cert-pki-kra',token='NSS Certificate DB',pin set >      >>          certificate: >      >> >     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >      >> cert-pki-kra',token='NSS Certificate DB' >      >>          CA: dogtag-ipa-ca-renew-agent >      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >      >> <http://MYDOMAIN.ORG> >      >>          subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >      >> <http://MYDOMAIN.ORG> >      >>          expires: 2020-06-27 01:54:32 EDT >      >>          key usage: >      >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >      >>          eku: id-kp-clientAuth >      >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >      >>          post-save command: >     /usr/libexec/ipa/certmonger/renew_ca_cert >      >> "storageCert cert-pki-kra" >      >>          track: yes >      >>          auto-renew: yes >      >> ``` >      >> >      >> Here are the sequence of events that seem to have led to this: >      >> >      >> 1. Install FreeIPA Master many years ago and continue to upgrade it >      >> from time to time. >      >> 2. Install FreeIPA Replica a few years after and continue to upgrade >      >> it from time to time. >      >> 3. Allow the certificates to expire on both nodes. >      >> 4. Attempt to patch the replica via `yum upgrade` on the second >     node. >      >> 5. Notice after reboot that `pki-tomcatd` is having trouble and >      >> discover certificate issues. >      >> 5. Issue `ipa-cert-fix`, reboot again, and notice that things are >      >> working. Try and create a key in the vault. >      >> 6. Attempt to patch the master via `yum upgrade` on the first node. >      >> 7. Notice after reboot that everything seems to be ok. Try and >     create >      >> a key in the vault. >      >> 8. Notice a few days later that renewal seems to be broken on the >      >> first node. >      >> >      >> At this point `ipa-cert-fix` just shows that everything is fine. >     If I >      >> run it with -v, and then check the "storageCert cert-pki-kra" >      >> certificate with `openssl x509 -text -in`, I'm shown: >      > >      > Hi, >      > just double-checking, but did you run ipa-cert-fix on the replica >     that >      > was repaired in step 5? If that's the case, it's normal that >      > ipa-cert-fix does not see any issue as it's running only locally and >      > does not attempt to repair remote nodes. >      > >      > You will need to login to the node with expired certs and run >      > ipa-cert-fix there. > >     I'd also look to see which one is the renewal master. That is the one >     that should renew the cert. I'm too curious why the renewal raised an >     error (as if it actually tried to renew) rather than either go into >     CA_WORKING or pick up the updated cert. > >     I'd also make sure that replication is working. On each master: > >     # ipa-csreplica-manage list -v `hostname` > >     rob > >      > >      > HTH, >      > flo >      > >      >> >      >>          Validity >      >>              Not Before: Jun 29 00:52:33 2020 GMT >      >>              Not After : Jun 19 00:52:33 2022 GMT >      >> >      >> On the second known, `getcert list` shows correct expirations for >      >> those certificates: >      >> >      >> Request ID '20191206005909': >      >>          status: MONITORING >      >>          stuck: no >      >>          key pair storage: >      >> >     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >      >> cert-pki-kra',token='NSS Certificate DB',pin set >      >>          certificate: >      >> >     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >      >> cert-pki-kra',token='NSS Certificate DB' >      >>          CA: dogtag-ipa-ca-renew-agent >      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >      >> <http://MYDOMAIN.ORG> >      >>          subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >      >> <http://MYDOMAIN.ORG> >      >>          expires: 2022-06-18 20:52:33 EDT >      >>          key usage: >      >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >      >>          eku: id-kp-clientAuth >      >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >      >>          post-save command: >     /usr/libexec/ipa/certmonger/renew_ca_cert >      >> "storageCert cert-pki-kra" >      >>          track: yes >      >>          auto-renew: yes >      >> >      >> It seems like _something_, perhaps `ipa-cert-fix` somehow renewed >      >> these certificates but...outside of certmonger? Is this some other >      >> version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The >      >> certificates are not in CA_WORKING though, they're in MONITORING. >      >> >      >> What can I do to get myself out of this state as it seems like >     I'm in >      >> a "this could explode at any moment" situation? >      >> >      >> This is on Fedora 30 with IP version: >      >> >      >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 >      >> 07:59:16 PM EDT. >      >> Installed Packages >      >> Name         : certmonger >      >> Version      : 0.79.9 >      >> Release      : 1.fc30 >      >> Architecture : x86_64 >      >> Size         : 3.4 M >      >> Source       : certmonger-0.79.9-1.fc30.src.rpm >      >> Repository   : @System >      >>  From repo    : updates >      >> >      >> .. snip .. >      >> >      >> Name         : freeipa-server >      >> Version      : 4.8.3 >      >> Release      : 1.fc30 >      >> Architecture : x86_64 >      >> Size         : 1.3 M >      >> Source       : freeipa-4.8.3-1.fc30.src.rpm >      >> Repository   : @System >      >>  From repo    : updates >      >> >      >> .. snip .. >      >> >      >> Thanks! >      >> >      >> >      >> Ilya Kogan >      >> w: github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> >     <http://github.com/ikogan> e: >      >> ikogan(a)mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> >     <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>> >      >> <http://twitter.com/ilkogan> >     <https://www.linkedin.com/in/ilyakogan/> >      >> >      >> >      >> _______________________________________________ >      >> FreeIPA-users mailing list -- >     freeipa-users(a)lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> >     <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> >      >> To unsubscribe send an email to >      >> freeipa-users-leave(a)lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> >     <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> >      >> Fedora Code of Conduct: >      >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >      >> List Guidelines: >     https://fedoraproject.org/wiki/Mailing_list_guidelines >      >> List Archives: >      >> >     https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >      >> >      >> >      > _______________________________________________ >      > FreeIPA-users mailing list -- >     freeipa-users(a)lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> >     <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> >      > To unsubscribe send an email to >     freeipa-users-leave(a)lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> >     <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> >      > Fedora Code of Conduct: >      > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >      > List Guidelines: >     https://fedoraproject.org/wiki/Mailing_list_guidelines >      > List Archives: >      > >     https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >      > > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... >

Thanks for that info, I don't see any suspicious errors in startup that I haven't seen before. Just the following: - Token named "NSS Generic Crypto Services", not "NSS Certificate DB", skipping. - Error opening "/etc/httpd/alias/pwdfile.txt": No such file or directory. I don't think either of these are really an issue but I could be wrong.

Grepping the request files does indeed show those ca-error values though. They don't really bother me if they won't cause issues. It seems like it's just the last error it got from the CA, which just won't be updated until it tries to request something next time.

On Wed, Jul 8, 2020, 2:41 PM Rob Crittenden <rcritten(a)redhat.com <mailto:rcritten@redhat.com>> wrote: Ilya Kogan wrote: > Wow ok, that was easy. `getcert list` now reports correct expiration > dates for those certificates and they're all in MONITORING. It still has > that ca-error field although it's no longer trying to renew. Is that > going to be an issue or is it just going to try again when it's time to > renew and succeed? I don't know. I'd check the journal to see if it logged any errors post-restart. I don't believe that the ca-error is stored between restarts. You could grep in /var/lib/certmonger/requests to see I suppose. rob > > On Wed, Jul 8, 2020, 9:43 AM Florence Blanc-Renaud <flo(a)redhat.com <mailto:flo@redhat.com> > <mailto:flo@redhat.com <mailto:flo@redhat.com>>> wrote: > >     On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote: >     > Hi, >     > >     > Thanks for the help so far! I've actually run `ipa-cert-fix` on both >     > nodes, it says everything is ok on both nodes. When I run it with >     > verbose mode, it spits out the command it's running and the >     certificate >     > it got, for example: >     > >     >     ``` >     >     ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', >     >     'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert >     >     cert-pki-kra', '-a', '-f', >     '/etc/pki/pki-tomcat/alias/pwdfile.txt'] >     >     ``` >     > >     > >     > If I then take that cert and ask what `openssl x509 -text -noout` >     thinks >     > about it, it tells me that it's valid from 2020-06-29 to 2022-06-29. >     > Strangely, though, when I ask `getcert list`, it shows that the >     certificate: >     > >     >     ``` >     >     certificate: >     >    >      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >     cert-pki-kra',token='NSS >     >     Certificate DB' >     >     ``` >     > >     > >     > expires on 2020-06-27. It's almost as if this node's certificate has >     > _already_ been renewed but certmonger (I think) doesn't know about >     it, >     > which might be why it's having trouble renewing it. >     > >     Hi, > >     you may want to restart certmonger to force it re-reading the >     certificate information: >     # sudo systemctl restart certmonger > >     flo > >     > Here's what the two nodes say about replication: >     > >     >  From node one: >     > >     >     ``` >     >     ipa-two.mydomain.org <http://ipa-two.mydomain.org> <http://ipa-two.mydomain.org> >     <http://ipa-two.mydomain.org> >     >        last update status: Error (0) Replica acquired successfully: >     >     Incremental update succeeded >     >        last update ended: 2020-07-06 17:46:17+00:00 >     >     ``` >     > >     > >     >  From node two: >     > >     >     ``` >     >     ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> <http://ipa-one.gaea.mythicnet.org> >     <http://ipa-one.gaea.mythicnet.org> >     >        last update status: Error (0) Replica acquired successfully: >     >     Incremental update succeeded >     >        last update ended: 2020-07-06 17:46:17+00:00 >     >     ``` >     > >     > >     > I suppose this might be a good time to mention that this is a >     simple two >     > node multi-master setup. Finally, I'm not sure if I'm doing this >     > correctly, but to make absolutely sure about which node is the >     renewal >     > master, I ran this on both nodes: >     > >     >     ``` >     >     ldapsearch -H ldap://ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> >     <http://ipa-one.gaea.mythicnet.org> >     >     <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' >     -W -b >     >     'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' >     >     '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >     >     ldapsearch -H ldap://ipa-two.gaea.mythicnet.org <http://ipa-two.gaea.mythicnet.org> >     <http://ipa-two.gaea.mythicnet.org> >     >     <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' >     -W -b >     >     'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' >     >     '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >     >     ``` >     > >     > >     > The result for both is: >     > >     >     ``` >     >     dn: cn=CA,cn=ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> >     <http://ipa-one.gaea.mythicnet.org> >     >    >      <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org >     >     ``` >     > >     > >     > So it looks like the renewal master is the one having this problem. >     > >     >        >     > Ilya Kogan >     > w:    github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> >     <http://github.com/ikogan> e: ikogan(a)mythicnet.org <mailto:ikogan@mythicnet.org> >     <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> >     > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>> >     > <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> >     > >     > >     > >     > On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten(a)redhat.com <mailto:rcritten@redhat.com> >     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>> >     > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>> wrote: >     > >     >     Florence Blanc-Renaud via FreeIPA-users wrote: >     >      > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: >     >      >> Hi, >     >      >> >     >      >> I seem to be facing a similar issue with one of my KRAs. >     My KRA >     >      >> certificates were, for some reason, not automatically >     renewed when >     >      >> they expired last month. Using `ipa-cert-fix` correctly fixed >     >     them on >     >      >> _one_ host. On the other, they seem to be stuck in the >     renewal state >     >      >> and `ipa-cert-fix` claims there's nothing to do: >     >      >> >     >      >> ``` >     >      >> Request ID '20191031183458': >     >      >>          status: MONITORING >     >      >>          ca-error: Server at >     >      >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" >     replied: >     >      >> Missing credential: sessionID >     >      >>          stuck: no >     >      >>          key pair storage: >     >      >> >     >    >      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >     >      >> cert-pki-kra',token='NSS Certificate DB',pin set >     >      >>          certificate: >     >      >> >     >    >      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >     >      >> cert-pki-kra',token='NSS Certificate DB' >     >      >>          CA: dogtag-ipa-ca-renew-agent >     >      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >     >     <http://MYDOMAIN.ORG> >     >      >> <http://MYDOMAIN.ORG> >     >      >>          subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >     >     <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> >     >      >>          expires: 2020-06-27 01:54:34 EDT >     >      >>          key usage: digitalSignature,nonRepudiation >     >      >>          pre-save command: >     /usr/libexec/ipa/certmonger/stop_pkicad >     >      >>          post-save command: >     >     /usr/libexec/ipa/certmonger/renew_ca_cert >     >      >> "auditSigningCert cert-pki-kra" >     >      >>          track: yes >     >      >>          auto-renew: yes >     >      >> Request ID '20191031183459': >     >      >>          status: MONITORING >     >      >>          ca-error: Server at >     >      >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" >     replied: >     >      >> Missing credential: sessionID >     >      >>          stuck: no >     >      >>          key pair storage: >     >      >> >     >    >      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert >     >     cert-pki-kra',token='NSS >     >      >> Certificate DB',pin set >     >      >>          certificate: >     >      >> >     >    >      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert >     >     cert-pki-kra',token='NSS >     >      >> Certificate DB' >     >      >>          CA: dogtag-ipa-ca-renew-agent >     >      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >     >     <http://MYDOMAIN.ORG> >     >      >> <http://MYDOMAIN.ORG> >     >      >>          subject: CN=KRA Transport >     Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> >     >     <http://MYDOMAIN.ORG> >     >      >> <http://MYDOMAIN.ORG> >     >      >>          expires: 2020-06-27 01:54:30 EDT >     >      >>          key usage: >     >      >> >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >     >      >>          eku: id-kp-clientAuth >     >      >>          pre-save command: >     /usr/libexec/ipa/certmonger/stop_pkicad >     >      >>          post-save command: >     >     /usr/libexec/ipa/certmonger/renew_ca_cert >     >      >> "transportCert cert-pki-kra" >     >      >>          track: yes >     >      >>          auto-renew: yes >     >      >> Request ID '20191031183500': >     >      >>          status: MONITORING >     >      >>          ca-error: Server at >     >      >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" >     replied: >     >      >> Missing credential: sessionID >     >      >>          stuck: no >     >      >>          key pair storage: >     >      >> >     >    >      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >     >      >> cert-pki-kra',token='NSS Certificate DB',pin set >     >      >>          certificate: >     >      >> >     >    >      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >     >      >> cert-pki-kra',token='NSS Certificate DB' >     >      >>          CA: dogtag-ipa-ca-renew-agent >     >      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >     >     <http://MYDOMAIN.ORG> >     >      >> <http://MYDOMAIN.ORG> >     >      >>          subject: CN=KRA Storage >     Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> >     >     <http://MYDOMAIN.ORG> >     >      >> <http://MYDOMAIN.ORG> >     >      >>          expires: 2020-06-27 01:54:32 EDT >     >      >>          key usage: >     >      >> >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >     >      >>          eku: id-kp-clientAuth >     >      >>          pre-save command: >     /usr/libexec/ipa/certmonger/stop_pkicad >     >      >>          post-save command: >     >     /usr/libexec/ipa/certmonger/renew_ca_cert >     >      >> "storageCert cert-pki-kra" >     >      >>          track: yes >     >      >>          auto-renew: yes >     >      >> ``` >     >      >> >     >      >> Here are the sequence of events that seem to have led to this: >     >      >> >     >      >> 1. Install FreeIPA Master many years ago and continue to >     upgrade it >     >      >> from time to time. >     >      >> 2. Install FreeIPA Replica a few years after and continue >     to upgrade >     >      >> it from time to time. >     >      >> 3. Allow the certificates to expire on both nodes. >     >      >> 4. Attempt to patch the replica via `yum upgrade` on the >     second >     >     node. >     >      >> 5. Notice after reboot that `pki-tomcatd` is having >     trouble and >     >      >> discover certificate issues. >     >      >> 5. Issue `ipa-cert-fix`, reboot again, and notice that >     things are >     >      >> working. Try and create a key in the vault. >     >      >> 6. Attempt to patch the master via `yum upgrade` on the >     first node. >     >      >> 7. Notice after reboot that everything seems to be ok. Try and >     >     create >     >      >> a key in the vault. >     >      >> 8. Notice a few days later that renewal seems to be broken >     on the >     >      >> first node. >     >      >> >     >      >> At this point `ipa-cert-fix` just shows that everything is >     fine. >     >     If I >     >      >> run it with -v, and then check the "storageCert cert-pki-kra" >     >      >> certificate with `openssl x509 -text -in`, I'm shown: >     >      > >     >      > Hi, >     >      > just double-checking, but did you run ipa-cert-fix on the >     replica >     >     that >     >      > was repaired in step 5? If that's the case, it's normal that >     >      > ipa-cert-fix does not see any issue as it's running only >     locally and >     >      > does not attempt to repair remote nodes. >     >      > >     >      > You will need to login to the node with expired certs and run >     >      > ipa-cert-fix there. >     > >     >     I'd also look to see which one is the renewal master. That is >     the one >     >     that should renew the cert. I'm too curious why the renewal >     raised an >     >     error (as if it actually tried to renew) rather than either go >     into >     >     CA_WORKING or pick up the updated cert. >     > >     >     I'd also make sure that replication is working. On each master: >     > >     >     # ipa-csreplica-manage list -v `hostname` >     > >     >     rob >     > >     >      > >     >      > HTH, >     >      > flo >     >      > >     >      >> >     >      >>          Validity >     >      >>              Not Before: Jun 29 00:52:33 2020 GMT >     >      >>              Not After : Jun 19 00:52:33 2022 GMT >     >      >> >     >      >> On the second known, `getcert list` shows correct >     expirations for >     >      >> those certificates: >     >      >> >     >      >> Request ID '20191206005909': >     >      >>          status: MONITORING >     >      >>          stuck: no >     >      >>          key pair storage: >     >      >> >     >    >      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >     >      >> cert-pki-kra',token='NSS Certificate DB',pin set >     >      >>          certificate: >     >      >> >     >    >      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >     >      >> cert-pki-kra',token='NSS Certificate DB' >     >      >>          CA: dogtag-ipa-ca-renew-agent >     >      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >     <http://MYDOMAIN.ORG> >     >     <http://MYDOMAIN.ORG> >     >      >> <http://MYDOMAIN.ORG> >     >      >>          subject: CN=KRA Storage >     Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> >     >     <http://MYDOMAIN.ORG> >     >      >> <http://MYDOMAIN.ORG> >     >      >>          expires: 2022-06-18 20:52:33 EDT >     >      >>          key usage: >     >      >> >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >     >      >>          eku: id-kp-clientAuth >     >      >>          pre-save command: >     /usr/libexec/ipa/certmonger/stop_pkicad >     >      >>          post-save command: >     >     /usr/libexec/ipa/certmonger/renew_ca_cert >     >      >> "storageCert cert-pki-kra" >     >      >>          track: yes >     >      >>          auto-renew: yes >     >      >> >     >      >> It seems like _something_, perhaps `ipa-cert-fix` somehow >     renewed >     >      >> these certificates but...outside of certmonger? Is this >     some other >     >      >> version of >     https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The >     >      >> certificates are not in CA_WORKING though, they're in >     MONITORING. >     >      >> >     >      >> What can I do to get myself out of this state as it seems like >     >     I'm in >     >      >> a "this could explode at any moment" situation? >     >      >> >     >      >> This is on Fedora 30 with IP version: >     >      >> >     >      >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 >     >      >> 07:59:16 PM EDT. >     >      >> Installed Packages >     >      >> Name         : certmonger >     >      >> Version      : 0.79.9 >     >      >> Release      : 1.fc30 >     >      >> Architecture : x86_64 >     >      >> Size         : 3.4 M >     >      >> Source       : certmonger-0.79.9-1.fc30.src.rpm >     >      >> Repository   : @System >     >      >>  From repo    : updates >     >      >> >     >      >> .. snip .. >     >      >> >     >      >> Name         : freeipa-server >     >      >> Version      : 4.8.3 >     >      >> Release      : 1.fc30 >     >      >> Architecture : x86_64 >     >      >> Size         : 1.3 M >     >      >> Source       : freeipa-4.8.3-1.fc30.src.rpm >     >      >> Repository   : @System >     >      >>  From repo    : updates >     >      >> >     >      >> .. snip .. >     >      >> >     >      >> Thanks! >     >      >> >     >      >> >     >      >> Ilya Kogan >     >      >> w: github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> >     <http://github.com/ikogan> >     >     <http://github.com/ikogan> e: >     >      >> ikogan(a)mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> >     <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>> >     >     <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> >     <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>>> >     >      >> <http://twitter.com/ilkogan> >     >     <https://www.linkedin.com/in/ilyakogan/> >     >      >> >     >      >> >     >      >> _______________________________________________ >     >      >> FreeIPA-users mailing list -- >     >     freeipa-users(a)lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> >     <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> >     >     <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> >     <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>>> >     >      >> To unsubscribe send an email to >     >      >> freeipa-users-leave(a)lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> >     <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> >     >     <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> >     <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>>> >     >      >> Fedora Code of Conduct: >     >      >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >     >      >> List Guidelines: >     >     https://fedoraproject.org/wiki/Mailing_list_guidelines >     >      >> List Archives: >     >      >> >     >    >      https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >     >      >> >     >      >> >     >      > _______________________________________________ >     >      > FreeIPA-users mailing list -- >     >     freeipa-users(a)lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> >     <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> >     >     <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> >     <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>>> >     >      > To unsubscribe send an email to >     >     freeipa-users-leave(a)lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> >     <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> >     >     <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> >     <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>>> >     >      > Fedora Code of Conduct: >     >      > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >     >      > List Guidelines: >     >     https://fedoraproject.org/wiki/Mailing_list_guidelines >     >      > List Archives: >     >      > >     >    >      https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >     >      > >     > >     > >     > _______________________________________________ >     > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> >     <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> >     > To unsubscribe send an email to >     freeipa-users-leave(a)lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> >     <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> >     > Fedora Code of Conduct: >     https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >     > List Guidelines: >     https://fedoraproject.org/wiki/Mailing_list_guidelines >     > List Archives: >     https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >     > > _______________________________________________ FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...