Hi,

I seem to be facing a similar issue with one of my KRAs. My KRA certificates were, for some reason, not automatically renewed when they expired last month. Using `ipa-cert-fix` correctly fixed them on _one_ host. On the other, they seem to be stuck in the renewal state and `ipa-cert-fix` claims there's nothing to do:

```
Request ID '20191031183458':
        status: MONITORING
        ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID   
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB',pin set    
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
        subject: CN=KRA Audit,O=MYDOMAIN.ORG
        expires: 2020-06-27 01:54:34 EDT
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-kra"
        track: yes
        auto-renew: yes
Request ID '20191031183459':
        status: MONITORING
        ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB',pin set       
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
        subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG
        expires: 2020-06-27 01:54:30 EDT
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "transportCert cert-pki-kra"
        track: yes
        auto-renew: yes
Request ID '20191031183500':
        status: MONITORING
        ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
        subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG
        expires: 2020-06-27 01:54:32 EDT
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra"
        track: yes
        auto-renew: yes
```

Here are the sequence of events that seem to have led to this:

1. Install FreeIPA Master many years ago and continue to upgrade it from time to time.
2. Install FreeIPA Replica a few years after and continue to upgrade it from time to time.
3. Allow the certificates to expire on both nodes.
4. Attempt to patch the replica via `yum upgrade` on the second node.
5. Notice after reboot that `pki-tomcatd` is having trouble and discover certificate issues.
5. Issue `ipa-cert-fix`, reboot again, and notice that things are working. Try and create a key in the vault.
6. Attempt to patch the master via `yum upgrade` on the first node.
7. Notice after reboot that everything seems to be ok. Try and create a key in the vault.
8. Notice a few days later that renewal seems to be broken on the first node.

At this point `ipa-cert-fix` just shows that everything is fine. If I run it with -v, and then check the "storageCert cert-pki-kra" certificate with `openssl x509 -text -in`, I'm shown:

        Validity
            Not Before: Jun 29 00:52:33 2020 GMT          
            Not After : Jun 19 00:52:33 2022 GMT

On the second known, `getcert list` shows correct expirations for those certificates:

Request ID '20191206005909':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
        subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG
        expires: 2022-06-18 20:52:33 EDT
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra"
        track: yes
        auto-renew: yes


It seems like _something_, perhaps `ipa-cert-fix` somehow renewed these certificates but...outside of certmonger? Is this some other version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The certificates are not in CA_WORKING though, they're in MONITORING.

What can I do to get myself out of this state as it seems like I'm in a "this could explode at any moment" situation?

This is on Fedora 30 with IP version:

Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 07:59:16 PM EDT.
Installed Packages
Name         : certmonger
Version      : 0.79.9
Release      : 1.fc30
Architecture : x86_64
Size         : 3.4 M
Source       : certmonger-0.79.9-1.fc30.src.rpm
Repository   : @System
From repo    : updates

.. snip ..

Name         : freeipa-server
Version      : 4.8.3
Release      : 1.fc30
Architecture : x86_64
Size         : 1.3 M
Source       : freeipa-4.8.3-1.fc30.src.rpm
Repository   : @System
From repo    : updates

.. snip ..

Thanks!

Ilya Kogan
w:github.com/ikogan   e:  ikogan@mythicnet.org