Thank you flo! Those are very good leads. I also found your blog with some very helpful posts, thanks! I see the Server-Cert must be after 2021-03-08 now, but also the IPA certs need to be after 2021-09-01. Few questions:

1. Also strangely we have 7 IPA certs issued, all identical except differing "Not Before" and "Not After" dates (probably some CLI command being issued multiple times in the past?).
2. Do I consider the Server-Cert as HTTP and the IPA certs as LDAP? I am a bit confused which is which.
3. One thing I seem to be stuck on, I am not finding an LE issued cert. I see all the LE CA certs. Is it possible the old master had the LE third party CA set up and now we dont on this new master? I suspect this is the case; browsing to our https site I see it is using the Server-Cert with id 0xfff0004.

Good news is setting the clock to 2021-09-02 brings pki-tomcatd up (yay, ty). Looks like I only had a 4 day window to reset the clock. Now to get the pki certs renewed... it looks like certmonger is not issuing the renew requests, but triggering the renews explicitly gets 3/4 updated. The "'auditSigningCert cert-pki-ca" is not wanting to update; journalctl is showing me "Failed to remove certificate ISRGRootCAX3". Went ahead and removed this with certutil from /etc/httpd/alias and the renew went through.

Cheers!
Jacob

# certutil -L -d /etc/httpd/alias -l

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Server-Cert                                                  u,u,u
IPA.COMPANY.COM IPA CA                                      CT,C,C
IPA.COMPANY.COM IPA CA                                      CT,C,C
IPA.COMPANY.COM IPA CA                                      CT,C,C
IPA.COMPANY.COM IPA CA                                      CT,C,C
DSTRootCAX3                                                  C,,
CN=R3,O=Let's Encrypt,C=US                                   C,,
CN=E1,O=Let's Encrypt,C=US                                   C,,
IPA.COMPANY.COM IPA CA                                      CT,C,C
IPA.COMPANY.COM IPA CA                                      CT,C,C
IPA.COMPANY.COM IPA CA                                      CT,C,C
ISRGRootCAX3                                                 C,,
ISRGRootCAX3                                                 C,,
ISRGRootCAX1                                                 C,,
CN=ISRG Root X2,O=Internet Security Research Group,C=US      C,,
CN=R4,O=Let's Encrypt,C=US                                   C,,
CN=E2,O=Let's Encrypt,C=US                                   C,,

# certutil -L -d /etc/httpd/alias -n "Server-Cert" | grep -E "Certificate:|Subject:|Validity|Not Before|Not After|Issuer|Serial Number"
Certificate:
        Serial Number: 268369924 (0xfff0004)
        Issuer: "CN=Certificate Authority,O=IPA.COMPANY.COM"
        Validity:
            Not Before: Mon Mar 08 22:31:53 2021
            Not After : Thu Mar 09 22:31:53 2023
        Subject: "CN=ipa.internal.company.com,O=IPA.COMPANY.COM"

# certutil -L -d /etc/httpd/alias -n "IPA.COMPANY.COM IPA CA" | grep -E "Certificate:|Subject:|Validity|Not Before|Not After|Issuer|Serial Number"
Certificate:
        Serial Number: 268369930 (0xfff000a)
        Issuer: "CN=Certificate Authority,O=IPA.COMPANY.COM"
        Validity:
            Not Before: Wed Sep 01 05:41:44 2021
            Not After : Sun Sep 01 05:41:44 2041
        Subject: "CN=Certificate Authority,O=IPA.COMPANY.COM"
Certificate:
        Serial Number: 268369929 (0xfff0009)
        Issuer: "CN=Certificate Authority,O=IPA.COMPANY.COM"
        Validity:
            Not Before: Wed Sep 01 05:34:27 2021
            Not After : Sun Sep 01 05:34:27 2041
        Subject: "CN=Certificate Authority,O=IPA.COMPANY.COM"
Certificate:
        Serial Number: 268369928 (0xfff0008)
        Issuer: "CN=Certificate Authority,O=IPA.COMPANY.COM"
        Validity:
            Not Before: Wed Sep 01 05:18:21 2021
            Not After : Sun Sep 01 05:18:21 2041
        Subject: "CN=Certificate Authority,O=IPA.COMPANY.COM"
Certificate:
        Serial Number: 268369927 (0xfff0007)
        Issuer: "CN=Certificate Authority,O=IPA.COMPANY.COM"
        Validity:
            Not Before: Wed Sep 01 05:16:37 2021
            Not After : Sun Sep 01 05:16:37 2041
        Subject: "CN=Certificate Authority,O=IPA.COMPANY.COM"
Certificate:
        Serial Number: 268369926 (0xfff0006)
        Issuer: "CN=Certificate Authority,O=IPA.COMPANY.COM"
        Validity:
            Not Before: Wed Sep 01 05:10:15 2021
            Not After : Sun Sep 01 05:10:15 2041
        Subject: "CN=Certificate Authority,O=IPA.COMPANY.COM"
Certificate:
        Serial Number: 268369925 (0xfff0005)
        Issuer: "CN=Certificate Authority,O=IPA.COMPANY.COM"
        Validity:
            Not Before: Wed Sep 01 05:04:01 2021
            Not After : Sun Sep 01 05:04:01 2041
        Subject: "CN=Certificate Authority,O=IPA.COMPANY.COM"
Certificate:
        Serial Number: 12 (0xc)
        Issuer: "CN=Certificate Authority,O=IPA.COMPANY.COM"
        Validity:
            Not Before: Fri Dec 22 18:36:51 2017
            Not After : Tue Dec 22 18:36:51 2037
        Subject: "CN=Certificate Authority,O=IPA.COMPANY.COM"

# ipa-cacert-manage renew
Renewing CA certificate, please wait
CA certificate successfully renewed
The ipa-cacert-manage command was successful

# getcert resubmit -i 20190405192207
# getcert resubmit -i 20190405204558
# getcert resubmit -i 20190405204559

# journalctl -f
Nov 30 04:26:11 ipa.internal.company.com server[943811]: CMSEngine.initializePasswordStore() begins
Nov 30 04:26:11 ipa.internal.company.com server[943811]: CMSEngine.initializePasswordStore(): tag=internaldb
Nov 30 04:26:11 ipa.internal.company.com server[943811]: CMSEngine.initializePasswordStore(): tag=replicationdb
Nov 30 04:26:13 ipa.internal.company.com server[943811]: -----------------------
Nov 30 04:26:13 ipa.internal.company.com server[943811]: Disabled "ca" subsystem
Nov 30 04:26:13 ipa.internal.company.com server[943811]: -----------------------
Nov 30 04:26:13 ipa.internal.company.com server[943811]: Subsystem ID: ca
Nov 30 04:26:13 ipa.internal.company.com server[943811]: Instance ID: pki-tomcat
Nov 30 04:26:13 ipa.internal.company.com server[943811]: Enabled: False
Nov 30 04:26:13 ipa.internal.company.com server[943811]: CA is started.
Nov 30 04:26:13 ipa.internal.company.com server[943811]: WARNING: PKIListener: Subsystem CA is disabled.
Nov 30 04:26:13 ipa.internal.company.com server[943811]: WARNING: PKIListener: Check /var/log/pki/pki-tomcat/ca/selftests.log for possible errors.
Nov 30 04:26:13 ipa.internal.company.com server[943811]: WARNING: PKIListener: To enable the subsystem:
Nov 30 04:26:13 ipa.internal.company.com server[943811]: WARNING: PKIListener:   pki-server subsystem-enable -i pki-tomcat ca

# cat /var/log/pki/pki-tomcat/ca/selftests.log

0.localhost-startStop-1 - [01/Sep/2021:05:00:56 UTC] [20] [1] SystemCertsVerification: system certs verification failure: Certutils.verifySystemCertValidityByNickname:  faliled: nickname: caSigningCert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  failed: nickname: caSigningCert cert-pki-ca

# journal -f
Sep 02 05:04:30 ipa.internal.company.com renew_ca_cert[945080]: Stopping pki_tomcatd
Sep 02 05:04:30 ipa.internal.company.com systemd[1]: Stopping PKI Tomcat Server pki-tomcat...
Sep 02 05:04:30 ipa.internal.company.com server[945093]: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
Sep 02 05:04:30 ipa.internal.company.com server[945093]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
Sep 02 05:04:30 ipa.internal.company.com server[945093]: main class used: org.apache.catalina.startup.Bootstrap
Sep 02 05:04:30 ipa.internal.company.com server[945093]: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni
Sep 02 05:04:30 ipa.internal.company.com server[945093]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
Sep 02 05:04:30 ipa.internal.company.com server[945093]: arguments used: stop
Sep 02 05:04:30 ipa.internal.company.com server[945093]: WARNING: Problem with JAR file [/usr/share/pki/server/common/lib/symkey.jar], exists: [false], canRead: [false]
Sep 02 05:04:31 ipa.internal.company.com systemd[1]: Stopped PKI Tomcat Server pki-tomcat.
Sep 02 05:04:31 ipa.internal.company.com renew_ca_cert[945080]: Stopped pki_tomcatd
Sep 02 05:04:31 ipa.internal.company.com renew_ca_cert[945080]: Updating entry cn=ac46e0eb-c924-420b-9795-9a7074ba8060,ou=authorities,ou=ca,o=ipaca
Sep 02 05:04:31 ipa.internal.company.com renew_ca_cert[945080]: Not updating CS.cfg
Sep 02 05:04:31 ipa.internal.company.com renew_ca_cert[945080]: Failed to remove certificate ISRGRootCAX3
Sep 02 05:04:32 ipa.internal.company.com renew_ca_cert[945080]: Starting pki_tomcatd

# certutil -d /etc/httpd/alias -D -n ISRGRootCAX3

# getcert resubmit -i 20190405204557

# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service

# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20190405192115':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-IPA-COMPANY-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-IPA-COMPANY-COM/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-IPA-COMPANY-COM',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-03-09 22:30:53 UTC
        dns: ipa.internal.company.com
        principal name: ldap/ipa.internal.company.com@IPA.COMPANY.COM
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv IPA-COMPANY-COM
        track: yes
        auto-renew: yes
Request ID '20190405192140':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-03-09 22:31:53 UTC
        dns: ipa.internal.company.com
        principal name: HTTP/ipa.internal.company.com@IPA.COMPANY.COM
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
Request ID '20190405192207':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=IPA RA,O=IPA.COMPANY.COM
        expires: 2023-08-23 05:08:49 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
Request ID '20190405192208':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
        certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-03-09 22:30:44 UTC
        principal name: krbtgt/IPA.COMPANY.COM@IPA.COMPANY.COM
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-pkinit-KPKdc
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
        track: yes
        auto-renew: yes
Request ID '20190405204557':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=CA Audit,O=IPA.COMPANY.COM
        expires: 2023-08-23 05:24:02 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204558':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=OCSP Subsystem,O=IPA.COMPANY.COM
        expires: 2023-08-23 05:12:19 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204559':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=CA Subsystem,O=IPA.COMPANY.COM
        expires: 2023-08-23 05:14:24 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204600':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=Certificate Authority,O=IPA.COMPANY.COM
        expires: 2041-09-02 05:05:18 UTC
        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204601':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-02-15 22:30:43 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
        track: yes
        auto-renew: yes

On Mon, Nov 29, 2021 at 4:22 AM Florence Blanc-Renaud <flo@redhat.com> wrote:
Hi,

The error "Peer's certificate issuer has been marked as not trusted by the user." points to PKI not trusting the LDAP certificate.

1. When moving the date back, you need to carefully pick the date. As the HTTP and LDAP certs have already been renewed, their "valid from" date is probably around 2021-03-08, meaning you need to pick a date between 2021-03-08 and 2021-09-05 for all the certs to be valid (otherwise the LDAP cert is not yet valid and not trusted).

2. Let's Encrypt changed their chain of trust in October (https://letsencrypt.org/certificates/). You need to check which chain was used to sign the LDAP certificate and make sure it is present in /etc/pki/pki-tomcat/alias.If the chain is missing from the PKI NSS DB, PKI won't trust the LDAP certificate.

HTH,
flo

On Sun, Nov 28, 2021 at 5:09 PM Jacob Block via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
Hi all,

I have read through pretty much every thread on this topic and unfortunately will be starting a new one. I am trying to upgrade an older IPA server that has had all the cert-pki-ca certs expired. Some other history, the initial master used to be on a VPS and was moved on-site several years ago by spinning up a replica on-site, promoting it to the new master, and shutting down the master. I am not entirely convinced there wasn't some issue also before the expired certs. There is also no other replica. I'd like to get this working, create a replica, and start upgrading to the latest.

# ipa --version
VERSION: 4.6.4, API_VERSION: 2.230

# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20190405192115':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-IPA-COMPANY-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-IPA-COMPANY-COM/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-IPA-COMPANY-COM',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-03-09 22:30:53 UTC
        dns: ipa.internal.company.com
        principal name: ldap/ipa.internal.company.com@IPA.COMPANY.COM
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv IPA-COMPANY-COM
        track: yes
        auto-renew: yes
Request ID '20190405192140':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-03-09 22:31:53 UTC
        dns: ipa.internal.company.com
        principal name: HTTP/ipa.internal.company.com@IPA.COMPANY.COM
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
Request ID '20190405192207':
        status: NEED_GUIDANCE
        stuck: yes
        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=IPA RA,O=IPA.COMPANY.COM
        expires: 2021-09-05 16:48:11 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
Request ID '20190405192208':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
        certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-03-09 22:30:44 UTC
        principal name: krbtgt/IPA.COMPANY.COM@IPA.COMPANY.COM
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-pkinit-KPKdc
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
        track: yes
        auto-renew: yes
Request ID '20190405204557':
        status: NEED_GUIDANCE
        stuck: yes
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=CA Audit,O=IPA.COMPANY.COM
        expires: 2021-09-05 16:48:31 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204558':
        status: GENERATING_CSR
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=OCSP Subsystem,O=IPA.COMPANY.COM
        expires: 2021-09-05 16:49:41 UTC
        eku: id-kp-OCSPSigning
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204559':
        status: NEED_GUIDANCE
        stuck: yes
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=CA Subsystem,O=IPA.COMPANY.COM
        expires: 2021-09-05 16:48:21 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204600':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=Certificate Authority,O=IPA.COMPANY.COM
        expires: 2041-09-01 05:41:44 UTC
        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204601':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-02-15 22:30:43 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
        track: yes
        auto-renew: yes

The renewal master used to be the remote VPS master that no longer exists. I've since updated that:

#  ipa config-show | grep renewal
  IPA CA renewal master: ipa.internal.company.com

One thing I am confused by is seeing four entries for "caSigningCert cert-pki-ca" (I also have a tenuous understanding of CAs and certs)

# certutil -L -d /var/lib/pki/pki-tomcat/alias/

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

subsystemCert cert-pki-ca                                    u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
DSTRootCAX3                                                  C,,
CN=R3,O=Let's Encrypt,C=US                                   C,,
CN=E1,O=Let's Encrypt,C=US                                   C,,
auditSigningCert cert-pki-ca                                 u,u,Pu
ocspSigningCert cert-pki-ca                                  u,u,u
Server-Cert cert-pki-ca                                      u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
ISRGRootCAX3                                                 C,,
ISRGRootCAX3                                                 C,,
ISRGRootCAX1                                                 C,,
CN=ISRG Root X2,O=Internet Security Research Group,C=US      C,,
CN=R4,O=Let's Encrypt,C=US                                   C,,
CN=E2,O=Let's Encrypt,C=US                                   C,,

I've tried rolling back the clock to before 2021-09-05 but pki-tomcatd still doesn't start:

Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore() begins
Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore(): tag=internaldb
Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore(): tag=replicationdb
Jun 01 05:15:45 ipa.internal.company.com server[919212]: Internal Database Error encountered: Could not connect to LDAP server host ipa.internal.company.com port 636 Error netscape.ldap.LDAPException: Unable to create socket: org.mozilla.jss.ssl.SSLSocketException: org.mozilla.jss.ssl.SSLSocketException: SSL_ForceHandshake failed: (-8172) Peer's certificate issuer has been marked as not trusted by the user. (-1)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@70aacdbc background process
Jun 01 05:15:55 ipa.internal.company.com server[919212]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at java.lang.Thread.run(Thread.java:748)

Maybe its pki certs + https certs are both having a problem? Maybe this is related to a recent LE CA?

Any thoughts would be greatly appreciated. Thank you!
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure