Hi,

The error "Peer's certificate issuer has been marked as not trusted by the user." points to PKI not trusting the LDAP certificate.

1. When moving the date back, you need to carefully pick the date. As the HTTP and LDAP certs have already been renewed, their "valid from" date is probably around 2021-03-08, meaning you need to pick a date between 2021-03-08 and 2021-09-05 for all the certs to be valid (otherwise the LDAP cert is not yet valid and not trusted).

2. Let's Encrypt changed their chain of trust in October (https://letsencrypt.org/certificates/). You need to check which chain was used to sign the LDAP certificate and make sure it is present in /etc/pki/pki-tomcat/alias.If the chain is missing from the PKI NSS DB, PKI won't trust the LDAP certificate.

HTH,
flo

On Sun, Nov 28, 2021 at 5:09 PM Jacob Block via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
Hi all,

I have read through pretty much every thread on this topic and unfortunately will be starting a new one. I am trying to upgrade an older IPA server that has had all the cert-pki-ca certs expired. Some other history, the initial master used to be on a VPS and was moved on-site several years ago by spinning up a replica on-site, promoting it to the new master, and shutting down the master. I am not entirely convinced there wasn't some issue also before the expired certs. There is also no other replica. I'd like to get this working, create a replica, and start upgrading to the latest.

# ipa --version
VERSION: 4.6.4, API_VERSION: 2.230

# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20190405192115':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-IPA-COMPANY-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-IPA-COMPANY-COM/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-IPA-COMPANY-COM',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-03-09 22:30:53 UTC
        dns: ipa.internal.company.com
        principal name: ldap/ipa.internal.company.com@IPA.COMPANY.COM
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv IPA-COMPANY-COM
        track: yes
        auto-renew: yes
Request ID '20190405192140':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-03-09 22:31:53 UTC
        dns: ipa.internal.company.com
        principal name: HTTP/ipa.internal.company.com@IPA.COMPANY.COM
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
Request ID '20190405192207':
        status: NEED_GUIDANCE
        stuck: yes
        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=IPA RA,O=IPA.COMPANY.COM
        expires: 2021-09-05 16:48:11 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
Request ID '20190405192208':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
        certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-03-09 22:30:44 UTC
        principal name: krbtgt/IPA.COMPANY.COM@IPA.COMPANY.COM
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-pkinit-KPKdc
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
        track: yes
        auto-renew: yes
Request ID '20190405204557':
        status: NEED_GUIDANCE
        stuck: yes
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=CA Audit,O=IPA.COMPANY.COM
        expires: 2021-09-05 16:48:31 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204558':
        status: GENERATING_CSR
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=OCSP Subsystem,O=IPA.COMPANY.COM
        expires: 2021-09-05 16:49:41 UTC
        eku: id-kp-OCSPSigning
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204559':
        status: NEED_GUIDANCE
        stuck: yes
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=CA Subsystem,O=IPA.COMPANY.COM
        expires: 2021-09-05 16:48:21 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204600':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=Certificate Authority,O=IPA.COMPANY.COM
        expires: 2041-09-01 05:41:44 UTC
        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190405204601':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
        subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
        expires: 2023-02-15 22:30:43 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
        track: yes
        auto-renew: yes

The renewal master used to be the remote VPS master that no longer exists. I've since updated that:

#  ipa config-show | grep renewal
  IPA CA renewal master: ipa.internal.company.com

One thing I am confused by is seeing four entries for "caSigningCert cert-pki-ca" (I also have a tenuous understanding of CAs and certs)

# certutil -L -d /var/lib/pki/pki-tomcat/alias/

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

subsystemCert cert-pki-ca                                    u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
DSTRootCAX3                                                  C,,
CN=R3,O=Let's Encrypt,C=US                                   C,,
CN=E1,O=Let's Encrypt,C=US                                   C,,
auditSigningCert cert-pki-ca                                 u,u,Pu
ocspSigningCert cert-pki-ca                                  u,u,u
Server-Cert cert-pki-ca                                      u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
ISRGRootCAX3                                                 C,,
ISRGRootCAX3                                                 C,,
ISRGRootCAX1                                                 C,,
CN=ISRG Root X2,O=Internet Security Research Group,C=US      C,,
CN=R4,O=Let's Encrypt,C=US                                   C,,
CN=E2,O=Let's Encrypt,C=US                                   C,,

I've tried rolling back the clock to before 2021-09-05 but pki-tomcatd still doesn't start:

Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore() begins
Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore(): tag=internaldb
Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore(): tag=replicationdb
Jun 01 05:15:45 ipa.internal.company.com server[919212]: Internal Database Error encountered: Could not connect to LDAP server host ipa.internal.company.com port 636 Error netscape.ldap.LDAPException: Unable to create socket: org.mozilla.jss.ssl.SSLSocketException: org.mozilla.jss.ssl.SSLSocketException: SSL_ForceHandshake failed: (-8172) Peer's certificate issuer has been marked as not trusted by the user. (-1)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@70aacdbc background process
Jun 01 05:15:55 ipa.internal.company.com server[919212]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at java.lang.Thread.run(Thread.java:748)

Maybe its pki certs + https certs are both having a problem? Maybe this is related to a recent LE CA?

Any thoughts would be greatly appreciated. Thank you!
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure