Hi,

it looks like some of the certificates used by PKI are also expired (they are stored in /etc/pki/pki-tomcat/alias). Since you're running IPA 4.9, you can use the command ipa-cert-fix. Please read the man page with extra care, it recommends to backup certificates and keys before you proceed.
You mentioned having a pair of IPA servers, do they both have expired certificates? If one of them is good, there are also other options to retrieve the renewed certificates from the good server and install them on the other one (the 3 certs ocspSigningCert, subsystemCert and auditSigningCert are shared on all the CA instances).

flo

On Fri, Mar 11, 2022 at 2:36 PM Morgan Cox via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:


Hi

We have a pair of Freeipa (4.9.x) Rhel8 Freeipa servers.

Previously we had installed a 3rd party cert for httpd + dirsrv (only) - this expired recently. I was unable to login to ui . This issue however may not be connected with this. It appears to be linked to Tomcat -> LDAPS connectiopn ?? - error when trying to login was  'Login failed due to an unknown reason'

I could login if I changed server time to the past - but the certificates page is broken 'Certificate operation cannot be completed: Unable to communicate with CMS (503)' (time has been set back to normal now)

As a result I cannot renew my httpd/dirsv cert

Can anyone help me restore pki-tomcatd ? This may not be connected to web/dirsv cert expiry (and just be a coincidence)

If I try using

# ipa-server-certinstall --http --dirsrv ireland.idm.domain.uk.key ireland.idm.domain.uk.crt

I get

-----

Directory Manager password:

Enter private key unlock password:

cannot connect to 'https://london.idm.domain.uk:443/acme/directory': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
The ipa-server-certinstall command failed.

----

I can however install the cert to just the dirsv

---
[root@london mcox]# ipa-server-certinstall --dirsrv london.idm.domain.uk.key london.idm.domain.uk.crt
Directory Manager password:

Enter private key unlock password:


Please restart ipa services after installing certificate (ipactl restart)
---

However after ipactl restart -> pki-tomcatd Service: STOPPED (all other services are working)

The main IPA system aside from this appears to work - i.e I can login and sudo to clients, and kinit, etc works

As a work-around I can login to the UI if I manually copy the cert/key to


/var/lib/ipa/certs/httpd.crt
/var/lib/ipa/private/httpd.key

However the pki-tomcatd service is still down - I see these errors

- On certifcates tab : IPA Error 4301: CertificateOperationError - Certificate operation cannot be completed: Unable to communicate with CMS (503)
- On Certificate authorities pages I see : Some operations failed -> details ->     Failed to authenticate to CA REST API

pki-tomcatd logs show

-------
Mar 11 12:54:10 london.idm.domain.uk systemd[1]: Starting PKI Tomcat Server pki-tomcat...
Mar 11 12:54:15 london.idm.domain.uk server[509585]: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
Mar 11 12:54:15 london.idm.domain.uk server[509585]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar
Mar 11 12:54:15 london.idm.domain.uk server[509585]: main class used: org.apache.catalina.startup.Bootstrap
Mar 11 12:54:15 london.idm.domain.uk server[509585]: flags used: -Dcom.redhat.fips=false
Mar 11 12:54:15 london.idm.domain.uk server[509585]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager     -Djava.security.manager     -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
Mar 11 12:54:15 london.idm.domain.uk server[509585]: arguments used: start
Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: pki.client: /usr/libexec/ipa/ipa-pki-wait-running:63: The subsystem in PKIConnection.__init__() has been deprecated (https://www.dogtagpki.org/wiki/PKI_10.8_Python_Changes).
Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Created connection http://london.idm.domain.uk:8080/ca
Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb6c264e668>: Failed to establish a new connection: [Errno 111] Connection refused',))
Mar 11 12:54:17 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb6c264ea58>: Failed to establish a new connection: [Errno 111] Connection refused',))
Mar 11 12:54:18 london.idm.domain.uk server[509585]: WARNING: Some of the specified [protocols] are not supported by the SSL engine and have been skipped: [[TLSv1, TLSv1.1]]
Mar 11 12:54:19 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0)
Mar 11 12:54:21 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file
Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: Context [/ca] startup failed due to previous errors
Mar 11 12:54:23 london.idm.domain.uk server[509585]: WARNING: The web application [ca] appears to have started a thread named [LDAPConnThread-0 ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  java.net.SocketInputStream.socketRead0(Native Method)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  java.net.SocketInputStream.read(SocketInputStream.java:171)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  java.net.SocketInputStream.read(SocketInputStream.java:141)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  java.net.SocketInputStream.read(SocketInputStream.java:127)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  netscape.ldap.ber.stream.BERElement.getElement(Unknown Source)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  netscape.ldap.LDAPConnThread.run(Unknown Source)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:  java.lang.Thread.run(Thread.java:748)
Mar 11 12:54:23 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file
Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: Context [/acme] startup failed due to previous errors
Mar 11 12:54:24 london.idm.domain.uk server[509585]: WARNING: The web application [acme] appears to have started a thread named [LDAPConnThread-1 ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  java.net.SocketInputStream.socketRead0(Native Method)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  java.net.SocketInputStream.read(SocketInputStream.java:171)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  java.net.SocketInputStream.read(SocketInputStream.java:141)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  java.net.SocketInputStream.read(SocketInputStream.java:127)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  netscape.ldap.ber.stream.BERElement.getElement(Unknown Source)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  netscape.ldap.LDAPConnThread.run(Unknown Source)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:  java.lang.Thread.run(Thread.java:748)
Mar 11 12:54:25 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error:  for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus
Mar 11 12:54:26 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error:  for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus
Mar 11 12:54:27 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error:  for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus
...
-------

Other logs show : (i've just added the main error - not entire java error

/var/log/pki/pki-tomcat/acme/debug.2022-03-11.log :

-----
 12:34:01 [main] SEVERE: Exception sending context initialized event to listener instance of class [org.dogtagpki.acme.server.ACMEEngine]
java.lang.RuntimeException: Unable to start ACME engine: Unable to connect to LDAP server: Authentication failed
-----

/var/log/pki/pki-tomcat/ca/debug.2022-03-11.log :

-----
2022-03-11 12:33:59 [main] SEVERE: Unable to start CA engine: Unable to connect to LDAP server: Authentication failed
Unable to connect to L2022-03-11 12:33:59 [main] INFO: Shutting down CA subsystem
....
2022-03-11 12:33:59 [main] SEVERE: Exception sending context destroyed event to listener instance of class [org.dogtagpki.server.ca.CAEngine]
java.lang.NullPointerException
DAP server: Authentication failed
-----

I have checked this ->

#  getcert list  |grep expire
        expires: 2024-02-13 00:32:37 GMT
        expires: unknown
        expires: unknown
        expires: unknown
        expires: unknown
        expires: 2024-01-22 00:29:51 GMT
        expires: 2024-01-22 00:30:38 GMT



And I have ran ipa-healthcheck

I can see

----
Expired Cert: ocsp_signing
Expired Cert: subsystem
Expired Cert: audit_signing

Internal server error 503 Server Error: Service Unavailable for url: http://london.idm.domain.uk:80/ca/rest/securityDomain/domainInfo
Internal server error HTTPSConnectionPool(host='london.idm.domain.uk', port=8443): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fc4e6c58198>: Failed to establish a new connection: [Errno 111] Connection refused',))

---

Also some expired certs

    "source": "pki.server.healthcheck.certs.expiration",
    "check": "CASystemCertExpiryCheck",
    "result": "ERROR",
    "uuid": "36c8fbed-571d-4d38-9919-53322fea4aa2",
    "when": "20220311130832Z",
    "duration": "0.188329",
    "kw": {
      "cert_id": "ocsp_signing",
      "expiry_date": "Mar 01 2022",
      "msg": "Certificate has ALREADY EXPIRED"
    }
  },
  {
    "source": "pki.server.healthcheck.certs.expiration",
    "check": "CASystemCertExpiryCheck",
    "result": "ERROR",
    "uuid": "195970e4-e2fd-4eca-aeac-f1e97e9c3b13",
    "when": "20220311130832Z",
    "duration": "0.360146",
    "kw": {
      "cert_id": "subsystem",
      "expiry_date": "Mar 01 2022",
      "msg": "Certificate has ALREADY EXPIRED"
    }
  },
  {
    "source": "pki.server.healthcheck.certs.expiration",
    "check": "CASystemCertExpiryCheck",
    "result": "ERROR",
    "uuid": "a84a9bc5-de4d-4cdc-b7fd-41b83f3a11af",
    "when": "20220311130833Z",
    "duration": "0.454225",
    "kw": {
      "cert_id": "audit_signing",
      "expiry_date": "Mar 01 2022",
      "msg": "Certificate has ALREADY EXPIRED"
    }


I have attached the full output of healthcheck to : https://pastebin.com/xfNLR0Ja  (domain name changed)

On the last ipa update there was also a issue with pki-tomcatd - i.e - I have to remove the block 'requiredSecret=' in /etc/pki/pki-tomcat/server.xml to fix it, this was however working for a month or so after .

Any help to troubleshooting this would be welcomed

Thanks
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure