[Freeipa-users] Re: Expired (web/dirsv) 3rd party cert - pki-tomcatd unable to start - cannot update cert - IPA Error 4301: CertificateOperationError - help!

Hi We have a pair of Freeipa (4.9.x) Rhel8 Freeipa servers. Previously we had installed a 3rd party cert for httpd + dirsrv (only) - this expired recently. I was unable to login to ui . This issue however may not be connected with this. It appears to be linked to Tomcat -> LDAPS connectiopn ?? - error when trying to login was 'Login failed due to an unknown reason' I could login if I changed server time to the past - but the certificates page is broken 'Certificate operation cannot be completed: Unable to communicate with CMS (503)' (time has been set back to normal now) As a result I cannot renew my httpd/dirsv cert Can anyone help me restore pki-tomcatd ? This may not be connected to web/dirsv cert expiry (and just be a coincidence) If I try using # ipa-server-certinstall --http --dirsrv ireland.idm.domain.uk.key ireland.idm.domain.uk.crt I get ----- Directory Manager password: Enter private key unlock password: cannot connect to 'https://london.idm.domain.uk:443/acme/directory': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) The ipa-server-certinstall command failed. ---- I can however install the cert to just the dirsv --- [root@london mcox]# ipa-server-certinstall --dirsrv london.idm.domain.uk.key london.idm.domain.uk.crt Directory Manager password: Enter private key unlock password: Please restart ipa services after installing certificate (ipactl restart) --- However after ipactl restart -> pki-tomcatd Service: STOPPED (all other services are working) The main IPA system aside from this appears to work - i.e I can login and sudo to clients, and kinit, etc works As a work-around I can login to the UI if I manually copy the cert/key to /var/lib/ipa/certs/httpd.crt /var/lib/ipa/private/httpd.key However the pki-tomcatd service is still down - I see these errors - On certifcates tab : IPA Error 4301: CertificateOperationError - Certificate operation cannot be completed: Unable to communicate with CMS (503) - On Certificate authorities pages I see : Some operations failed -> details -> Failed to authenticate to CA REST API pki-tomcatd logs show ------- Mar 11 12:54:10 london.idm.domain.uk systemd[1]: Starting PKI Tomcat Server pki-tomcat... Mar 11 12:54:15 london.idm.domain.uk server[509585]: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java Mar 11 12:54:15 london.idm.domain.uk server[509585]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar Mar 11 12:54:15 london.idm.domain.uk server[509585]: main class used: org.apache.catalina.startup.Bootstrap Mar 11 12:54:15 london.idm.domain.uk server[509585]: flags used: -Dcom.redhat.fips=false Mar 11 12:54:15 london.idm.domain.uk server[509585]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy Mar 11 12:54:15 london.idm.domain.uk server[509585]: arguments used: start Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: pki.client: /usr/libexec/ipa/ipa-pki-wait-running:63: The subsystem in PKIConnection.__init__() has been deprecated ( https://www.dogtagpki.org/wiki/PKI_10.8_Python_Changes). Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Created connection http://london.idm.domain.uk:8080/ca Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' london.idm.domain.uk', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb6c264e668>: Failed to establish a new connection: [Errno 111] Connection refused',)) Mar 11 12:54:17 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' london.idm.domain.uk', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb6c264ea58>: Failed to establish a new connection: [Errno 111] Connection refused',)) Mar 11 12:54:18 london.idm.domain.uk server[509585]: WARNING: Some of the specified [protocols] are not supported by the SSL engine and have been skipped: [[TLSv1, TLSv1.1]] Mar 11 12:54:19 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) Mar 11 12:54:21 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: Context [/ca] startup failed due to previous errors Mar 11 12:54:23 london.idm.domain.uk server[509585]: WARNING: The web application [ca] appears to have started a thread named [LDAPConnThread-0 ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead0(Native Method) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead(SocketInputStream.java:116) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:171) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:141) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:127) Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method) Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505) Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.fill(BufferedInputStream.java:246) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.read(BufferedInputStream.java:265) Mar 11 12:54:23 london.idm.domain.uk server[509585]: netscape.ldap.ber.stream.BERElement.getElement(Unknown Source) Mar 11 12:54:23 london.idm.domain.uk server[509585]: netscape.ldap.LDAPConnThread.run(Unknown Source) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.lang.Thread.run(Thread.java:748) Mar 11 12:54:23 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: Context [/acme] startup failed due to previous errors Mar 11 12:54:24 london.idm.domain.uk server[509585]: WARNING: The web application [acme] appears to have started a thread named [LDAPConnThread-1 ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead0(Native Method) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead(SocketInputStream.java:116) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:171) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:141) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:127) Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method) Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505) Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.fill(BufferedInputStream.java:246) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.read(BufferedInputStream.java:265) Mar 11 12:54:24 london.idm.domain.uk server[509585]: netscape.ldap.ber.stream.BERElement.getElement(Unknown Source) Mar 11 12:54:24 london.idm.domain.uk server[509585]: netscape.ldap.LDAPConnThread.run(Unknown Source) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.lang.Thread.run(Thread.java:748) Mar 11 12:54:25 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus Mar 11 12:54:26 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus Mar 11 12:54:27 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus ... ------- Other logs show : (i've just added the main error - not entire java error /var/log/pki/pki-tomcat/acme/debug.2022-03-11.log : ----- 12:34:01 [main] SEVERE: Exception sending context initialized event to listener instance of class [org.dogtagpki.acme.server.ACMEEngine] java.lang.RuntimeException: Unable to start ACME engine: Unable to connect to LDAP server: Authentication failed ----- /var/log/pki/pki-tomcat/ca/debug.2022-03-11.log : ----- 2022-03-11 12:33:59 [main] SEVERE: Unable to start CA engine: Unable to connect to LDAP server: Authentication failed Unable to connect to L2022-03-11 12:33:59 [main] INFO: Shutting down CA subsystem .... 2022-03-11 12:33:59 [main] SEVERE: Exception sending context destroyed event to listener instance of class [org.dogtagpki.server.ca.CAEngine] java.lang.NullPointerException DAP server: Authentication failed ----- I have checked this -> # getcert list |grep expire expires: 2024-02-13 00:32:37 GMT expires: unknown expires: unknown expires: unknown expires: unknown expires: 2024-01-22 00:29:51 GMT expires: 2024-01-22 00:30:38 GMT And I have ran ipa-healthcheck I can see ---- Expired Cert: ocsp_signing Expired Cert: subsystem Expired Cert: audit_signing Internal server error 503 Server Error: Service Unavailable for url: http://london.idm.domain.uk:80/ca/rest/securityDomain/domainInfo Internal server error HTTPSConnectionPool(host='london.idm.domain.uk', port=8443): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fc4e6c58198>: Failed to establish a new connection: [Errno 111] Connection refused',)) --- Also some expired certs "source": "pki.server.healthcheck.certs.expiration", "check": "CASystemCertExpiryCheck", "result": "ERROR", "uuid": "36c8fbed-571d-4d38-9919-53322fea4aa2", "when": "20220311130832Z", "duration": "0.188329", "kw": { "cert_id": "ocsp_signing", "expiry_date": "Mar 01 2022", "msg": "Certificate has ALREADY EXPIRED" } }, { "source": "pki.server.healthcheck.certs.expiration", "check": "CASystemCertExpiryCheck", "result": "ERROR", "uuid": "195970e4-e2fd-4eca-aeac-f1e97e9c3b13", "when": "20220311130832Z", "duration": "0.360146", "kw": { "cert_id": "subsystem", "expiry_date": "Mar 01 2022", "msg": "Certificate has ALREADY EXPIRED" } }, { "source": "pki.server.healthcheck.certs.expiration", "check": "CASystemCertExpiryCheck", "result": "ERROR", "uuid": "a84a9bc5-de4d-4cdc-b7fd-41b83f3a11af", "when": "20220311130833Z", "duration": "0.454225", "kw": { "cert_id": "audit_signing", "expiry_date": "Mar 01 2022", "msg": "Certificate has ALREADY EXPIRED" } I have attached the full output of healthcheck to : https://pastebin.com/xfNLR0Ja (domain name changed) On the last ipa update there was also a issue with pki-tomcatd - i.e - I have to remove the block 'requiredSecret=' in /etc/pki/pki-tomcat/server.xml to fix it, this was however working for a month or so after . Any help to troubleshooting this would be welcomed Thanks _______________________________________________ FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure