Expired (web/dirsv) 3rd party cert - pki-tomcatd unable to start - cannot update cert - IPA Error 4301: CertificateOperationError - help!
Hi
We have a pair of Freeipa (4.9.x) Rhel8 Freeipa servers.
Previously we had installed a 3rd party cert for httpd + dirsrv (only) - this expired recently. I was unable to login to ui . This issue however may not be connected with this. It appears to be linked to Tomcat -> LDAPS connectiopn ?? - error when trying to login was 'Login failed due to an unknown reason'
I could login if I changed server time to the past - but the certificates page is broken 'Certificate operation cannot be completed: Unable to communicate with CMS (503)' (time has been set back to normal now)
As a result I cannot renew my httpd/dirsv cert
Can anyone help me restore pki-tomcatd ? This may not be connected to web/dirsv cert expiry (and just be a coincidence)
If I try using
# ipa-server-certinstall --http --dirsrv ireland.idm.domain.uk.key ireland.idm.domain.uk.crt
I get
-----
Directory Manager password:
Enter private key unlock password:
cannot connect to 'https://london.idm.domain.uk:443/acme/directory': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) The ipa-server-certinstall command failed.
----
I can however install the cert to just the dirsv
--- [root@london mcox]# ipa-server-certinstall --dirsrv london.idm.domain.uk.key london.idm.domain.uk.crt Directory Manager password:
Enter private key unlock password:
Please restart ipa services after installing certificate (ipactl restart) ---
However after ipactl restart -> pki-tomcatd Service: STOPPED (all other services are working)
The main IPA system aside from this appears to work - i.e I can login and sudo to clients, and kinit, etc works
As a work-around I can login to the UI if I manually copy the cert/key to
/var/lib/ipa/certs/httpd.crt /var/lib/ipa/private/httpd.key
However the pki-tomcatd service is still down - I see these errors
- On certifcates tab : IPA Error 4301: CertificateOperationError - Certificate operation cannot be completed: Unable to communicate with CMS (503) - On Certificate authorities pages I see : Some operations failed -> details -> Failed to authenticate to CA REST API
pki-tomcatd logs show
------- Mar 11 12:54:10 london.idm.domain.uk systemd[1]: Starting PKI Tomcat Server pki-tomcat... Mar 11 12:54:15 london.idm.domain.uk server[509585]: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java Mar 11 12:54:15 london.idm.domain.uk server[509585]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar Mar 11 12:54:15 london.idm.domain.uk server[509585]: main class used: org.apache.catalina.startup.Bootstrap Mar 11 12:54:15 london.idm.domain.uk server[509585]: flags used: -Dcom.redhat.fips=false Mar 11 12:54:15 london.idm.domain.uk server[509585]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy Mar 11 12:54:15 london.idm.domain.uk server[509585]: arguments used: start Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: pki.client: /usr/libexec/ipa/ipa-pki-wait-running:63: The subsystem in PKIConnection.__init__() has been deprecated (https://www.dogtagpki.org/wiki/PKI_10.8_Python_Changes). Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Created connection http://london.idm.domain.uk:8080/ca Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb6c264e668>: Failed to establish a new connection: [Errno 111] Connection refused',)) Mar 11 12:54:17 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb6c264ea58>: Failed to establish a new connection: [Errno 111] Connection refused',)) Mar 11 12:54:18 london.idm.domain.uk server[509585]: WARNING: Some of the specified [protocols] are not supported by the SSL engine and have been skipped: [[TLSv1, TLSv1.1]] Mar 11 12:54:19 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) Mar 11 12:54:21 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: Context [/ca] startup failed due to previous errors Mar 11 12:54:23 london.idm.domain.uk server[509585]: WARNING: The web application [ca] appears to have started a thread named [LDAPConnThread-0 ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead0(Native Method) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead(SocketInputStream.java:116) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:171) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:141) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:127) Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method) Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505) Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.fill(BufferedInputStream.java:246) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.read(BufferedInputStream.java:265) Mar 11 12:54:23 london.idm.domain.uk server[509585]: netscape.ldap.ber.stream.BERElement.getElement(Unknown Source) Mar 11 12:54:23 london.idm.domain.uk server[509585]: netscape.ldap.LDAPConnThread.run(Unknown Source) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.lang.Thread.run(Thread.java:748) Mar 11 12:54:23 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: Context [/acme] startup failed due to previous errors Mar 11 12:54:24 london.idm.domain.uk server[509585]: WARNING: The web application [acme] appears to have started a thread named [LDAPConnThread-1 ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead0(Native Method) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead(SocketInputStream.java:116) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:171) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:141) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:127) Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method) Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505) Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.fill(BufferedInputStream.java:246) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.read(BufferedInputStream.java:265) Mar 11 12:54:24 london.idm.domain.uk server[509585]: netscape.ldap.ber.stream.BERElement.getElement(Unknown Source) Mar 11 12:54:24 london.idm.domain.uk server[509585]: netscape.ldap.LDAPConnThread.run(Unknown Source) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.lang.Thread.run(Thread.java:748) Mar 11 12:54:25 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus Mar 11 12:54:26 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus Mar 11 12:54:27 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus ... -------
Other logs show : (i've just added the main error - not entire java error
/var/log/pki/pki-tomcat/acme/debug.2022-03-11.log :
----- 12:34:01 [main] SEVERE: Exception sending context initialized event to listener instance of class [org.dogtagpki.acme.server.ACMEEngine] java.lang.RuntimeException: Unable to start ACME engine: Unable to connect to LDAP server: Authentication failed -----
/var/log/pki/pki-tomcat/ca/debug.2022-03-11.log :
----- 2022-03-11 12:33:59 [main] SEVERE: Unable to start CA engine: Unable to connect to LDAP server: Authentication failed Unable to connect to L2022-03-11 12:33:59 [main] INFO: Shutting down CA subsystem .... 2022-03-11 12:33:59 [main] SEVERE: Exception sending context destroyed event to listener instance of class [org.dogtagpki.server.ca.CAEngine] java.lang.NullPointerException DAP server: Authentication failed -----
I have checked this ->
# getcert list |grep expire expires: 2024-02-13 00:32:37 GMT expires: unknown expires: unknown expires: unknown expires: unknown expires: 2024-01-22 00:29:51 GMT expires: 2024-01-22 00:30:38 GMT
And I have ran ipa-healthcheck
I can see
---- Expired Cert: ocsp_signing Expired Cert: subsystem Expired Cert: audit_signing
Internal server error 503 Server Error: Service Unavailable for url: http://london.idm.domain.uk:80/ca/rest/securityDomain/domainInfo Internal server error HTTPSConnectionPool(host='london.idm.domain.uk', port=8443): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fc4e6c58198>: Failed to establish a new connection: [Errno 111] Connection refused',))
---
Also some expired certs
"source": "pki.server.healthcheck.certs.expiration", "check": "CASystemCertExpiryCheck", "result": "ERROR", "uuid": "36c8fbed-571d-4d38-9919-53322fea4aa2", "when": "20220311130832Z", "duration": "0.188329", "kw": { "cert_id": "ocsp_signing", "expiry_date": "Mar 01 2022", "msg": "Certificate has ALREADY EXPIRED" } }, { "source": "pki.server.healthcheck.certs.expiration", "check": "CASystemCertExpiryCheck", "result": "ERROR", "uuid": "195970e4-e2fd-4eca-aeac-f1e97e9c3b13", "when": "20220311130832Z", "duration": "0.360146", "kw": { "cert_id": "subsystem", "expiry_date": "Mar 01 2022", "msg": "Certificate has ALREADY EXPIRED" } }, { "source": "pki.server.healthcheck.certs.expiration", "check": "CASystemCertExpiryCheck", "result": "ERROR", "uuid": "a84a9bc5-de4d-4cdc-b7fd-41b83f3a11af", "when": "20220311130833Z", "duration": "0.454225", "kw": { "cert_id": "audit_signing", "expiry_date": "Mar 01 2022", "msg": "Certificate has ALREADY EXPIRED" }
I have attached the full output of healthcheck to : https://pastebin.com/xfNLR0Ja (domain name changed)
On the last ipa update there was also a issue with pki-tomcatd - i.e - I have to remove the block 'requiredSecret=' in /etc/pki/pki-tomcat/server.xml to fix it, this was however working for a month or so after .
Any help to troubleshooting this would be welcomed
Thanks
Hi,
it looks like some of the certificates used by PKI are also expired (they are stored in /etc/pki/pki-tomcat/alias). Since you're running IPA 4.9, you can use the command ipa-cert-fix. Please read the man page with extra care, it recommends to backup certificates and keys before you proceed. You mentioned having a pair of IPA servers, do they both have expired certificates? If one of them is good, there are also other options to retrieve the renewed certificates from the good server and install them on the other one (the 3 certs ocspSigningCert, subsystemCert and auditSigningCert are shared on all the CA instances).
flo
On Fri, Mar 11, 2022 at 2:36 PM Morgan Cox via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
Hi
We have a pair of Freeipa (4.9.x) Rhel8 Freeipa servers.
Previously we had installed a 3rd party cert for httpd + dirsrv (only) - this expired recently. I was unable to login to ui . This issue however may not be connected with this. It appears to be linked to Tomcat -> LDAPS connectiopn ?? - error when trying to login was 'Login failed due to an unknown reason'
I could login if I changed server time to the past - but the certificates page is broken 'Certificate operation cannot be completed: Unable to communicate with CMS (503)' (time has been set back to normal now)
As a result I cannot renew my httpd/dirsv cert
Can anyone help me restore pki-tomcatd ? This may not be connected to web/dirsv cert expiry (and just be a coincidence)
If I try using
# ipa-server-certinstall --http --dirsrv ireland.idm.domain.uk.key ireland.idm.domain.uk.crt
I get
Directory Manager password:
Enter private key unlock password:
cannot connect to 'https://london.idm.domain.uk:443/acme/directory': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) The ipa-server-certinstall command failed.
I can however install the cert to just the dirsv
[root@london mcox]# ipa-server-certinstall --dirsrv london.idm.domain.uk.key london.idm.domain.uk.crt Directory Manager password:
Enter private key unlock password:
Please restart ipa services after installing certificate (ipactl restart)
However after ipactl restart -> pki-tomcatd Service: STOPPED (all other services are working)
The main IPA system aside from this appears to work - i.e I can login and sudo to clients, and kinit, etc works
As a work-around I can login to the UI if I manually copy the cert/key to
/var/lib/ipa/certs/httpd.crt /var/lib/ipa/private/httpd.key
However the pki-tomcatd service is still down - I see these errors
- On certifcates tab : IPA Error 4301: CertificateOperationError -
Certificate operation cannot be completed: Unable to communicate with CMS (503)
- On Certificate authorities pages I see : Some operations failed ->
details -> Failed to authenticate to CA REST API
pki-tomcatd logs show
Mar 11 12:54:10 london.idm.domain.uk systemd[1]: Starting PKI Tomcat Server pki-tomcat... Mar 11 12:54:15 london.idm.domain.uk server[509585]: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java Mar 11 12:54:15 london.idm.domain.uk server[509585]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar Mar 11 12:54:15 london.idm.domain.uk server[509585]: main class used: org.apache.catalina.startup.Bootstrap Mar 11 12:54:15 london.idm.domain.uk server[509585]: flags used: -Dcom.redhat.fips=false Mar 11 12:54:15 london.idm.domain.uk server[509585]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy Mar 11 12:54:15 london.idm.domain.uk server[509585]: arguments used: start Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: pki.client: /usr/libexec/ipa/ipa-pki-wait-running:63: The subsystem in PKIConnection.__init__() has been deprecated ( https://www.dogtagpki.org/wiki/PKI_10.8_Python_Changes). Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Created connection http://london.idm.domain.uk:8080/ca Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' london.idm.domain.uk', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb6c264e668>: Failed to establish a new connection: [Errno 111] Connection refused',)) Mar 11 12:54:17 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' london.idm.domain.uk', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb6c264ea58>: Failed to establish a new connection: [Errno 111] Connection refused',)) Mar 11 12:54:18 london.idm.domain.uk server[509585]: WARNING: Some of the specified [protocols] are not supported by the SSL engine and have been skipped: [[TLSv1, TLSv1.1]] Mar 11 12:54:19 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) Mar 11 12:54:21 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: Context [/ca] startup failed due to previous errors Mar 11 12:54:23 london.idm.domain.uk server[509585]: WARNING: The web application [ca] appears to have started a thread named [LDAPConnThread-0 ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead0(Native Method) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead(SocketInputStream.java:116) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:171) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:141) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:127) Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method) Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505) Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.fill(BufferedInputStream.java:246) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.read(BufferedInputStream.java:265) Mar 11 12:54:23 london.idm.domain.uk server[509585]: netscape.ldap.ber.stream.BERElement.getElement(Unknown Source) Mar 11 12:54:23 london.idm.domain.uk server[509585]: netscape.ldap.LDAPConnThread.run(Unknown Source) Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.lang.Thread.run(Thread.java:748) Mar 11 12:54:23 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: Context [/acme] startup failed due to previous errors Mar 11 12:54:24 london.idm.domain.uk server[509585]: WARNING: The web application [acme] appears to have started a thread named [LDAPConnThread-1 ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead0(Native Method) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead(SocketInputStream.java:116) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:171) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:141) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:127) Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method) Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505) Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.fill(BufferedInputStream.java:246) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.read(BufferedInputStream.java:265) Mar 11 12:54:24 london.idm.domain.uk server[509585]: netscape.ldap.ber.stream.BERElement.getElement(Unknown Source) Mar 11 12:54:24 london.idm.domain.uk server[509585]: netscape.ldap.LDAPConnThread.run(Unknown Source) Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.lang.Thread.run(Thread.java:748) Mar 11 12:54:25 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus Mar 11 12:54:26 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus Mar 11 12:54:27 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus ...
Other logs show : (i've just added the main error - not entire java error
/var/log/pki/pki-tomcat/acme/debug.2022-03-11.log :
12:34:01 [main] SEVERE: Exception sending context initialized event to listener instance of class [org.dogtagpki.acme.server.ACMEEngine] java.lang.RuntimeException: Unable to start ACME engine: Unable to connect to LDAP server: Authentication failed
/var/log/pki/pki-tomcat/ca/debug.2022-03-11.log :
2022-03-11 12:33:59 [main] SEVERE: Unable to start CA engine: Unable to connect to LDAP server: Authentication failed Unable to connect to L2022-03-11 12:33:59 [main] INFO: Shutting down CA subsystem .... 2022-03-11 12:33:59 [main] SEVERE: Exception sending context destroyed event to listener instance of class [org.dogtagpki.server.ca.CAEngine] java.lang.NullPointerException DAP server: Authentication failed
I have checked this ->
# getcert list |grep expire expires: 2024-02-13 00:32:37 GMT expires: unknown expires: unknown expires: unknown expires: unknown expires: 2024-01-22 00:29:51 GMT expires: 2024-01-22 00:30:38 GMT
And I have ran ipa-healthcheck
I can see
Expired Cert: ocsp_signing Expired Cert: subsystem Expired Cert: audit_signing
Internal server error 503 Server Error: Service Unavailable for url: http://london.idm.domain.uk:80/ca/rest/securityDomain/domainInfo Internal server error HTTPSConnectionPool(host='london.idm.domain.uk', port=8443): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fc4e6c58198>: Failed to establish a new connection: [Errno 111] Connection refused',))
Also some expired certs
"source": "pki.server.healthcheck.certs.expiration", "check": "CASystemCertExpiryCheck", "result": "ERROR", "uuid": "36c8fbed-571d-4d38-9919-53322fea4aa2", "when": "20220311130832Z", "duration": "0.188329", "kw": { "cert_id": "ocsp_signing", "expiry_date": "Mar 01 2022", "msg": "Certificate has ALREADY EXPIRED" }}, { "source": "pki.server.healthcheck.certs.expiration", "check": "CASystemCertExpiryCheck", "result": "ERROR", "uuid": "195970e4-e2fd-4eca-aeac-f1e97e9c3b13", "when": "20220311130832Z", "duration": "0.360146", "kw": { "cert_id": "subsystem", "expiry_date": "Mar 01 2022", "msg": "Certificate has ALREADY EXPIRED" } }, { "source": "pki.server.healthcheck.certs.expiration", "check": "CASystemCertExpiryCheck", "result": "ERROR", "uuid": "a84a9bc5-de4d-4cdc-b7fd-41b83f3a11af", "when": "20220311130833Z", "duration": "0.454225", "kw": { "cert_id": "audit_signing", "expiry_date": "Mar 01 2022", "msg": "Certificate has ALREADY EXPIRED" }
I have attached the full output of healthcheck to : https://pastebin.com/xfNLR0Ja (domain name changed)
On the last ipa update there was also a issue with pki-tomcatd - i.e - I have to remove the block 'requiredSecret=' in /etc/pki/pki-tomcat/server.xml to fix it, this was however working for a month or so after .
Any help to troubleshooting this would be welcomed
Thanks _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Morgan Cox