Hi
We have a pair of Freeipa (4.9.x) Rhel8 Freeipa servers.
Previously we had installed a 3rd party cert for httpd + dirsrv (only) - this expired recently. I was unable to login to ui . This issue however may not be connected with this. It appears to be linked to Tomcat -> LDAPS connectiopn ?? - error when trying to login was 'Login failed due to an unknown reason'
I could login if I changed server time to the past - but the certificates page is broken 'Certificate operation cannot be completed: Unable to communicate with CMS (503)' (time has been set back to normal now)
As a result I cannot renew my httpd/dirsv cert
Can anyone help me restore pki-tomcatd ? This may not be connected to web/dirsv cert expiry (and just be a coincidence)
If I try using
# ipa-server-certinstall --http --dirsrv ireland.idm.domain.uk.key ireland.idm.domain.uk.crt
I get
-----
Directory Manager password:
Enter private key unlock password:
cannot connect to 'https://london.idm.domain.uk:443/acme/directory': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
The ipa-server-certinstall command failed.
----
I can however install the cert to just the dirsv
---
[root@london mcox]# ipa-server-certinstall --dirsrv london.idm.domain.uk.key london.idm.domain.uk.crt
Directory Manager password:
Enter private key unlock password:
Please restart ipa services after installing certificate (ipactl restart)
---
However after ipactl restart -> pki-tomcatd Service: STOPPED (all other services are working)
The main IPA system aside from this appears to work - i.e I can login and sudo to clients, and kinit, etc works
As a work-around I can login to the UI if I manually copy the cert/key to
/var/lib/ipa/certs/httpd.crt
/var/lib/ipa/private/httpd.key
However the pki-tomcatd service is still down - I see these errors
- On certifcates tab : IPA Error 4301: CertificateOperationError - Certificate operation cannot be completed: Unable to communicate with CMS (503)
- On Certificate authorities pages I see : Some operations failed -> details -> Failed to authenticate to CA REST API
pki-tomcatd logs show
-------
Mar 11 12:54:10 london.idm.domain.uk systemd[1]: Starting PKI Tomcat Server pki-tomcat...
Mar 11 12:54:15 london.idm.domain.uk server[509585]: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
Mar 11 12:54:15 london.idm.domain.uk server[509585]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar
Mar 11 12:54:15 london.idm.domain.uk server[509585]: main class used: org.apache.catalina.startup.Bootstrap
Mar 11 12:54:15 london.idm.domain.uk server[509585]: flags used: -Dcom.redhat.fips=false
Mar 11 12:54:15 london.idm.domain.uk server[509585]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
Mar 11 12:54:15 london.idm.domain.uk server[509585]: arguments used: start
Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: pki.client: /usr/libexec/ipa/ipa-pki-wait-running:63: The subsystem in PKIConnection.__init__() has been deprecated (https://www.dogtagpki.org/wiki/PKI_10.8_Python_Changes).
Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Created connection http://london.idm.domain.uk:8080/ca
Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb6c264e668>: Failed to establish a new connection: [Errno 111] Connection refused',))
Mar 11 12:54:17 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb6c264ea58>: Failed to establish a new connection: [Errno 111] Connection refused',))
Mar 11 12:54:18 london.idm.domain.uk server[509585]: WARNING: Some of the specified [protocols] are not supported by the SSL engine and have been skipped: [[TLSv1, TLSv1.1]]
Mar 11 12:54:19 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0)
Mar 11 12:54:21 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file
Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: Context [/ca] startup failed due to previous errors
Mar 11 12:54:23 london.idm.domain.uk server[509585]: WARNING: The web application [ca] appears to have started a thread named [LDAPConnThread-0 ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead0(Native Method)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:171)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:141)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:127)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: netscape.ldap.ber.stream.BERElement.getElement(Unknown Source)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: netscape.ldap.LDAPConnThread.run(Unknown Source)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: java.lang.Thread.run(Thread.java:748)
Mar 11 12:54:23 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file
Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: Context [/acme] startup failed due to previous errors
Mar 11 12:54:24 london.idm.domain.uk server[509585]: WARNING: The web application [acme] appears to have started a thread named [LDAPConnThread-1 ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead0(Native Method)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:171)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:141)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.net.SocketInputStream.read(SocketInputStream.java:127)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: netscape.ldap.ber.stream.BERElement.getElement(Unknown Source)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: netscape.ldap.LDAPConnThread.run(Unknown Source)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: java.lang.Thread.run(Thread.java:748)
Mar 11 12:54:25 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus
Mar 11 12:54:26 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus
Mar 11 12:54:27 london.idm.domain.uk ipa-pki-wait-running[509586]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus
...
-------
Other logs show : (i've just added the main error - not entire java error
/var/log/pki/pki-tomcat/acme/debug.2022-03-11.log :
-----
12:34:01 [main] SEVERE: Exception sending context initialized event to listener instance of class [org.dogtagpki.acme.server.ACMEEngine]
java.lang.RuntimeException: Unable to start ACME engine: Unable to connect to LDAP server: Authentication failed
-----
/var/log/pki/pki-tomcat/ca/debug.2022-03-11.log :
-----
2022-03-11 12:33:59 [main] SEVERE: Unable to start CA engine: Unable to connect to LDAP server: Authentication failed
Unable to connect to L2022-03-11 12:33:59 [main] INFO: Shutting down CA subsystem
....
2022-03-11 12:33:59 [main] SEVERE: Exception sending context destroyed event to listener instance of class [org.dogtagpki.server.ca.CAEngine]
java.lang.NullPointerException
DAP server: Authentication failed
-----
I have checked this ->
# getcert list |grep expire
expires: 2024-02-13 00:32:37 GMT
expires: unknown
expires: unknown
expires: unknown
expires: unknown
expires: 2024-01-22 00:29:51 GMT
expires: 2024-01-22 00:30:38 GMT
And I have ran ipa-healthcheck
I can see
----
Expired Cert: ocsp_signing
Expired Cert: subsystem
Expired Cert: audit_signing
Internal server error 503 Server Error: Service Unavailable for url: http://london.idm.domain.uk:80/ca/rest/securityDomain/domainInfo
Internal server error HTTPSConnectionPool(host='london.idm.domain.uk', port=8443): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fc4e6c58198>: Failed to establish a new connection: [Errno 111] Connection refused',))
---
Also some expired certs
"source": "pki.server.healthcheck.certs.expiration",
"check": "CASystemCertExpiryCheck",
"result": "ERROR",
"uuid": "36c8fbed-571d-4d38-9919-53322fea4aa2",
"when": "20220311130832Z",
"duration": "0.188329",
"kw": {
"cert_id": "ocsp_signing",
"expiry_date": "Mar 01 2022",
"msg": "Certificate has ALREADY EXPIRED"
}
},
{
"source": "pki.server.healthcheck.certs.expiration",
"check": "CASystemCertExpiryCheck",
"result": "ERROR",
"uuid": "195970e4-e2fd-4eca-aeac-f1e97e9c3b13",
"when": "20220311130832Z",
"duration": "0.360146",
"kw": {
"cert_id": "subsystem",
"expiry_date": "Mar 01 2022",
"msg": "Certificate has ALREADY EXPIRED"
}
},
{
"source": "pki.server.healthcheck.certs.expiration",
"check": "CASystemCertExpiryCheck",
"result": "ERROR",
"uuid": "a84a9bc5-de4d-4cdc-b7fd-41b83f3a11af",
"when": "20220311130833Z",
"duration": "0.454225",
"kw": {
"cert_id": "audit_signing",
"expiry_date": "Mar 01 2022",
"msg": "Certificate has ALREADY EXPIRED"
}
I have attached the full output of healthcheck to : https://pastebin.com/xfNLR0Ja (domain name changed)
On the last ipa update there was also a issue with pki-tomcatd - i.e - I have to remove the block 'requiredSecret=' in /etc/pki/pki-tomcat/server.xml to fix it, this was however working for a month or so after .
Any help to troubleshooting this would be welcomed
Thanks
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure