Manuel Gujo via FreeIPA-users wrote:
If the CA isn't running then there is no point in resubmitting the certmonger requests. It is guaranteed to fail with UNREACHABLE.
Check the journalctl output and the other logs, like catalina, in /var/log/pki/pki-tomcat for more information on why it failed to start.
Is this host memory-constrained? How much RAM does it have?
rob
there's new log on debug. Catalina does not log anything (0kb per file). in debug:
Could not connect to LDAP server host ipa1.itec.lab port 636 Error netscape.ldap.LDAPException: Unable to create socket: org.mozilla.jss.ssl.SSLSocketException: org.mozilla.jss.ssl.SSLSocketExc eption: SSL_ForceHandshake failed: (-8181) Peer's Certificate has expired. (-1)
in "system" logs says the same thing of debugs'
When I try to run 'ipactl start' without -f option, it says this: # ipactl start IPA version error: data needs to be upgraded (expected version '4.6.8-5.el7.centos', current version '4.4.0-14.el7.centos.4')
then after a while it fails and in /var/log/ipaupgrade.log says:
2020-11-17T18:25:05Z DEBUG httplib request failed: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 220, in _httplib_request conn.request(method, path, body=request_body, headers=headers) File "/usr/lib64/python2.7/httplib.py", line 1056, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1090, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 1052, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 890, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 852, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 1266, in connect HTTPConnection.connect(self) File "/usr/lib64/python2.7/httplib.py", line 833, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused 2020-11-17T18:25:05Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2020-11-17T18:25:05Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2176, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2059, in upgrade_configuration cainstance.repair_profile_caIPAserviceCert() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1949, in repair_profile_caIPAserviceCert with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1311, in __enter__ method='GET' File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 167, in https_request method=method, headers=headers) File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 229, in _httplib_request raise NetworkError(uri=uri, error=str(e))
2020-11-17T18:25:05Z DEBUG The ipa-server-upgrade command failed, exception: NetworkError: cannot connect to 'https://ipa1.itec.lab:8443/ca/rest/account/login': [Errno 111] Connection refused 2020-11-17T18:25:05Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
After this run, I noticed that some of the certs went on Monitoring state
# getcert list Number of certificates and requests being tracked: 9. Request ID '20191231201955': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: SelfSign issuer: CN=ipa1.itec.lab,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2022-02-08 15:59:12 UTC principal name: krbtgt/ITEC.LAB@ITEC.LAB certificate template/profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20201117182331': status: CA_UNREACHABLE ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=CA Audit,O=ITEC.LAB expires: 2020-12-08 09:35:14 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182333': status: CA_UNREACHABLE ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=OCSP Subsystem,O=ITEC.LAB expires: 2020-12-08 09:38:07 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182335': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=CA Subsystem,O=ITEC.LAB expires: 2022-11-07 18:24:47 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182336': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=Certificate Authority,O=ITEC.LAB expires: 2037-01-25 14:22:25 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182338': status: CA_UNREACHABLE ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=IPA RA,O=ITEC.LAB expires: 2020-12-08 09:37:47 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20201117182339': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2022-11-07 18:24:56 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182342': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-ITEC-LAB',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-ITEC-LAB/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-ITEC-LAB',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2020-12-30 09:35:16 UTC principal name: ldap/ipa1.itec.lab@ITEC.LAB key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv ITEC-LAB track: yes auto-renew: yes Request ID '20201117182351': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2020-12-30 09:35:04 UTC principal name: HTTP/ipa1.itec.lab@ITEC.LAB key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: STOPPED httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: STOPPED pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
but pki-tomcatd still fails if I try to restart it and in the debug logs:
[17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet:service() uri = /ca/admin/ca/getStatus [17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet: caGetStatus start to service. [17/Nov/2020:18:32:34][http-bio-8080-exec-7]: Failed to read product version String. java.io.FileNotFoundException: /usr/share/pki/CS_SERVER_VERSION (No such file or directory) [17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet: curDate=Tue Nov 17 18:32:34 UTC 2020 id=caGetStatus time=9
IPA VM has 2 CPU and 4GB of RAM, it never goes up to 90% of the usage