I see some one else opened another thread with similar issue, but the error messages are different so I'm going ahead & seeking help on a new thread.
I've inherited a FreeIPA installation from somebody used among 5 physical servers with one FreeIPA server (everything CA etc on it) while other 4 physical servers act as clients. Being someone very new at LDAP & FreeIPA, I tried to troublshoot by googling.
System / Server Info:
OS - CentOS 7.6, Installed IPA packages version - 4.6.4, Self-Signed CA
Here are the issues & what steps I've taken so far.
1) Before certificates were expired the pki-tomcatd service was failing & I see the following message in /var/log/pki/pki-tomcat/ca/debug: Error: netscape.ldap.LDAPException: Authentication failed (48) After some googling I've found this link (https://access.redhat.com/solutions/3081821) which asks to check if certificate blob & serial number in pkiuser matches to the 'subsystemCert cert-pki-ca' in our case it does so there was nothing to do but we still get that error.
2) Certificates have expired - Now the certificates have expired, they were not auto-renewed, was it because above (pki-tomcatd service failure) not sure.
2a) For this I've tried to move back the date & tried to renew them through ipa-certupdate, the output says sucessfull but the certificates are not getting renewed. Here is the output of one such output(renamed domain to ourorg.com for privacy).
ipapython.admintool: DEBUG: Not logging to a file ipalib.plugable: DEBUG: importing all plugin modules in ipaclient.remote_plugins.schema$5131ac65... ipalib.plugable: DEBUG: importing plugin module ipaclient.remote_plugins.schema$5131ac65.plugins ipalib.plugable: DEBUG: importing all plugin modules in ipaclient.plugins... ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.automember ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.automount ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.ca ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.cert ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.certmap ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.certprofile ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.csrgen ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.dns ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.hbacrule ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.hbactest ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.host ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.idrange ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.internal ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.location ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.migration ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.misc ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.otptoken ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.otptoken_yubikey ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.passwd ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.permission ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.rpcclient ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.server ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.service ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.sudorule ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.topology ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.trust ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.user ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.vault ipalib.rpc: INFO: trying https://login1.ourorg.com/ipa/json ipalib.backend: DEBUG: Created connection context.rpcclient_139790894262416 ipalib.install.kinit: DEBUG: Initializing principal host/login1.ourorg.com@ourorg.COM using keytab /etc/krb5.keytab ipalib.install.kinit: DEBUG: using ccache /tmp/tmp-O7QeRu/ccache ipalib.install.kinit: DEBUG: Attempt 1/1: success ipalib.frontend: DEBUG: raw: ca_is_enabled(version=u'2.107') ipalib.frontend: DEBUG: ca_is_enabled(version=u'2.107') ipalib.rpc: INFO: [try 1]: Forwarding 'ca_is_enabled/1' to json server 'https://login1.ourorg.com/ipa/json' ipalib.rpc: DEBUG: New HTTP connection (login1.ourorg.com) ipalib.rpc: DEBUG: received Set-Cookie (<type 'list'>)'['ipa_session=MagBearerToken=3bDSVwqoHDuM1MRVLGVRKY2DhplAszGxcdGLUBtRRZTLVV3vj8%2bNHrexIE9KX2JdrFkcYUtCfGkQmUVoYuCUj4DRqwJBoe9Z7i3J14DadLtOVCi2fNwxNR8irDD%2fG2bn4T7ULiLR6b7k1dpS%2bXWo iJGHOknn5EYLzi0wEOz88PauUZ7Qh1HioKfddyQhOLl1kQ6LnAsu%2fm2cACveJ8JSe2Mfmqruu8a%2fbQAIXPmRwXnC5oGN8cIk0omO4KuFQaRHWmjSNiLyG1%2bdyPiyWlxKBw%3d%3d;path=/ipa;httponly;secure;']' ipalib.rpc: DEBUG: storing cookie 'ipa_session=MagBearerToken=3bDSVwqoHDuM1MRVLGVRKY2DhplAszGxcdGLUBtRRZTLVV3vj8%2bNHrexIE9KX2JdrFkcYUtCfGkQmUVoYuCUj4DRqwJBoe9Z7i3J14DadLtOVCi2fNwxNR8irDD%2fG2bn4T7ULiLR6b7k1dpS%2bXWoiJGHOknn5EYLzi0wEOz88P auUZ7Qh1HioKfddyQhOLl1kQ6LnAsu%2fm2cACveJ8JSe2Mfmqruu8a%2fbQAIXPmRwXnC5oGN8cIk0omO4KuFQaRHWmjSNiLyG1%2bdyPiyWlxKBw%3d%3d;' for principal None ipapython.ipaldap: DEBUG: retrieving schema for SchemaCache url=ldap://login1.ourorg.com:389 conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f239a5242d8> ipalib.frontend: DEBUG: raw: ca_find(None, version=u'2.230') ipalib.frontend: DEBUG: ca_find(None, version=u'2.230') ipalib.rpc: INFO: [try 1]: Forwarding 'ca_find/1' to json server 'https://login1.ourorg.com/ipa/json' ipalib.rpc: DEBUG: HTTP connection keep-alive (login1.ourorg.com) ipalib.rpc: DEBUG: received Set-Cookie (<type 'list'>)'['ipa_session=MagBearerToken=kmtXWE4j%2buLPMXwC6RCOBvqfLCIBziy9XiM7f%2fep%2b7FYBiSPmVPwjf6USK94djhkQ6k0Rleh9KhokFWNf1AWxcH5SyVe5V6QZYLIIGzt%2fF%2f1mHl3uKOLocAauyCAz%2bVxm2FUG%2fR8ORi5 YghKrOidtRk%2bQvERwvHJKOJ8jjikvPzlWcj1x8CjO1b6ricWSigD3%2bl1UbPEYTOMKxNSL0JEW8Q0ghkPt1bryt9aEuWZVRBU%2f%2fAYnQN6WgYkrvgyBBeYXuceYPKQFtpxUmnl2js%2bDg%3d%3d;path=/ipa;httponly;secure;']' ipalib.rpc: DEBUG: storing cookie 'ipa_session=MagBearerToken=kmtXWE4j%2buLPMXwC6RCOBvqfLCIBziy9XiM7f%2fep%2b7FYBiSPmVPwjf6USK94djhkQ6k0Rleh9KhokFWNf1AWxcH5SyVe5V6QZYLIIGzt%2fF%2f1mHl3uKOLocAauyCAz%2bVxm2FUG%2fR8ORi5YghKrOidtRk%2bQvERwvHJ KOJ8jjikvPzlWcj1x8CjO1b6ricWSigD3%2bl1UbPEYTOMKxNSL0JEW8Q0ghkPt1bryt9aEuWZVRBU%2f%2fAYnQN6WgYkrvgyBBeYXuceYPKQFtpxUmnl2js%2bDg%3d%3d;' for principal None ipalib.install.sysrestore: DEBUG: Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-ourorg-COM -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/dirsrv/slapd-ourorg-COM/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-ourorg-COM -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/dirsrv/slapd-ourorg-COM/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl is-active dirsrv@ourorg-COM.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout=active
ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl --system daemon-reload ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl restart dirsrv@ourorg-COM.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl is-active dirsrv@ourorg-COM.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout=active
ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: wait_for_open_ports: localhost [389] timeout 300 ipapython.ipautil: DEBUG: waiting for port: 389 ipapython.ipautil: DEBUG: SUCCESS: port: 389 ipaplatform.base.services: DEBUG: Restart of dirsrv@ourorg-COM.service complete ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/httpd/alias -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/httpd/alias/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/httpd/alias -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/httpd/alias/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl is-active httpd.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout=active
ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl restart httpd.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl is-active httpd.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout=active
ipapython.ipautil: DEBUG: stderr= ipaplatform.base.services: DEBUG: Restart of httpd.service complete ipaclient.install.ipa_certupdate: DEBUG: resubmitting certmonger request '20190129222612' ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'GENERATING_CSR', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'MONITORING', variant_level=1) ipaclient.install.ipa_certupdate: DEBUG: modifying certmonger request '20190129222612' ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -L -n IPA CA -a -f /etc/ipa/nssdb/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=255 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found
ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -L -n External CA cert -a -f /etc/ipa/nssdb/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=255 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: External CA cert : PR_FILE_NOT_FOUND_ERROR: File not found
ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/update-ca-trust ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipaplatform.redhat.tasks: INFO: Systemwide CA database updated. ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/update-ca-trust ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipaplatform.redhat.tasks: INFO: Systemwide CA database updated. ipalib.backend: DEBUG: Destroyed connection context.rpcclient_139790894262416 ipapython.admintool: INFO: The ipa-certupdate command was successful
In above output there are two occasions where it is mentioned "ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found" not sure if these are relevant, if so how to debug
2b) I've also used "ipa-cacert-manage renew" following this link https://www.freeipa.org/page/V4/CA_certificate_renewal. Not sure if this was necessary or if doing this caused any more issues.
3) Since certificates have expired kerberos broke, can't do "kinit admin" any longer. Can't change passwords / create users & of course can't access webui. For any of these actions need to move the date back. For now 'sudo' works (without having to move the date back) & general logins work, but not sure how long they continue to work before completely break?.
4) This is a production installation with hardly any time to take down FreeIPA let alone physical server. Is there any way to recover from this situation?.
5) If it can't be recovered can we setup another FreeIPA server installation with the same realm / domain(need to procure another system /server) with a new CA & etc from scratch and make all the current 5 physical servers (including current broken FreeIPA server) as clients to the new FreeIPA installation with the same domain / realm?.
On 2/8/21 4:11 PM, SRM via FreeIPA-users wrote:
I see some one else opened another thread with similar issue, but the error messages are different so I'm going ahead & seeking help on a new thread.
I've inherited a FreeIPA installation from somebody used among 5 physical servers with one FreeIPA server (everything CA etc on it) while other 4 physical servers act as clients. Being someone very new at LDAP & FreeIPA, I tried to troublshoot by googling.
System / Server Info:
OS - CentOS 7.6, Installed IPA packages version - 4.6.4, Self-Signed CA
Here are the issues & what steps I've taken so far.
- Before certificates were expired the pki-tomcatd service was failing & I see the following message in /var/log/pki/pki-tomcat/ca/debug: Error: netscape.ldap.LDAPException: Authentication failed (48)
After some googling I've found this link (https://access.redhat.com/solutions/3081821) which asks to check if certificate blob & serial number in pkiuser matches to the 'subsystemCert cert-pki-ca' in our case it does so there was nothing to do but we still get that error.
Certificates have expired - Now the certificates have expired, they were not auto-renewed, was it because above (pki-tomcatd service failure) not sure.
2a) For this I've tried to move back the date & tried to renew them through ipa-certupdate, the output says sucessfull but the certificates are not getting renewed. Here is the output of one such output(renamed domain to ourorg.com for privacy).
ipa-certupdate is not a tool for renewing expired certificates, please refer to its man page or https://floblanc.wordpress.com/2017/12/05/demystifying-the-certificate-autho... if you want to understand the various certificate-related tools in IPA.
ipapython.admintool: DEBUG: Not logging to a file ipalib.plugable: DEBUG: importing all plugin modules in ipaclient.remote_plugins.schema$5131ac65... ipalib.plugable: DEBUG: importing plugin module ipaclient.remote_plugins.schema$5131ac65.plugins ipalib.plugable: DEBUG: importing all plugin modules in ipaclient.plugins... ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.automember ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.automount ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.ca ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.cert ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.certmap ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.certprofile ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.csrgen ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.dns ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.hbacrule ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.hbactest ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.host ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.idrange ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.internal ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.location ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.migration ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.misc ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.otptoken ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.otptoken_yubikey ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.passwd ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.permission ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.rpcclient ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.server ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.service ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.sudorule ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.topology ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.trust ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.user ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.vault ipalib.rpc: INFO: trying https://login1.ourorg.com/ipa/json ipalib.backend: DEBUG: Created connection context.rpcclient_139790894262416 ipalib.install.kinit: DEBUG: Initializing principal host/login1.ourorg.com@ourorg.COM using keytab /etc/krb5.keytab ipalib.install.kinit: DEBUG: using ccache /tmp/tmp-O7QeRu/ccache ipalib.install.kinit: DEBUG: Attempt 1/1: success ipalib.frontend: DEBUG: raw: ca_is_enabled(version=u'2.107') ipalib.frontend: DEBUG: ca_is_enabled(version=u'2.107') ipalib.rpc: INFO: [try 1]: Forwarding 'ca_is_enabled/1' to json server 'https://login1.ourorg.com/ipa/json' ipalib.rpc: DEBUG: New HTTP connection (login1.ourorg.com) ipalib.rpc: DEBUG: received Set-Cookie (<type 'list'>)'['ipa_session=MagBearerToken=3bDSVwqoHDuM1MRVLGVRKY2DhplAszGxcdGLUBtRRZTLVV3vj8%2bNHrexIE9KX2JdrFkcYUtCfGkQmUVoYuCUj4DRqwJBoe9Z7i3J14DadLtOVCi2fNwxNR8irDD%2fG2bn4T7ULiLR6b7k1dpS%2bXWo iJGHOknn5EYLzi0wEOz88PauUZ7Qh1HioKfddyQhOLl1kQ6LnAsu%2fm2cACveJ8JSe2Mfmqruu8a%2fbQAIXPmRwXnC5oGN8cIk0omO4KuFQaRHWmjSNiLyG1%2bdyPiyWlxKBw%3d%3d;path=/ipa;httponly;secure;']' ipalib.rpc: DEBUG: storing cookie 'ipa_session=MagBearerToken=3bDSVwqoHDuM1MRVLGVRKY2DhplAszGxcdGLUBtRRZTLVV3vj8%2bNHrexIE9KX2JdrFkcYUtCfGkQmUVoYuCUj4DRqwJBoe9Z7i3J14DadLtOVCi2fNwxNR8irDD%2fG2bn4T7ULiLR6b7k1dpS%2bXWoiJGHOknn5EYLzi0wEOz88P auUZ7Qh1HioKfddyQhOLl1kQ6LnAsu%2fm2cACveJ8JSe2Mfmqruu8a%2fbQAIXPmRwXnC5oGN8cIk0omO4KuFQaRHWmjSNiLyG1%2bdyPiyWlxKBw%3d%3d;' for principal None ipapython.ipaldap: DEBUG: retrieving schema for SchemaCache url=ldap://login1.ourorg.com:389 conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f239a5242d8> ipalib.frontend: DEBUG: raw: ca_find(None, version=u'2.230') ipalib.frontend: DEBUG: ca_find(None, version=u'2.230') ipalib.rpc: INFO: [try 1]: Forwarding 'ca_find/1' to json server 'https://login1.ourorg.com/ipa/json' ipalib.rpc: DEBUG: HTTP connection keep-alive (login1.ourorg.com) ipalib.rpc: DEBUG: received Set-Cookie (<type 'list'>)'['ipa_session=MagBearerToken=kmtXWE4j%2buLPMXwC6RCOBvqfLCIBziy9XiM7f%2fep%2b7FYBiSPmVPwjf6USK94djhkQ6k0Rleh9KhokFWNf1AWxcH5SyVe5V6QZYLIIGzt%2fF%2f1mHl3uKOLocAauyCAz%2bVxm2FUG%2fR8ORi5 YghKrOidtRk%2bQvERwvHJKOJ8jjikvPzlWcj1x8CjO1b6ricWSigD3%2bl1UbPEYTOMKxNSL0JEW8Q0ghkPt1bryt9aEuWZVRBU%2f%2fAYnQN6WgYkrvgyBBeYXuceYPKQFtpxUmnl2js%2bDg%3d%3d;path=/ipa;httponly;secure;']' ipalib.rpc: DEBUG: storing cookie 'ipa_session=MagBearerToken=kmtXWE4j%2buLPMXwC6RCOBvqfLCIBziy9XiM7f%2fep%2b7FYBiSPmVPwjf6USK94djhkQ6k0Rleh9KhokFWNf1AWxcH5SyVe5V6QZYLIIGzt%2fF%2f1mHl3uKOLocAauyCAz%2bVxm2FUG%2fR8ORi5YghKrOidtRk%2bQvERwvHJ KOJ8jjikvPzlWcj1x8CjO1b6ricWSigD3%2bl1UbPEYTOMKxNSL0JEW8Q0ghkPt1bryt9aEuWZVRBU%2f%2fAYnQN6WgYkrvgyBBeYXuceYPKQFtpxUmnl2js%2bDg%3d%3d;' for principal None ipalib.install.sysrestore: DEBUG: Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-ourorg-COM -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/dirsrv/slapd-ourorg-COM/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-ourorg-COM -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/dirsrv/slapd-ourorg-COM/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl is-active dirsrv@ourorg-COM.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout=active ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl --system daemon-reload ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl restart dirsrv@ourorg-COM.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl is-active dirsrv@ourorg-COM.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout=active ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: wait_for_open_ports: localhost [389] timeout 300 ipapython.ipautil: DEBUG: waiting for port: 389 ipapython.ipautil: DEBUG: SUCCESS: port: 389 ipaplatform.base.services: DEBUG: Restart of dirsrv@ourorg-COM.service complete ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/httpd/alias -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/httpd/alias/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/httpd/alias -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/httpd/alias/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl is-active httpd.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout=active ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl restart httpd.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/bin/systemctl is-active httpd.service ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout=active ipapython.ipautil: DEBUG: stderr= ipaplatform.base.services: DEBUG: Restart of httpd.service complete ipaclient.install.ipa_certupdate: DEBUG: resubmitting certmonger request '20190129222612' ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'GENERATING_CSR', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'MONITORING', variant_level=1) ipaclient.install.ipa_certupdate: DEBUG: modifying certmonger request '20190129222612' ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -L -n IPA CA -a -f /etc/ipa/nssdb/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=255 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -L -n External CA cert -a -f /etc/ipa/nssdb/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=255 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: External CA cert : PR_FILE_NOT_FOUND_ERROR: File not found ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/update-ca-trust ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipaplatform.redhat.tasks: INFO: Systemwide CA database updated. ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/update-ca-trust ipapython.ipautil: DEBUG: Process finished, return code=0 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr= ipaplatform.redhat.tasks: INFO: Systemwide CA database updated. ipalib.backend: DEBUG: Destroyed connection context.rpcclient_139790894262416 ipapython.admintool: INFO: The ipa-certupdate command was successful In above output there are two occasions where it is mentioned "ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found" not sure if these are relevant, if so how to debug 2b) I've also used "ipa-cacert-manage renew" following this link https://www.freeipa.org/page/V4/CA_certificate_renewal. Not sure if this was necessary or if doing this caused any more issues.
This tool renews IPA CA, not the other certificates, but it's highly unlikely that the CA cert was expired.
Since the deployment has only one IPA server, you need to fix this server. Please provide the output of "getcert list", it will show the expiration dates for all the certificates tracked by certmonger. You will need to change the system date to a date where all the certificates were still valid, start the services (but not ntp/chrony) and let certmonger renew the certs, then move back the date to the current date.
flo
Since certificates have expired kerberos broke, can't do "kinit admin" any longer. Can't change passwords / create users & of course can't access webui. For any of these actions need to move the date back. For now 'sudo' works (without having to move the date back) & general logins work, but not sure how long they continue to work before completely break?.
This is a production installation with hardly any time to take down FreeIPA let alone physical server. Is there any way to recover from this situation?.
If it can't be recovered can we setup another FreeIPA server installation with the same realm / domain(need to procure another system /server) with a new CA & etc from scratch and make all the current 5 physical servers (including current broken FreeIPA server) as clients to the new FreeIPA installation with the same domain / realm?.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
First of all thank you for taking time & replying. I thought "ipa-cacert-manage renew" is for renewing IPA CA & "ipa-certupdate" is for renewing certificates, so should I use "ipa cert-request" to get renew / new certificates. And pki-tomcatd service broke even before certificates got expired with authentication error (48). By the way here is the Reddit thread I've created, which has better formatting.
Here is the output of gercert list command: Please note the status of first 6 changes from SUBMITTING to MONITORING while the status of the last 3 changes from SUBMITTING to CA_UNREACHABLE
Number of certificates and requests being tracked: 9. Request ID '20190129222559': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=IPA RA,O=ourorg.COM expires: 2021-01-18 22:25:59 UTC key usage: digitalSignature,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20190129222609': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=CA Audit,O=ourorg.COM expires: 2021-01-18 22:25:41 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190129222610': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=OCSP Subsystem,O=ourorg.COM expires: 2021-01-18 22:25:41 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190129222611': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=CA Subsystem,O=ourorg.COM expires: 2021-01-18 22:25:41 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190129222612': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=Certificate Authority,O=ourorg.COM expires: 2039-02-04 17:27:12 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190129222613': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=login1.ourorg.com,O=ourorg.COM expires: 2021-01-18 22:25:41 UTC dns: login1.ourorg.com key usage: digitalSignature,keyEncipherment,dataEncipherment eku: id-kp-serverAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20190129222625': status: CA_UNREACHABLE ca-error: Server at https://login1.ourorg.com/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: TCP connection reset by peer). stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-ourorg-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-ourorg-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-ourorg-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=login1.ourorg.com,O=ourorg.COM expires: 2021-01-29 22:26:25 UTC dns: login1.ourorg.com principal name: ldap/login1.ourorg.com@ourorg.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv ourorg-COM track: yes auto-renew: yes Request ID '20190129222654': status: CA_UNREACHABLE ca-error: Server at https://login1.ourorg.com/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: TCP connection reset by peer). stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=login1.ourorg.com,O=ourorg.COM expires: 2021-01-29 22:26:54 UTC dns: login1.ourorg.com principal name: HTTP/login1.ourorg.com@ourorg.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20190129222703': status: SUBMITTING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=login1.ourorg.com,O=ourorg.COM expires: 2021-01-29 22:27:03 UTC principal name: krbtgt/ourorg.COM@ourorg.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
On 2/9/21 10:40 AM, SRM via FreeIPA-users wrote:
First of all thank you for taking time & replying. I thought "ipa-cacert-manage renew" is for renewing IPA CA & "ipa-certupdate" is for renewing certificates, so should I use "ipa cert-request" to get renew / new certificates. And pki-tomcatd service broke even before certificates got expired with authentication error (48). By the way here is the Reddit thread I've created, which has better formatting.
Here is the output of gercert list command: Please note the status of first 6 changes from SUBMITTING to MONITORING while the status of the last 3 changes from SUBMITTING to CA_UNREACHABLE
CA UNREACHABLE may correspond to many different errors but let's check first if the CA is running. What is the output of ipactl status? Can you run "kinit admin; ipa cert-show 1"?
flo
Number of certificates and requests being tracked: 9. Request ID '20190129222559': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=IPA RA,O=ourorg.COM expires: 2021-01-18 22:25:59 UTC key usage: digitalSignature,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20190129222609': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=CA Audit,O=ourorg.COM expires: 2021-01-18 22:25:41 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190129222610': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=OCSP Subsystem,O=ourorg.COM expires: 2021-01-18 22:25:41 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190129222611': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=CA Subsystem,O=ourorg.COM expires: 2021-01-18 22:25:41 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190129222612': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=Certificate Authority,O=ourorg.COM expires: 2039-02-04 17:27:12 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190129222613': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=login1.ourorg.com,O=ourorg.COM expires: 2021-01-18 22:25:41 UTC dns: login1.ourorg.com key usage: digitalSignature,keyEncipherment,dataEncipherment eku: id-kp-serverAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20190129222625': status: CA_UNREACHABLE ca-error: Server at https://login1.ourorg.com/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: TCP connection reset by peer). stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-ourorg-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-ourorg-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-ourorg-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=login1.ourorg.com,O=ourorg.COM expires: 2021-01-29 22:26:25 UTC dns: login1.ourorg.com principal name: ldap/login1.ourorg.com@ourorg.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv ourorg-COM track: yes auto-renew: yes Request ID '20190129222654': status: CA_UNREACHABLE ca-error: Server at https://login1.ourorg.com/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: TCP connection reset by peer). stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=login1.ourorg.com,O=ourorg.COM expires: 2021-01-29 22:26:54 UTC dns: login1.ourorg.com principal name: HTTP/login1.ourorg.com@ourorg.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20190129222703': status: SUBMITTING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=ourorg.COM subject: CN=login1.ourorg.com,O=ourorg.COM expires: 2021-01-29 22:27:03 UTC principal name: krbtgt/ourorg.COM@ourorg.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Thanks for your reply. Here is the output of "kinit admin; ipa cert-show 1": ipa: DEBUG: failed to find session_cookie in persistent storage for principal 'admin@ourorg.COM' ipa: INFO: trying https://login1.ourorg.com/ipa/json ipa: DEBUG: Created connection context.rpcclient_140248688553680 ipa: INFO: [try 1]: Forwarding 'schema' to json server 'https://login1.ourorg.com/ipa/json' ipa: DEBUG: HTTP connection destroyed (login1.ourorg.com) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 694, in single_request h = self.make_connection(host) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 573, in make_connection conn.connect() File "/usr/lib64/python2.7/httplib.py", line 1275, in connect server_hostname=sni_hostname) File "/usr/lib64/python2.7/ssl.py", line 348, in wrap_socket _context=self) File "/usr/lib64/python2.7/ssl.py", line 609, in __init__ self.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake self._sslobj.do_handshake() SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) ipa: DEBUG: Destroyed connection context.rpcclient_140248688553680 ipa: ERROR: cannot connect to 'https://login1.ourorg.com/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
And output of "ipactl status", note as I mentioned in the first post pki-tomcatd service was failing even before certificates got expired.
Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa: INFO: The ipactl command was successful
SRM via FreeIPA-users wrote:
Thanks for your reply. Here is the output of "kinit admin; ipa cert-show 1": ipa: DEBUG: failed to find session_cookie in persistent storage for principal 'admin@ourorg.COM' ipa: INFO: trying https://login1.ourorg.com/ipa/json ipa: DEBUG: Created connection context.rpcclient_140248688553680 ipa: INFO: [try 1]: Forwarding 'schema' to json server 'https://login1.ourorg.com/ipa/json' ipa: DEBUG: HTTP connection destroyed (login1.ourorg.com) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 694, in single_request h = self.make_connection(host) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 573, in make_connection conn.connect() File "/usr/lib64/python2.7/httplib.py", line 1275, in connect server_hostname=sni_hostname) File "/usr/lib64/python2.7/ssl.py", line 348, in wrap_socket _context=self) File "/usr/lib64/python2.7/ssl.py", line 609, in __init__ self.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake self._sslobj.do_handshake() SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) ipa: DEBUG: Destroyed connection context.rpcclient_140248688553680 ipa: ERROR: cannot connect to 'https://login1.ourorg.com/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
And output of "ipactl status", note as I mentioned in the first post pki-tomcatd service was failing even before certificates got expired.
Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa: INFO: The ipactl command was successful
We need to start by getting the CA running properly while back in time when the certs are still valid. There is no way to re-issue the certificates without it.
Can you share the logging and output from your verification of the pki subsystem certificate?
rob
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Rob Crittenden -
SRM