[Freeipa-users] Re: How To Renew Expired Certificates & pki-tomcatd not starting

On 2/8/21 4:11 PM, SRM via FreeIPA-users wrote:

I see some one else opened another thread with similar issue, but the error messages are different so I'm going ahead & seeking help on a new thread.

I've inherited a FreeIPA installation from somebody used among 5 physical servers with one FreeIPA server (everything CA etc on it) while other 4 physical servers act as clients. Being someone very new at LDAP & FreeIPA, I tried to troublshoot by googling.

System / Server Info:

OS - CentOS 7.6, Installed IPA packages version - 4.6.4, Self-Signed CA

Here are the issues & what steps I've taken so far.

1. Before certificates were expired the pki-tomcatd service was failing & I see the following message in /var/log/pki/pki-tomcat/ca/debug: Error: netscape.ldap.LDAPException: Authentication failed (48)

After some googling I've found this link (https://access.redhat.com/solutions/3081821) which asks to check if certificate blob & serial number in pkiuser matches to the 'subsystemCert cert-pki-ca' in our case it does so there was nothing to do but we still get that error.

2. Certificates have expired - Now the certificates have expired, they were not auto-renewed, was it because above (pki-tomcatd service failure) not sure.

   2a) For this I've tried to move back the date & tried to renew them through ipa-certupdate, the output says sucessfull but the certificates are not getting renewed. Here is the output of one such output(renamed domain to ourorg.com for privacy).

ipa-certupdate is not a tool for renewing expired certificates, please refer to its man page or https://floblanc.wordpress.com/2017/12/05/demystifying-the-certificate-autho... if you want to understand the various certificate-related tools in IPA.

      ipapython.admintool: DEBUG: Not logging to a file
     ipalib.plugable: DEBUG: importing all plugin modules in ipaclient.remote_plugins.schema$5131ac65...
     ipalib.plugable: DEBUG: importing plugin module ipaclient.remote_plugins.schema$5131ac65.plugins
     ipalib.plugable: DEBUG: importing all plugin modules in ipaclient.plugins...
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.automember
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.automount
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.ca
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.cert
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.certmap
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.certprofile
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.csrgen
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.dns
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.hbacrule
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.hbactest
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.host
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.idrange
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.internal
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.location
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.migration
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.misc
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.otptoken
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.otptoken_yubikey
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.passwd
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.permission
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.rpcclient
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.server
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.service
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.sudorule
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.topology
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.trust
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.user
     ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.vault
     ipalib.rpc: INFO: trying https://login1.ourorg.com/ipa/json
     ipalib.backend: DEBUG: Created connection context.rpcclient_139790894262416
     ipalib.install.kinit: DEBUG: Initializing principal host/login1.ourorg.com@ourorg.COM using keytab /etc/krb5.keytab
     ipalib.install.kinit: DEBUG: using ccache /tmp/tmp-O7QeRu/ccache
     ipalib.install.kinit: DEBUG: Attempt 1/1: success
     ipalib.frontend: DEBUG: raw: ca_is_enabled(version=u'2.107')
     ipalib.frontend: DEBUG: ca_is_enabled(version=u'2.107')
     ipalib.rpc: INFO: [try 1]: Forwarding 'ca_is_enabled/1' to json server 'https://login1.ourorg.com/ipa/json'
     ipalib.rpc: DEBUG: New HTTP connection (login1.ourorg.com)
     ipalib.rpc: DEBUG: received Set-Cookie (<type 'list'>)'['ipa_session=MagBearerToken=3bDSVwqoHDuM1MRVLGVRKY2DhplAszGxcdGLUBtRRZTLVV3vj8%2bNHrexIE9KX2JdrFkcYUtCfGkQmUVoYuCUj4DRqwJBoe9Z7i3J14DadLtOVCi2fNwxNR8irDD%2fG2bn4T7ULiLR6b7k1dpS%2bXWo
     iJGHOknn5EYLzi0wEOz88PauUZ7Qh1HioKfddyQhOLl1kQ6LnAsu%2fm2cACveJ8JSe2Mfmqruu8a%2fbQAIXPmRwXnC5oGN8cIk0omO4KuFQaRHWmjSNiLyG1%2bdyPiyWlxKBw%3d%3d;path=/ipa;httponly;secure;']'
     ipalib.rpc: DEBUG: storing cookie 'ipa_session=MagBearerToken=3bDSVwqoHDuM1MRVLGVRKY2DhplAszGxcdGLUBtRRZTLVV3vj8%2bNHrexIE9KX2JdrFkcYUtCfGkQmUVoYuCUj4DRqwJBoe9Z7i3J14DadLtOVCi2fNwxNR8irDD%2fG2bn4T7ULiLR6b7k1dpS%2bXWoiJGHOknn5EYLzi0wEOz88P
     auUZ7Qh1HioKfddyQhOLl1kQ6LnAsu%2fm2cACveJ8JSe2Mfmqruu8a%2fbQAIXPmRwXnC5oGN8cIk0omO4KuFQaRHWmjSNiLyG1%2bdyPiyWlxKBw%3d%3d;' for principal None
     ipapython.ipaldap: DEBUG: retrieving schema for SchemaCache url=ldap://login1.ourorg.com:389 conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f239a5242d8>
     ipalib.frontend: DEBUG: raw: ca_find(None, version=u'2.230')
     ipalib.frontend: DEBUG: ca_find(None, version=u'2.230')
     ipalib.rpc: INFO: [try 1]: Forwarding 'ca_find/1' to json server 'https://login1.ourorg.com/ipa/json'
     ipalib.rpc: DEBUG: HTTP connection keep-alive (login1.ourorg.com)
     ipalib.rpc: DEBUG: received Set-Cookie (<type 'list'>)'['ipa_session=MagBearerToken=kmtXWE4j%2buLPMXwC6RCOBvqfLCIBziy9XiM7f%2fep%2b7FYBiSPmVPwjf6USK94djhkQ6k0Rleh9KhokFWNf1AWxcH5SyVe5V6QZYLIIGzt%2fF%2f1mHl3uKOLocAauyCAz%2bVxm2FUG%2fR8ORi5
     YghKrOidtRk%2bQvERwvHJKOJ8jjikvPzlWcj1x8CjO1b6ricWSigD3%2bl1UbPEYTOMKxNSL0JEW8Q0ghkPt1bryt9aEuWZVRBU%2f%2fAYnQN6WgYkrvgyBBeYXuceYPKQFtpxUmnl2js%2bDg%3d%3d;path=/ipa;httponly;secure;']'
     ipalib.rpc: DEBUG: storing cookie 'ipa_session=MagBearerToken=kmtXWE4j%2buLPMXwC6RCOBvqfLCIBziy9XiM7f%2fep%2b7FYBiSPmVPwjf6USK94djhkQ6k0Rleh9KhokFWNf1AWxcH5SyVe5V6QZYLIIGzt%2fF%2f1mHl3uKOLocAauyCAz%2bVxm2FUG%2fR8ORi5YghKrOidtRk%2bQvERwvHJ
     KOJ8jjikvPzlWcj1x8CjO1b6ricWSigD3%2bl1UbPEYTOMKxNSL0JEW8Q0ghkPt1bryt9aEuWZVRBU%2f%2fAYnQN6WgYkrvgyBBeYXuceYPKQFtpxUmnl2js%2bDg%3d%3d;' for principal None
     ipalib.install.sysrestore: DEBUG: Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-ourorg-COM -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/dirsrv/slapd-ourorg-COM/pwdfile.txt
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-ourorg-COM -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/dirsrv/slapd-ourorg-COM/pwdfile.txt
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/bin/systemctl is-active dirsrv@ourorg-COM.service
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=active

     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/bin/systemctl --system daemon-reload
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/bin/systemctl restart dirsrv@ourorg-COM.service
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/bin/systemctl is-active dirsrv@ourorg-COM.service
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=active

     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: wait_for_open_ports: localhost [389] timeout 300
     ipapython.ipautil: DEBUG: waiting for port: 389
     ipapython.ipautil: DEBUG: SUCCESS: port: 389
     ipaplatform.base.services: DEBUG: Restart of dirsrv@ourorg-COM.service complete
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/httpd/alias -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/httpd/alias/pwdfile.txt
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/httpd/alias -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/httpd/alias/pwdfile.txt
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/bin/systemctl is-active httpd.service
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=active

     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/bin/systemctl restart httpd.service
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/bin/systemctl is-active httpd.service
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=active

     ipapython.ipautil: DEBUG: stderr=
     ipaplatform.base.services: DEBUG: Restart of httpd.service complete
     ipaclient.install.ipa_certupdate: DEBUG: resubmitting certmonger request '20190129222612'
     ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'GENERATING_CSR', variant_level=1)
     ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1)
     ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1)
     ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1)
     ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1)
     ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1)
     ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1)
     ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1)
     ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'MONITORING', variant_level=1)
     ipaclient.install.ipa_certupdate: DEBUG: modifying certmonger request '20190129222612'
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -L -n IPA CA -a -f /etc/ipa/nssdb/pwdfile.txt
     ipapython.ipautil: DEBUG: Process finished, return code=255
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: IPA CA
     : PR_FILE_NOT_FOUND_ERROR: File not found

     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -L -n External CA cert -a -f /etc/ipa/nssdb/pwdfile.txt
     ipapython.ipautil: DEBUG: Process finished, return code=255
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: External CA cert
     : PR_FILE_NOT_FOUND_ERROR: File not found

     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/usr/bin/update-ca-trust
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipaplatform.redhat.tasks: INFO: Systemwide CA database updated.
     ipapython.ipautil: DEBUG: Starting external process
     ipapython.ipautil: DEBUG: args=/usr/bin/update-ca-trust
     ipapython.ipautil: DEBUG: Process finished, return code=0
     ipapython.ipautil: DEBUG: stdout=
     ipapython.ipautil: DEBUG: stderr=
     ipaplatform.redhat.tasks: INFO: Systemwide CA database updated.
     ipalib.backend: DEBUG: Destroyed connection context.rpcclient_139790894262416
     ipapython.admintool: INFO: The ipa-certupdate command was successful

  In above output there are two occasions where it is mentioned "ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: IPA CA    : PR_FILE_NOT_FOUND_ERROR: File not found" not sure if these are relevant, if so how to debug

2b) I've also used "ipa-cacert-manage renew" following this link https://www.freeipa.org/page/V4/CA_certificate_renewal. Not sure if this was necessary or if doing this caused any more issues.

This tool renews IPA CA, not the other certificates, but it's highly unlikely that the CA cert was expired.

Since the deployment has only one IPA server, you need to fix this server. Please provide the output of "getcert list", it will show the expiration dates for all the certificates tracked by certmonger. You will need to change the system date to a date where all the certificates were still valid, start the services (but not ntp/chrony) and let certmonger renew the certs, then move back the date to the current date.

flo

3. Since certificates have expired kerberos broke, can't do "kinit admin" any longer. Can't change passwords / create users & of course can't access webui. For any of these actions need to move the date back. For now 'sudo' works (without having to move the date back) & general logins work, but not sure how long they continue to work before completely break?.

4. This is a production installation with hardly any time to take down FreeIPA let alone physical server. Is there any way to recover from this situation?.

5. If it can't be recovered can we setup another FreeIPA server installation with the same realm / domain(need to procure another system /server) with a new CA & etc from scratch and make all the current 5 physical servers (including current broken FreeIPA server) as clients to the new FreeIPA installation with the same domain / realm?.

---

FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...