Setup: Cluster of 3 FreeIPA Masters with one as the CA Renewal Master ipa version 4.8.4
Problem: One of our certs for one of our servers recently expired, but it was supposed to auto-renew. Looking into the issue I found that I couldn't access any certs via CLI or the webUI. When trying to do either, I get the following error: IPA Error 4301: CertificateOperationError Certificate operation cannot be completed: Unable to communicate with CMS (403)
After doing some research it seems the issue may be with the IPA RA, though I found a userCertificate in the LDAP that was issued the same day as the one being used by the ipa server (it had the userCertificate being used by the ipa server as well as another userCertificate, both have the same dates, but different certificates), changing the ra-agent.pem did not seem to solve any problems. Looking in /var/log/pki/pki-tomcat/ca/debug I found the following errors: WARNING: CertProcessor: No authenticator credentials required SEVERE: AgentCertAuthentication: No SSL Client Certs Found SEVERE: CAProcessor: authentication error: Invalid Credential.
I'm a little lost and not sure what to do next, any help would be greatly appreciated.
Corey Devenport via FreeIPA-users wrote:
Setup: Cluster of 3 FreeIPA Masters with one as the CA Renewal Master ipa version 4.8.4
Problem: One of our certs for one of our servers recently expired, but it was supposed to auto-renew. Looking into the issue I found that I couldn't access any certs via CLI or the webUI. When trying to do either, I get the following error: IPA Error 4301: CertificateOperationError Certificate operation cannot be completed: Unable to communicate with CMS (403)
After doing some research it seems the issue may be with the IPA RA, though I found a userCertificate in the LDAP that was issued the same day as the one being used by the ipa server (it had the userCertificate being used by the ipa server as well as another userCertificate, both have the same dates, but different certificates), changing the ra-agent.pem did not seem to solve any problems. Looking in /var/log/pki/pki-tomcat/ca/debug I found the following errors: WARNING: CertProcessor: No authenticator credentials required SEVERE: AgentCertAuthentication: No SSL Client Certs Found SEVERE: CAProcessor: authentication error: Invalid Credential.
I'm a little lost and not sure what to do next, any help would be greatly appreciated.
The CA is a servlet that runs in tomcat and a 403 suggests it isn't running. Most likely because its subsystem certificates have expired.
This should provide details on the status of the certificates.
# getcert list
I'd also figure out which server is supposed to be doing the renewal:
$ ipa config-show |grep renew
Based on that we can make a plan forward. rob
Number of certificates and requests being tracked: 9. Request ID '20200130175256': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=... subject: CN=athena...,O=... expires: 2022-01-30 10:56:14 MST principal name: krbtgt/...@... key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20200922184055': status: NEWLY_ADDED_NEED_KEYINFO_READ_PIN stuck: yes key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca' certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca' CA: dogtag-ipa-ca-renew-agent issuer: subject: expires: unknown pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20200922184056': status: NEWLY_ADDED_NEED_KEYINFO_READ_PIN stuck: yes key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca' certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca' CA: dogtag-ipa-ca-renew-agent issuer: subject: expires: unknown pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20200922184057': status: NEWLY_ADDED_NEED_KEYINFO_READ_PIN stuck: yes key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca' certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca' CA: dogtag-ipa-ca-renew-agent issuer: subject: expires: unknown pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20200922184058': status: NEWLY_ADDED_NEED_KEYINFO_READ_PIN stuck: yes key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca' certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca' CA: dogtag-ipa-ca-renew-agent issuer: subject: expires: unknown pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20200922184059': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=... subject: CN=athena...,O=... expires: 2022-01-23 12:16:31 MST key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20200922184100': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=... subject: CN=IPA RA,O=... expires: 2022-01-16 09:45:38 MST key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20200922184101': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd...',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-.../pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-...',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=... subject: CN=athena...,O=... expires: 2022-01-30 10:52:15 MST dns: athena.cs.byu.edu principal name: ldap/athena...@... key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv ... track: yes auto-renew: yes Request ID '20200922184102': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/athena.cs.byu.edu-443-RSA' certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt' CA: IPA issuer: CN=Certificate Authority,O=... subject: CN=athena...,O=... expires: 2022-01-30 10:52:42 MST dns: athena... principal name: HTTP/athena...@... key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
I would assume our issue is with the ones with status: NEWLY_ADDED_NEED_KEYINFO_READ_PIN which are stuck... not sure how I missed those... guess I was too focused on the IPA RA which seemed to be fine here.
The renewal master is named nike.
Thanks for the quick reply!
Update:
I was able to get all of the subsystem certs unstuck using the following: getcert start-tracking -d /etc/pki/pki-tomcat/alias -n <cert-name> -P <pwd> -c dogtag-ipa-ca-renew-agent
However, the caSigningCert is still having issues, and the status is not very helpful: Request ID '20200922184058': status: NEED_GUIDANCE stuck: yes key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=... subject: CN=Certificate Authority,O=... expires: 2036-02-17 17:05:50 MST key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes
Any ideas on how to fix this?
Corey Devenport via FreeIPA-users wrote:
Request ID '20200922184058': status: NEED_GUIDANCE stuck: yes key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=... subject: CN=Certificate Authority,O=... expires: 2036-02-17 17:05:50 MST key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes
From getcert-list(1):
NEED_GUIDANCE An unhandled error was encountered while attempting to contact the CA, or there is the service has just been told to monitor a certificate which does not exist and for which it has no location specified for storing a key pair that could be used to generate a signing request to obtain one.
I'd check the journal for errors. It's safe to assume the cert exists in the database if the CA is operational.
The tracking looks right. Restarting certmonger may refresh its status. It could be leftover from when the CA wasn't working.
rob
Corey Devenport via FreeIPA-users wrote:
From getcert-list(1):
NEED_GUIDANCE An unhandled error was encountered while attempting to contact the CA, or there is the service has just been told to monitor a certificate which does not exist and for which it has no location specified for storing a key pair that could be used to generate a signing request to obtain one.
I'd check the journal for errors. It's safe to assume the cert exists in the database if the CA is operational.
The tracking looks right. Restarting certmonger may refresh its status. It could be leftover from when the CA wasn't working.
rob
There don't seem to be any errors in the journal...
I restarted certmonger and the status remains the same. I'm still unable to perform any certificate operation from the webUI or through the CLI. I get the same error of "Unable to communicate with CMS (403)"
Corey Devenport via FreeIPA-users wrote:
Corey Devenport via FreeIPA-users wrote:
From getcert-list(1):
NEED_GUIDANCE An unhandled error was encountered while attempting to contact the CA, or there is the service has just been told to monitor a certificate which does not exist and for which it has no location specified for storing a key pair that could be used to generate a signing request to obtain one.
I'd check the journal for errors. It's safe to assume the cert exists in the database if the CA is operational.
The tracking looks right. Restarting certmonger may refresh its status. It could be leftover from when the CA wasn't working.
rob
There don't seem to be any errors in the journal...
I restarted certmonger and the status remains the same. I'm still unable to perform any certificate operation from the webUI or through the CLI. I get the same error of "Unable to communicate with CMS (403)"
That means the CA isn't running which could also explain the NEED_GUIDANCE.
Look in the ca debug log for details but you have to read top-down and not bottom up.
rob
I'm getting the same messages as before: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: Authenticating certificate chain: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - CN=IPA RA, O=... 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: CertUserDBAuthentication: UID ipara authenticated. 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: User ID: ipara 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: UGSubsystem: retrieving user uid=ipara,ou=People,o=ipaca 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: User DN: uid=ipara,ou=people,o=ipaca 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: Roles: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Certificate Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Registration Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: AAclAuthz: Granting login permission for certServer.ca.account 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: Creating session 7BC692BAA9A8FD6ABBB2574BE9AB3D20 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: Principal: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - ID: ipara 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Full Name: ipara 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Email: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Roles: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Certificate Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Registration Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: Authenticating certificate chain: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: - CN=IPA RA, O=... 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: CertUserDBAuthentication: UID ipara authenticated. 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: User ID: ipara 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: UGSubsystem: retrieving user uid=ipara,ou=People,o=ipaca 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: User DN: uid=ipara,ou=people,o=ipaca 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: Roles: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: - Certificate Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: - Registration Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: AAclAuthz: Granting logout permission for certServer.ca.account 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: Destroying session 2BB59DC4F6CBF1B42B54E99B35BE45B3 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-8] INFO: Receiving certificate request 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-8] WARNING: CertProcessor: No authenticator credentials required 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-8] SEVERE: AgentCertAuthentication: No SSL Client Certs Found 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-8] SEVERE: CAProcessor: authentication error: Invalid Credential. Invalid Credential.
Corey Devenport via FreeIPA-users wrote:
I'm getting the same messages as before: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: Authenticating certificate chain: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - CN=IPA RA, O=... 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: CertUserDBAuthentication: UID ipara authenticated. 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: User ID: ipara 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: UGSubsystem: retrieving user uid=ipara,ou=People,o=ipaca 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: User DN: uid=ipara,ou=people,o=ipaca 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: Roles: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Certificate Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Registration Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: AAclAuthz: Granting login permission for certServer.ca.account 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: Creating session 7BC692BAA9A8FD6ABBB2574BE9AB3D20 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: Principal: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - ID: ipara 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Full Name: ipara 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Email: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Roles: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Certificate Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-5] INFO: - Registration Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: Authenticating certificate chain: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: - CN=IPA RA, O=... 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: CertUserDBAuthentication: UID ipara authenticated. 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: User ID: ipara 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: UGSubsystem: retrieving user uid=ipara,ou=People,o=ipaca 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: User DN: uid=ipara,ou=people,o=ipaca 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: Roles: 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: - Certificate Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: - Registration Manager Agents 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: AAclAuthz: Granting logout permission for certServer.ca.account 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-7] INFO: Destroying session 2BB59DC4F6CBF1B42B54E99B35BE45B3 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-8] INFO: Receiving certificate request 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-8] WARNING: CertProcessor: No authenticator credentials required 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-8] SEVERE: AgentCertAuthentication: No SSL Client Certs Found 2020-11-10 12:01:38 [ajp-nio-127.0.0.1-8009-exec-8] SEVERE: CAProcessor: authentication error: Invalid Credential. Invalid Credential.
You'll want to look at that entry, uid=ipara,ou=people,o=ipaca, to see if the cert blob matches and the description has the correct serial number.
You might also want to give ipa-healthcheck a run to see if it detects any other issues.
rob
ldap has many cert blobs, the ipa server has the second to last blob, which is said to expire in 2022. Performing an ldap search gives two serial numbers:
description: 2;805175304;CN=Certificate Authority,O=...;CN=IPA RA,O=... description: 2;804978693;CN=Certificate Authority,O=...;CN=IPA RA,O=...'
The ipa server's ra-agent has the first serial number. Could there be two conflicting certs?
For some reason ipa-healthcheck doesn't seem to be installed, which I find odd because it seems it should come included in ipa 4.8 right?
Corey Devenport via FreeIPA-users wrote:
ldap has many cert blobs, the ipa server has the second to last blob, which is said to expire in 2022. Performing an ldap search gives two serial numbers:
description: 2;805175304;CN=Certificate Authority,O=...;CN=IPA RA,O=... description: 2;804978693;CN=Certificate Authority,O=...;CN=IPA RA,O=...'
I'm pretty sure this is expected to be single-valued. The usercertificate should be ok being multi-valued.
The ipa server's ra-agent has the first serial number. Could there be two conflicting certs?
For some reason ipa-healthcheck doesn't seem to be installed, which I find odd because it seems it should come included in ipa 4.8 right?
It's a separate package not installed by default.
rob
Corey Devenport via FreeIPA-users wrote:
I'm pretty sure this is expected to be single-valued. The usercertificate should be ok being multi-valued.
So should I delete one of the descriptions from LDAP? Then the question is which one, the ipa servers seem to all have the 805175304 description, so should I remove the 804978693 description? I have tried editing the ra-agent.pem to use the other cert associated with the other certificate, but this doesn't seem to change anything
It's a separate package not installed by default.
rob
I was able to install ipa-healthcheck, and running it gave the same ole error of not being able to communicate with CMS: Loading instance: pki-tomcat Loading global Tomcat config: /etc/tomcat/tomcat.conf Loading PKI Tomcat config: /usr/share/pki/etc/tomcat.conf Loading instance Tomcat config: /etc/pki/pki-tomcat/tomcat.conf Loading password config: /etc/pki/pki-tomcat/password.conf Loading instance registry: /etc/sysconfig/pki/tomcat/pki-tomcat/pki-tomcat Loading subsystem: ca Loading subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg Getting sslserver cert info for ca from CS.cfg Getting subsystem cert info for ca from CS.cfg Getting audit_signing cert info for ca from CS.cfg Getting ocsp_signing cert info for ca from CS.cfg Getting signing cert info for ca from CS.cfg ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) Unhandler rdtype 28 Unhandler rdtype 28 Unhandler rdtype 28
Corey Devenport via FreeIPA-users wrote:
Corey Devenport via FreeIPA-users wrote:
I'm pretty sure this is expected to be single-valued. The usercertificate should be ok being multi-valued.
So should I delete one of the descriptions from LDAP? Then the question is which one, the ipa servers seem to all have the 805175304 description, so should I remove the 804978693 description? I have tried editing the ra-agent.pem to use the other cert associated with the other certificate, but this doesn't seem to change anything
You'd want to keep the most recently issued. This is likely the highest serial number.
It's a separate package not installed by default.
rob
I was able to install ipa-healthcheck, and running it gave the same ole error of not being able to communicate with CMS: Loading instance: pki-tomcat Loading global Tomcat config: /etc/tomcat/tomcat.conf Loading PKI Tomcat config: /usr/share/pki/etc/tomcat.conf Loading instance Tomcat config: /etc/pki/pki-tomcat/tomcat.conf Loading password config: /etc/pki/pki-tomcat/password.conf Loading instance registry: /etc/sysconfig/pki/tomcat/pki-tomcat/pki-tomcat Loading subsystem: ca Loading subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg Getting sslserver cert info for ca from CS.cfg Getting subsystem cert info for ca from CS.cfg Getting audit_signing cert info for ca from CS.cfg Getting ocsp_signing cert info for ca from CS.cfg Getting signing cert info for ca from CS.cfg ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) ra.get_certificate(): Unable to communicate with CMS (403) Unhandler rdtype 28 Unhandler rdtype 28 Unhandler rdtype 28
And nothing else? It should at least report the failures as well as other information.
rob
Corey Devenport via FreeIPA-users wrote:
You'd want to keep the most recently issued. This is likely the highest serial number.
I removed the description with the lower serial number, this doesn't seem to have changed anything. On one of the replicas I was able to get all of the certs to have the status MONITORING by retracking the subsystem certs before the caSigningCert, but even then I can't do any cert operations, and I get the same errors within the ca log about ipa ra and not being able to authenticate.
And nothing else? It should at least report the failures as well as other information.
rob
Here's the full output from ipa-healthcheck --failures-only [ { "source": "ipahealthcheck.dogtag.ca", "check": "DogtagCertsConnectivityCheck", "result": "ERROR", "uuid": "e50d483d-4df3-4b73-b8b7-e16965e37498", "when": "20201112174907Z", "duration": "0.041723", "kw": { "msg": "Request for certificate failed, Certificate operation cannot be completed: Unable to communicate with CMS (403)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "5da33d67-49ee-4da1-93c2-4ef708706a92", "when": "20201112174909Z", "duration": "0.293753", "kw": { "key": "20200903174544", "msg": "Request for certificate failed, Certificate operation cannot be completed: Unable to communicate with CMS (403)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "e2410971-50d1-4f46-a97a-689a8c724f19", "when": "20201112174909Z", "duration": "0.457327", "kw": { "key": "20200903174539", "msg": "Request for certificate failed, Certificate operation cannot be completed: Unable to communicate with CMS (403)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "28c5f45f-cb92-4ed8-b772-3b2ea318ddfd", "when": "20201112174909Z", "duration": "0.606343", "kw": { "key": "20200903174540", "msg": "Request for certificate failed, Certificate operation cannot be completed: Unable to communicate with CMS (403)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "b2439d4d-d3a6-43bc-8f01-aaf82a550ca2", "when": "20201112174909Z", "duration": "0.758415", "kw": { "key": "20200903174541", "msg": "Request for certificate failed, Certificate operation cannot be completed: Unable to communicate with CMS (403)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "1db6d06e-2055-4bb2-a06b-d01090acb79e", "when": "20201112174910Z", "duration": "0.949354", "kw": { "key": "20200903174542", "msg": "Request for certificate failed, Certificate operation cannot be completed: Unable to communicate with CMS (403)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "d463cae9-a700-4f97-b337-b8c4f6bffadf", "when": "20201112174910Z", "duration": "1.142487", "kw": { "key": "20200903174543", "msg": "Request for certificate failed, Certificate operation cannot be completed: Unable to communicate with CMS (403)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "af75c420-ca9e-4fb4-8726-14b1c0e5ece4", "when": "20201112174910Z", "duration": "1.228820", "kw": { "key": "20200903174546", "msg": "Request for certificate failed, Certificate operation cannot be completed: Unable to communicate with CMS (403)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "d4b8fd36-dcad-42a8-8449-fe3d8cf4d684", "when": "20201112174910Z", "duration": "1.382383", "kw": { "key": "20200903174545", "msg": "Request for certificate failed, Certificate operation cannot be completed: Unable to communicate with CMS (403)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "34a18c02-2804-4bbe-aacd-9f4c3d4d06c0", "when": "20201112174910Z", "duration": "1.491313", "kw": { "key": "20200205195905", "msg": "Request for certificate failed, Certificate operation cannot be completed: Unable to communicate with CMS (403)" } }, { "source": "ipahealthcheck.ipa.idns", "check": "IPADNSSystemRecordsCheck", "result": "WARNING", "uuid": "4984a75e-1473-4c22-baa8-30c79be696ec", "when": "20201112174911Z", "duration": "0.112628", "kw": { "msg": "Expected SRV record missing", "key": "_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs....:athena..." } }, There's a bunch more of those warnings regarding SRV records missing, but I don't think that has to deal with our problem, I could be wrong though.
Update: In using the command ipa-certupdate all of the IPA Servers have all the certs as MONITORING, including the caSigningCert. However, the authentication problem persists, and I still get the 403 cannot communicate with CMS when trying to perform cert operations. From what I can tell this is caused by the IPA RA cert, and differences between the LDAP and the cert on the servers, but I can't find any noticeable difference. Is there a way to request a new IPA RA cert? Or force an update so that both LDAP and the servers have the same information?
On 11/17/20 6:27 PM, Corey Devenport via FreeIPA-users wrote:
Update: In using the command ipa-certupdate all of the IPA Servers have all the certs as MONITORING, including the caSigningCert. However, the authentication problem persists, and I still get the 403 cannot communicate with CMS when trying to perform cert operations. From what I can tell this is caused by the IPA RA cert, and differences between the LDAP and the cert on the servers, but I can't find any noticeable difference. Is there a way to request a new IPA RA cert? Or force an update so that both LDAP and the servers have the same information? _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi,
you need first to identify the right RA cert to use. On all the servers, check the content of /var/lib/ipa/ra-agent.pem, for instance with: # openssl x509 -noout -text -in /var/lib/ipa/ra-agent.pem
The right one is likely to be the most recent one, i.e. the one with the farther expiration date.
Then check if this certificate is stored in the entry uid=ipara,ou=people,o=ipaca (run the check on all the servers, if the replication is broken the entry may not have been updated everywhere): # ldapsearch -LLL -o ldif-wrap=no -D "cn=directory\ manager" -W -b uid=ipara,ou=people,o=ipaca dn usercertificate description
You should see the certificate blob from /var/lib/ipa/ra-agent.pem in one of the attribute values for "usercertificate".
The final check is the description attribute. It needs to contain something like: description: 2;<serial>;<issuer>;<subject> where serial, issuer and subject are the same values as in ra-agent.pem.
HTH, flo
On 11/17/20 6:27 PM, Corey Devenport via FreeIPA-users wrote:
Hi,
you need first to identify the right RA cert to use. On all the servers, check the content of /var/lib/ipa/ra-agent.pem, for instance with: # openssl x509 -noout -text -in /var/lib/ipa/ra-agent.pem
The right one is likely to be the most recent one, i.e. the one with the farther expiration date.
Then check if this certificate is stored in the entry uid=ipara,ou=people,o=ipaca (run the check on all the servers, if the replication is broken the entry may not have been updated everywhere): # ldapsearch -LLL -o ldif-wrap=no -D "cn=directory\ manager" -W -b uid=ipara,ou=people,o=ipaca dn usercertificate description
You should see the certificate blob from /var/lib/ipa/ra-agent.pem in one of the attribute values for "usercertificate".
The final check is the description attribute. It needs to contain something like: description: 2;<serial>;<issuer>;<subject> where serial, issuer and subject are the same values as in ra-agent.pem.
HTH, flo
Hey Flo, I did the steps you mentioned above and found the following: All of the IPA Servers are using the same cert with the serial number 805... The same usercertificate is found in LDAP on all three servers, as well as the same description. Interestingly enough, there is another cert which is found in LDAP that was issued the exact same day as the 805... cert, but it has the serial number of 804... and was issued earlier that same day. Should I try switching everything over to that cert and see if that resolves the issue? If so what would be the best way to go about that?
On 11/18/20 5:23 PM, Corey Devenport via FreeIPA-users wrote:
On 11/17/20 6:27 PM, Corey Devenport via FreeIPA-users wrote:
Hi,
you need first to identify the right RA cert to use. On all the servers, check the content of /var/lib/ipa/ra-agent.pem, for instance with: # openssl x509 -noout -text -in /var/lib/ipa/ra-agent.pem
The right one is likely to be the most recent one, i.e. the one with the farther expiration date.
Then check if this certificate is stored in the entry uid=ipara,ou=people,o=ipaca (run the check on all the servers, if the replication is broken the entry may not have been updated everywhere): # ldapsearch -LLL -o ldif-wrap=no -D "cn=directory\ manager" -W -b uid=ipara,ou=people,o=ipaca dn usercertificate description
You should see the certificate blob from /var/lib/ipa/ra-agent.pem in one of the attribute values for "usercertificate".
The final check is the description attribute. It needs to contain something like: description: 2;<serial>;<issuer>;<subject> where serial, issuer and subject are the same values as in ra-agent.pem.
HTH, flo
Hey Flo, I did the steps you mentioned above and found the following: All of the IPA Servers are using the same cert with the serial number 805... The same usercertificate is found in LDAP on all three servers, as well as the same description. Interestingly enough, there is another cert which is found in LDAP that was issued the exact same day as the 805... cert, but it has the serial number of 804... and was issued earlier that same day. Should I try switching everything over to that cert and see if that resolves the issue? If so what would be the best way to go about that?
Hi,
multiple certs have the same validity period and are likely to expire and get renewed the same day, meaning that 804 is likely a completely different cert.
I would rather add more debug information in order to understand what is happening. You can create a file /etc/ipa/server.conf with the following content: # cat /etc/ipa/server.conf [global] debug=True
Then restart ipa with ipactl restart and check the logs in /var/log/httpd/error_log. When you run the commands # kinit admin # ipa cert-show 1 this should trigger fresh traces in the error_log and may help diagnose.
You may also be able to see more info in /var/log/pki/pki-tomcat/ca/debug.
HTH, flo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On 11/18/20 5:23 PM, Corey Devenport via FreeIPA-users wrote:
Hi,
multiple certs have the same validity period and are likely to expire and get renewed the same day, meaning that 804 is likely a completely different cert.
I would rather add more debug information in order to understand what is happening. You can create a file /etc/ipa/server.conf with the following content: # cat /etc/ipa/server.conf [global] debug=True
Then restart ipa with ipactl restart and check the logs in /var/log/httpd/error_log. When you run the commands # kinit admin # ipa cert-show 1 this should trigger fresh traces in the error_log and may help diagnose.
You may also be able to see more info in /var/log/pki/pki-tomcat/ca/debug.
HTH, flo
Here are the relevant lines from both /var/log/httpd/error_log and /var/log/pki/pki-tomcat/ca/debug: [Mon Nov 23 09:30:19.366663 2020] [ssl:error] [pid 22022:tid 139669164889856] [client ...:33510] AH: verify client post handshake [Mon Nov 23 09:30:19.366732 2020] [ssl:error] [pid 22022:tid 139669164889856] [client ...:33510] AH02263: Re-negotiation handshake failed: Client certificate missing [Mon Nov 23 09:30:19.369427 2020] [wsgi:error] [pid 22018:tid 139669101709056] [remote ...:33504] ipa: DEBUG: response status 403
2020-11-23 09:05:08 [ajp-nio-127.0.0.1-8009-exec-9] WARNING: CertProcessor: No authenticator credentials required 2020-11-23 09:05:08 [ajp-nio-127.0.0.1-8009-exec-9] SEVERE: AgentCertAuthentication: No SSL Client Certs Found 2020-11-23 09:05:08 [ajp-nio-127.0.0.1-8009-exec-9] SEVERE: CAProcessor: authentication error: Invalid Credential.
It seems that it can't find the SSL Client Cert (is that the RA cert?) How would I go about fixing this?
I resolved the issue!!! Turns out it was a bug recently patched this year lol: https://bugzilla.redhat.com/show_bug.cgi?id=1775158
I did the following: dnf update httpd
That solved the issue, what a headache solved by a mere package update.
Thanks for all the help!
freeipa-users@lists.fedorahosted.org
-
Corey Devenport -
Florence Blanc-Renaud -
Rob Crittenden