Hi,
I've been running IPA on CentOS 7 for some time on two servers with integrated CA. With the release of CentOS 8.1 I tried upgrading with a second replica - but scrapped that due to the problem with the wrong samba libraries linked. Since no fix is in sight I thought about migrating to Fedora 32 instead - which I've started yesterday.
Topology: freeipa1 + freeipa2: CentOS Linux release 7.8.2003 (Core) (upgrade from older CentOS 7 releases) DNS, CA, KRA, AD trust freeipa1 is CA renewal master
freeipa3: current Fedora 32 with the same services, ipa-replica-install has chosen freeipa2 to replicate from. I've manually added an aditional replica agreement betwen freeipa1 and freeipa3.
WebUI works, ipactl status is RUNNING, I get kerberos tickets, so I guess we are most likely ok. Replication is also fine.
Before I start decomissioning freeipa2 I checked ipa-healthcheck: $ ipa-healthcheck --output-type human --failures-only
ERROR: pki.server.healthcheck.meta.csconfig.DogtagCertsConfigCheck.kra_transport: Certificate 'transportCert cert-pki-kra' does not match the value of kra.transport.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg ERROR: pki.server.healthcheck.meta.csconfig.DogtagCertsConfigCheck.kra_storage: Certificate 'storageCert cert-pki-kra' does not match the value of kra.storage.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg ERROR: pki.server.healthcheck.meta.csconfig.DogtagCertsConfigCheck.kra_audit_signing: Certificate 'auditSigningCert cert-pki-kra' does not match the value of kra.audit_signing.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConfigCheck.transportCert cert-pki-kra: Certificate 'transportCert cert-pki-kra' does not match the value of ca.connector.KRA.transportCert in /var/lib/pki/pki-tomcat/conf/ca/CS.cfg WARNING: ipahealthcheck.ipa.dna.IPADNARangeCheck: No DNA range defined. If no masters define a range then users and groups cannot be created.
The warning is ok and I know how to deal with that. But for the errors my expactation was that I shouldn't get any certificate errors on a new replica. I've checked the certs/log (here for transportCert only):
args=['/usr/bin/certutil', '-d', 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'transportCert cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] Process finished, return code=0 stdout=-----BEGIN CERTIFICATE----- MIIDdDCCAlygAwIBAgIED/0AUjANBgkqhkiG9w0BAQsFADA1MRMwEQYDVQQKDApK ... LjQX6mD/oR3hZnmE920+ABhk8QcJaRoi -----END CERTIFICATE-----
And:
kra.transport.cert= MIIDdDCCAlygAwIBAgIED/wABTANBgkqhkiG9w0BAQsFADA1MRMwEQYDVQQKDApK T0NIRU4uT1JHMR4wHAYDVQQDDBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMTcx ^first diff [other changes...] Some lines identical [more differences].
So, ipa-healtcheck seems to be right. What's the best way to fix it? And why is a fresh replica not clean?
Thanks for your help, Jochen
Jochen Kellner via FreeIPA-users wrote:
Hi,
I've been running IPA on CentOS 7 for some time on two servers with integrated CA. With the release of CentOS 8.1 I tried upgrading with a second replica - but scrapped that due to the problem with the wrong samba libraries linked. Since no fix is in sight I thought about migrating to Fedora 32 instead - which I've started yesterday.
Topology: freeipa1 + freeipa2: CentOS Linux release 7.8.2003 (Core) (upgrade from older CentOS 7 releases) DNS, CA, KRA, AD trust freeipa1 is CA renewal master
freeipa3: current Fedora 32 with the same services, ipa-replica-install has chosen freeipa2 to replicate from. I've manually added an aditional replica agreement betwen freeipa1 and freeipa3.
WebUI works, ipactl status is RUNNING, I get kerberos tickets, so I guess we are most likely ok. Replication is also fine.
Before I start decomissioning freeipa2 I checked ipa-healthcheck: $ ipa-healthcheck --output-type human --failures-only
ERROR: pki.server.healthcheck.meta.csconfig.DogtagCertsConfigCheck.kra_transport: Certificate 'transportCert cert-pki-kra' does not match the value of kra.transport.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg ERROR: pki.server.healthcheck.meta.csconfig.DogtagCertsConfigCheck.kra_storage: Certificate 'storageCert cert-pki-kra' does not match the value of kra.storage.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg ERROR: pki.server.healthcheck.meta.csconfig.DogtagCertsConfigCheck.kra_audit_signing: Certificate 'auditSigningCert cert-pki-kra' does not match the value of kra.audit_signing.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConfigCheck.transportCert cert-pki-kra: Certificate 'transportCert cert-pki-kra' does not match the value of ca.connector.KRA.transportCert in /var/lib/pki/pki-tomcat/conf/ca/CS.cfg WARNING: ipahealthcheck.ipa.dna.IPADNARangeCheck: No DNA range defined. If no masters define a range then users and groups cannot be created.
The warning is ok and I know how to deal with that. But for the errors my expactation was that I shouldn't get any certificate errors on a new replica. I've checked the certs/log (here for transportCert only):
args=['/usr/bin/certutil', '-d', 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'transportCert cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] Process finished, return code=0 stdout=-----BEGIN CERTIFICATE----- MIIDdDCCAlygAwIBAgIED/0AUjANBgkqhkiG9w0BAQsFADA1MRMwEQYDVQQKDApK ... LjQX6mD/oR3hZnmE920+ABhk8QcJaRoi -----END CERTIFICATE-----
And:
kra.transport.cert= MIIDdDCCAlygAwIBAgIED/wABTANBgkqhkiG9w0BAQsFADA1MRMwEQYDVQQKDApK T0NIRU4uT1JHMR4wHAYDVQQDDBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMTcx ^first diff [other changes...] Some lines identical [more differences].
So, ipa-healtcheck seems to be right. What's the best way to fix it? And why is a fresh replica not clean?
I think we'd need a bit more context. Is T0N a new line/entry, separate from MII? If so I'd run that through a base64 decoder to see what you get. I suspect it is a double-encoded cert. And if it is, what cert is it?
rob
Rob Crittenden via FreeIPA-users freeipa-users@lists.fedorahosted.org writes:
Jochen Kellner via FreeIPA-users wrote:
Topology: freeipa1 + freeipa2: CentOS Linux release 7.8.2003 (Core) (upgrade from older CentOS 7 releases) DNS, CA, KRA, AD trust freeipa1 is CA renewal master
ERROR: pki.server.healthcheck.meta.csconfig.DogtagCertsConfigCheck.kra_transport: Certificate 'transportCert cert-pki-kra' does not match the value of kra.transport.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg ERROR: pki.server.healthcheck.meta.csconfig.DogtagCertsConfigCheck.kra_storage: Certificate 'storageCert cert-pki-kra' does not match the value of kra.storage.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg ERROR: pki.server.healthcheck.meta.csconfig.DogtagCertsConfigCheck.kra_audit_signing: Certificate 'auditSigningCert cert-pki-kra' does not match the value of kra.audit_signing.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConfigCheck.transportCert cert-pki-kra: Certificate 'transportCert cert-pki-kra' does not match the value of ca.connector.KRA.transportCert in /var/lib/pki/pki-tomcat/conf/ca/CS.cfg WARNING: ipahealthcheck.ipa.dna.IPADNARangeCheck: No DNA range defined. If no masters define a range then users and groups cannot be created.
The warning is ok and I know how to deal with that. But for the errors my expactation was that I shouldn't get any certificate errors on a new replica. I've checked the certs/log (here for transportCert only):
args=['/usr/bin/certutil', '-d', 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'transportCert cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] Process finished, return code=0 stdout=-----BEGIN CERTIFICATE----- MIIDdDCCAlygAwIBAgIED/0AUjANBgkqhkiG9w0BAQsFADA1MRMwEQYDVQQKDApK ... LjQX6mD/oR3hZnmE920+ABhk8QcJaRoi -----END CERTIFICATE-----
And:
kra.transport.cert= MIIDdDCCAlygAwIBAgIED/wABTANBgkqhkiG9w0BAQsFADA1MRMwEQYDVQQKDApK T0NIRU4uT1JHMR4wHAYDVQQDDBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMTcx ^first diff [other changes...] Some lines identical [more differences].
So, ipa-healtcheck seems to be right. What's the best way to fix it? And why is a fresh replica not clean?
I think we'd need a bit more context. Is T0N a new line/entry, separate from MII? If so I'd run that through a base64 decoder to see what you get. I suspect it is a double-encoded cert. And if it is, what cert is it?
Yes, I think I need to tell more. In /var/lib/pki/pki-tomcat/kra/conf/CS.cfg there is a line kra.transport.cert=... This is one long line that I broke down like the output from certutil. There are parts in the certificate that are identical, but others are different. I guess, it could be the same CSR, but signed at a different time or something like that. I'm not proficient with certutil, but I'll try to have a deeper look into the certificates.
In IPA I have four certificates for "IPA RA" - one (the oldest) revoked, two are expired in 2017 and 2019 and one valid until next year.
The certificate in CS.cfg is expired:
Serial Number: 268173317 (0xffc0005) ... Validity Not Before: Dec 30 06:29:19 2017 GMT Not After : Dec 20 06:29:19 2019 GMT Subject: O = EXAMPLE.ORG, CN = KRA Transport Certificate
certutl has the correct (valid) cert:
Serial Number: 268238930 (0xffd0052) ... Validity Not Before: Dec 13 13:56:29 2019 GMT Not After : Dec 2 13:56:29 2021 GMT
So, when installing the replica I got an older, expired cert in CS.cfg, but the certificate in nssdb is newer and valid.
I've now checked /var/lib/pki/pki-tomcat/kra/conf/CS.cfg on all three IPA servers, they all have the same expired kra.transport.cert= entry.
I see the certmonger tracking request only with "getcert list", not ipa-getcert.
Request ID '20200606205724': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.ORG subject: CN=KRA Transport Certificate,O=EXAMPLE.ORG expires: 2021-12-02 14:56:29 CET key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "transportCert cert-pki-kra" track: yes auto-renew: yes
I've now looked at the post-save command. There we have:
elif nickname == 'caSigningCert cert-pki-ca': # Update CS.cfg cfg_path = paths.CA_CS_CFG_PATH config = directivesetter.get_directive( cfg_path, 'subsystem.select', '=') if config == 'New': syslog.syslog(syslog.LOG_NOTICE, "Updating CS.cfg")
but nothing similar for the "transportCert cert-pki-kra"... Would that explain what I see? I think I can (and will) fix the CS.cfg files. But they will get wrong when the certificate expires next time.
Thanks for the "I need more context" ping. I looked at IPA bugs but nothing looked similar to this case. OTOH I would expect that far more people would also have this problem. Do I need these certificates for the IPA Vaults? That's something I looked at some time ago, but never really used.
Jochen
Jochen Kellner via FreeIPA-users freeipa-users@lists.fedorahosted.org writes:
In IPA I have four certificates for "IPA RA" - one (the oldest) revoked, two are expired in 2017 and 2019 and one valid until next year.
The certificate in CS.cfg is expired:
Serial Number: 268173317 (0xffc0005)
... Validity Not Before: Dec 30 06:29:19 2017 GMT Not After : Dec 20 06:29:19 2019 GMT Subject: O = EXAMPLE.ORG, CN = KRA Transport Certificate
certutl has the correct (valid) cert:
Serial Number: 268238930 (0xffd0052)
... Validity Not Before: Dec 13 13:56:29 2019 GMT Not After : Dec 2 13:56:29 2021 GMT
So, when installing the replica I got an older, expired cert in CS.cfg, but the certificate in nssdb is newer and valid.
I've fixed that manually on the new replica by copying the valid certificate from LDAP into the CS.cfg files.
Thanks for the "I need more context" ping. I looked at IPA bugs but nothing looked similar to this case. OTOH I would expect that far more people would also have this problem.
I'll see what the last replica looks like after the refresh when all other replicas have been fixed.
Jochen
freeipa-users@lists.fedorahosted.org