On Fri, 26 May 2017, Rob Crittenden wrote:
Rob Foehl via FreeIPA-users wrote:
> On Fri, 26 May 2017, Fraser Tweedale wrote:
>
>> What is the validity of the leaf certificates? Is the notAfter time
>> of the leaf certificate pegged to the notAfter time of the CA
>> certificate? If so, this is (IMO) a bug.
>
> The leaf certs' expiration is pegged to that of the CA cert that was
> used to issue them -- the old one, in this case -- but that is expected
> behavior for any CA. It wouldn't be semantically valid otherwise, and
> there's no guarantee that the CA cert will actually be renewed without
> changing the key.
>
> The odd behavior here is that certmonger woke up, noticed that every IPA
> cert including the externally-signed IPA CA needed to be renewed, and
> immediately caused the CA to renew them all. The IPA CA cert itself
> yielded a log entry like this:
>
> May 25 00:25:21
ipa.example.com dogtag-ipa-ca-renew-agent-submit[868]:
> Certificate with subject 'CN=Certificate Authority,O=EXAMPLE.COM' is
> about to expire, use ipa-cacert-manage to renew it
>
> The other 7 or so IPA-generated certificates (host, RA, OCSP, etc.) were
> renewed using the existing CA cert, with new validity periods tied to
> that cert. As mentioned, certmonger would likely figure this out and
> renew them all again using the since-replaced CA cert within the ~2 week
> period until they all expire again, but this seems like unexpected
> behavior when the IPA CA cert is signed by an external CA and can't be
> auto-renewed.
>
> (Actually, based on the order the renewals were submitted, this seems
> like it'd be an issue even if the CA cert were automatically renewed --
> it wasn't the first one to be submitted, either. Incidentally, the
> certs which were renewed aren't a complete list -- both the
> "CN=ipa-ca-agent" and "CN=Object Signing Cert" certs weren't
renewed and
> aren't tracked by certmonger.)
certmonger doesn't have the context to know internal vs external. It
just knows a cert is expiring within its window so it renews it. IMHO
this is completely expected.
Right, I wouldn't expect it to know the provenance of the CA cert... I am
wondering whether it should be able to recognize the dependency between
the certs, though -- it should be able to recognize the chain, at least.
I believe that certmonger will renew it again as the final day
approaches.
So, out of curiosity, I left the VM running through the original CA
expiration date to see what would happen. The results aren't pretty:
- the running httpd kept using the old certificate (and CA chain), which
broke https sessions to the UI/API (as might be expected);
- certmonger thinks it's renewed everything, with the new expiration dates
lined up with that of the replaced external CA;
- none of the services recognize that they have new certs installed, for
example the same httpd issue as seen in another thread:
[Fri Jun 09 01:07:37.413789 2017] [:error] [pid 14616] SSL Library Error: -8162 The
certificate issuer's certificate has expired. Check your system date and time
[Fri Jun 09 01:07:37.413828 2017] [:error] [pid 14616] Unable to verify certificate
'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the server
can start until the problem can be resolved.
vs.
# getcert list -d /etc/httpd/alias -n Server-Cert
Number of certificates and requests being tracked: 8.
Request ID '20170508063315':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate
Authority,O=EXAMPLE.COM
subject:
CN=ipa1.example.com,O=EXAMPLE.COM
expires: 2017-06-24 04:32:24 UTC
dns:
ipa1.example.com
principal name: HTTP/ipa1.example.com(a)EXAMPLE.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
- neither httpd nor pki-tomcatd will (re)start as a result, the former
fails and the latter just spews stack traces into the logs every
minute if forcibly started (even if httpd is band-aided first):
Jun 09 01:14:24
ipa1.example.com server[15236]: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@43523130 background process
Jun 09 01:14:24
ipa1.example.com server[15236]: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Jun 09 01:14:24
ipa1.example.com server[15236]: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:130)
Jun 09 01:14:24
ipa1.example.com server[15236]: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1154)
Jun 09 01:14:24
ipa1.example.com server[15236]: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5707)
Jun 09 01:14:24
ipa1.example.com server[15236]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
Jun 09 01:14:24
ipa1.example.com server[15236]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
Jun 09 01:14:24
ipa1.example.com server[15236]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
Jun 09 01:14:24
ipa1.example.com server[15236]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
Jun 09 01:14:24
ipa1.example.com server[15236]: at
java.lang.Thread.run(Thread.java:748)
- certmonger never actually logged anything about replacing these certs a
second time, and the first round were signed with the now-expired CA
cert instead of the new one;
- the UI is only partially functional, after clicking through the
certificate warnings in the browser;
- pki-tomcatd is non-functional even if forcibly started, leading to
repeated "IPA Error 4301: CertificateOperationError" in the UI when
trying to access anything CA-related;
- possibly other issues I haven't discovered yet.
In short, that didn't go particularly well at all, which in some ways
brings me back to the original as-yet-unanswered deployment question:
Is trying to do this with an external CA worth the pain?
If I go that route, at some point I'm going to have to replace the CA
cert -- and between this test VM and the "phase 2" mentioned in
https://www.freeipa.org/page/V4/Distribution_of_CA_certificates_to_clients
and the linked Pagure issue 4322, I have basically zero confidence in any
of this...
-Rob