It seems solved now, reporting back.
It looks to me like in February, when the certificate renewal failed, I had
hit the bug described here:
Yesterday I updated the packages, including the fix to this bug, but then I
still had an expired certificate. Which didn't allow to complete
ipa-server-upgrade.
Went back in time, asked certmonger to renew, but I was then missing
certificate profiles, because the upgrade was not completed.
Now however the certificate was valid, because the date was changed. With
that, I could manually run ipa-server-upgrade, which successfully imported
all profiles.
Restarted ipa, restarted certmonger, got new certificates.
Went back to today's date, restarted ipa, and everything seems fine.
On Wed, 7 Jun 2017 at 23:25 Roberto Cornacchia <roberto.cornacchia(a)gmail.com>
wrote:
A relatively good news:
The current error (Insufficient access: Principal 'HTTP/
spinque04.hq.spinque.com(a)HQ.SPINQUE.COM' is not permitted to use CA '.'
with profile 'caIPAserviceCert' for certificate issuance.) might not be
due to the package upgrade.
I looked at the journal of 16 Feb 2017 (28 days before the expiration
date): certmonger correctly tries to renew the certificate but fails with
the exact same error as I have now. So this explains why I ended up with an
expired certificate in the first place.
So hopefully I'm back to the original issue that caused all this. Any help
is highly appreciated.
On Wed, 7 Jun 2017 at 19:15 Rob Crittenden <rcritten(a)redhat.com> wrote:
> Roberto Cornacchia via FreeIPA-users wrote:
> > Sorry for accidentally dropping freeipa-users.
> >
> > I was impatient so went back in time before your answer, but I did chose
> > a good date
> >
> > Before this, I had the following two entries with an expired date:
> >
> > Request ID '20150316184508':
> > status: NEED_TO_SUBMIT
> > ca-error: Error setting up ccache for "host" service on client using
> > default keytab: Cannot contact any KDC for requested realm.
> >
> > Request ID '20150316184529':
> > status: CA_UNREACHABLE
> > ca-error: Error setting up ccache for "host" service on client using
> > default keytab: Cannot contact any KDC for requested realm.
> >
> > After restarting certmonger, I have:
> >
> > Request ID '20150316184508':
> > status: MONITORING
> > ca-error: Server at
https://spinque04.hq.spinque.com/ipa/xml
> > denied our request, giving up: 2100 (RPC failed at server. Insufficient
> > access: Principal 'ldap/spinque04.hq.spinque.com(a)HQ.SPINQUE.COM
> > <mailto:spinque04.hq.spinque.com@HQ.SPINQUE.COM>' is not permitted to
> > use CA '.' with profile 'caIPAserviceCert' for certificate
issuance.).
> >
> > Request ID '20150316184529':
> > status: MONITORING
> > ca-error: Server at
https://spinque04.hq.spinque.com/ipa/xml
> > denied our request, giving up: 2100 (RPC failed at server. Insufficient
> > access: Principal 'HTTP/spinque04.hq.spinque.com(a)HQ.SPINQUE.COM
> > <mailto:spinque04.hq.spinque.com@HQ.SPINQUE.COM>' is not permitted to
> > use CA '.' with profile 'caIPAserviceCert' for certificate
issuance.).
>
> I think this is a side-effect of updating the packages with expired
> certs (you are about half upgraded right now). The new CA ACL rules
> don't seem to have been applied. I'm not sure what the safest course in,
> maybe Flo or Fraser know.
>
> rob
>
> >
> > The journal shows continuous Tomcat errors such as this:
> >
> > Mar 01 00:09:28
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> named-pkcs11[5692]: validating
> >
docs.oracle.com/CNAME <
http://docs.oracle.com/CNAME>: bad cache hit
> > (
oracle.com/DS <
http://oracle.com/DS>)
> > Mar 01 00:09:28
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> named-pkcs11[5692]: broken trust
> chain
> > resolving 'docs.oracle.com/A/IN <
http://docs.oracle.com/A/IN>';:
> 8.8.8.8#53
> > Mar 01 00:09:28
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> named-pkcs11[5692]: validating
> >
docs.oracle.com/CNAME <
http://docs.oracle.com/CNAME>: bad cache hit
> > (
oracle.com/DS <
http://oracle.com/DS>)
> > Mar 01 00:09:28
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> named-pkcs11[5692]: broken trust
> chain
> > resolving 'docs.oracle.com/AAAA/IN
<
http://docs.oracle.com/AAAA/IN>';:
> > 8.8.8.8#53
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]: Mar 01, 2017 12:09:32
> AM
> > org.apache.catalina.core.ContainerBase backgroundProcess
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]: WARNING: Exception
> > processing realm com.netscape.cms.tomcat.ProxyRealm@4077b502 background
> > process
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]:
> > java.lang.NullPointerException
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]: at
> >
> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:109)
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]: at
> >
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1154)
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]: at
> >
>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5697)
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]: at
> >
>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]: at
> >
>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]: at
> >
>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]: at
> >
>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
> > Mar 01 00:09:32
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com> server[5912]: at
> > java.lang.Thread.run(Thread.java:745)
> >
> > On Wed, 7 Jun 2017 at 16:22 Rob Crittenden <rcritten(a)redhat.com
> > <mailto:rcritten@redhat.com>> wrote:
> >
> > Adding freeipa-users back.
> >
> > Roberto Cornacchia wrote:
> > > Just to confirm, this is indeed the expired certificate
> > >
> > > $ getcert list -d /etc/httpd/alias -n Server-Cert
> > > Number of certificates and requests being tracked: 8.
> > > Request ID '20150316184529':
> > > status: CA_UNREACHABLE
> > > ca-error: Error setting up ccache for "host" service on
client
> using
> > > default keytab: Cannot contact any KDC for requested realm.
> > > stuck: no
> > > key pair storage:
> > >
> >
>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
> > > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
> > > certificate:
> > >
> >
>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
> > > Certificate DB'
> > > CA: IPA
> > > issuer: CN=Certificate
Authority,O=HQ.SPINQUE.COM
> > <
http://HQ.SPINQUE.COM> <
http://HQ.SPINQUE.COM>
> > > subject:
CN=spinque04.hq.spinque.com <
>
http://spinque04.hq.spinque.com>
> > > <
http://spinque04.hq.spinque.com>,O=HQ.SPINQUE.COM
> > <
http://HQ.SPINQUE.COM> <
http://HQ.SPINQUE.COM>
> > > expires: 2017-03-16 18:45:29 UTC
> > > key usage:
> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > > eku: id-kp-serverAuth,id-kp-clientAuth
> > > pre-save command:
> > > post-save command: /usr/lib64/ipa/certmonger/restart_httpd
> > > track: yes
> > > auto-renew: yes
> >
> > Since you have Apache up and limping along I'd first try:
> >
> > # getcert resubmit -i 20150316184529
> >
> > It is very possible that the tool will reject the server cert since
> it
> > is expired. If that happens then you'll need to set time back.
> >
> > I'd encourage you to run: getcert list | grep expires
> >
> > This will show you what certs need to be renewed and should give
> some
> > insight into the window of opportunity for setting the date back.
> >
> > Assuming it's just the Apache cert that is expired I'd try setting
> the
> > date back to March 15, restart IPA, then I'd restart certmonger or
> try
> > the about resubmit command again.
> >
> > Things can get tricky when some certs have renewed and some haven't.
> >
> > rob
> >
> > >
> > >
> > > On Wed, 7 Jun 2017 at 15:36 Roberto Cornacchia
> > > <roberto.cornacchia(a)gmail.com
> > <mailto:roberto.cornacchia@gmail.com>
> > <mailto:roberto.cornacchia@gmail.com
> > <mailto:roberto.cornacchia@gmail.com>>> wrote:
> > >
> > > Thanks Rob,
> > >
> > > I've seen in similar posts that you recommend to go back in
> > time and
> > > restart certmonger. Would it work in this case?
> > >
> > >
> > > On Wed, 7 Jun 2017 at 15:30 Rob Crittenden
> > <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>
> wrote:
> > >
> > > Roberto Cornacchia via FreeIPA-users wrote:
> > > > OK, I did so and httpd restarts.
> > > >
> > > > $ openssl s_client -connect 127.0.0.1:443
> > <
http://127.0.0.1:443>
> > > <
http://127.0.0.1:443> <
http://127.0.0.1:443>
-showcerts
> > > > CONNECTED(00000003)
> > > > depth=1 O =
HQ.SPINQUE.COM <
http://HQ.SPINQUE.COM>
> > <
http://HQ.SPINQUE.COM>
> > > <
http://HQ.SPINQUE.COM>, CN = Certificate
> > > > Authority
> > > > verify return:1
> > > > depth=0 O =
HQ.SPINQUE.COM <
http://HQ.SPINQUE.COM>
> > <
http://HQ.SPINQUE.COM>
> > > <
http://HQ.SPINQUE.COM>, CN =
> > > >
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com>
<
http://spinque04.hq.spinque.com>
> > > <
http://spinque04.hq.spinque.com>
> > > > verify error:num=10:certificate has expired
> > > > notAfter=Mar 16 18:45:29 2017 GMT
> > > > verify return:1
> > > > depth=0 O =
HQ.SPINQUE.COM <
http://HQ.SPINQUE.COM>
> > <
http://HQ.SPINQUE.COM>
> > > <
http://HQ.SPINQUE.COM>, CN =
> > > >
spinque04.hq.spinque.com
> > <
http://spinque04.hq.spinque.com>
<
http://spinque04.hq.spinque.com>
> > > <
http://spinque04.hq.spinque.com>
> > > > notAfter=Mar 16 18:45:29 2017 GMT
> > > > verify return:1
> > > > ---
> > > > Certificate chain
> > > > 0
s:/O=HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com
> > <
http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
> > > <
http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
> > > > <
http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
> > > >
i:/O=HQ.SPINQUE.COM/CN=Certificate
> > <
http://HQ.SPINQUE.COM/CN=Certificate>
> > > <
http://HQ.SPINQUE.COM/CN=Certificate>
> > > > <
http://HQ.SPINQUE.COM/CN=Certificate> Authority
> > > > ...
> > > >
> > > > Fair enough, but why does this say it expires in 2019?
> Are
> > > they two
> > > > different certificates?
> > > >
> > > > $ getcert list -d /etc/httpd/alias -n ipaCert
> > > > Number of certificates and requests being tracked: 8.
> > > > Request ID '20160501114633':
> > > > status: MONITORING
> > > > stuck: no
> > > > key pair storage:
> > > >
> > >
> >
>
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
> > > > Certificate
DB',pinfile='/etc/httpd/alias/pwdfile.txt'
> > > > certificate:
> > > >
> > >
> >
>
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
> > > > Certificate DB'
> > > > CA: dogtag-ipa-ca-renew-agent
> > > > issuer: CN=Certificate
Authority,O=HQ.SPINQUE.COM
> > <
http://HQ.SPINQUE.COM>
> > > <
http://HQ.SPINQUE.COM> <
http://HQ.SPINQUE.COM>
> > > > subject: CN=IPA
RA,O=HQ.SPINQUE.COM
> > <
http://HQ.SPINQUE.COM> <
http://HQ.SPINQUE.COM>
> > > <
http://HQ.SPINQUE.COM>
> > > > expires: 2019-01-26 19:41:51 UTC
> > > > key usage:
> > >
> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > > > eku: id-kp-serverAuth,id-kp-clientAuth
> > > > pre-save command:
> > /usr/lib64/ipa/certmonger/renew_ra_cert_pre
> > > > post-save command:
> /usr/lib64/ipa/certmonger/renew_ra_cert
> > > > track: yes
> > > > auto-renew: yes
> > > >
> > > > What's the right way to solve this?
> > >
> > > You're looking at the wrong cert.
> > >
> > > # getcert list -d /etc/httpd/alias -n Server-Cert
> > >
> > > And really, you should examine all certificate status, not
> > just
> > > a single
> > > one.
> > >
> > > I was also strongly urge you to wait until all problems
> > are resolved
> > > before attempting to update packages in the future (unless
> > a package
> > > claims to fix a specific problem), particularly when it
> > comes to
> > > certificates.
> > >
> > > rob
> > >
> >
> >
> >
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to
> freeipa-users-leave(a)lists.fedorahosted.org
> >
>
>