Hi Florence,
I just posted that the problem is solved, but thank for coming back to me!
Now (on the fixed system) I get:
$ getcert list-cas -c IPA
CA 'IPA':
is-default: no
ca-type: EXTERNAL
helper-location: /usr/libexec/certmonger/ipa-server-guard
/usr/libexec/certmonger/ipa-submit
One thing I didn't mention in the previous post is that the ACL was also
gone, I had to recreate it manually.
Now it looks like this:
$ ipa caacl-find
----------------
1 CA ACL matched
----------------
ACL name: hosts_services_caIPAserviceCert
Enabled: TRUE
Host category: all
Service category: all
Profiles: caIPAserviceCert
----------------------------
Number of entries returned 1
----------------------------
On Thu, 8 Jun 2017 at 11:02 Roberto Cornacchia <roberto.cornacchia(a)gmail.com>
wrote:
It seems solved now, reporting back.
It looks to me like in February, when the certificate renewal failed, I
had hit the bug described here:
https://www.redhat.com/archives/freeipa-users/2016-February/msg00441.html
Yesterday I updated the packages, including the fix to this bug, but then
I still had an expired certificate. Which didn't allow to complete
ipa-server-upgrade.
Went back in time, asked certmonger to renew, but I was then missing
certificate profiles, because the upgrade was not completed.
Now however the certificate was valid, because the date was changed. With
that, I could manually run ipa-server-upgrade, which successfully imported
all profiles.
Restarted ipa, restarted certmonger, got new certificates.
Went back to today's date, restarted ipa, and everything seems fine.
On Wed, 7 Jun 2017 at 23:25 Roberto Cornacchia <
roberto.cornacchia(a)gmail.com> wrote:
> A relatively good news:
> The current error (Insufficient access: Principal 'HTTP/
> spinque04.hq.spinque.com(a)HQ.SPINQUE.COM' is not permitted to use CA '.'
> with profile 'caIPAserviceCert' for certificate issuance.) might not be
> due to the package upgrade.
>
> I looked at the journal of 16 Feb 2017 (28 days before the expiration
> date): certmonger correctly tries to renew the certificate but fails with
> the exact same error as I have now. So this explains why I ended up with an
> expired certificate in the first place.
>
> So hopefully I'm back to the original issue that caused all this. Any
> help is highly appreciated.
>
>
> On Wed, 7 Jun 2017 at 19:15 Rob Crittenden <rcritten(a)redhat.com> wrote:
>
>> Roberto Cornacchia via FreeIPA-users wrote:
>> > Sorry for accidentally dropping freeipa-users.
>> >
>> > I was impatient so went back in time before your answer, but I did
>> chose
>> > a good date
>> >
>> > Before this, I had the following two entries with an expired date:
>> >
>> > Request ID '20150316184508':
>> > status: NEED_TO_SUBMIT
>> > ca-error: Error setting up ccache for "host" service on client
using
>> > default keytab: Cannot contact any KDC for requested realm.
>> >
>> > Request ID '20150316184529':
>> > status: CA_UNREACHABLE
>> > ca-error: Error setting up ccache for "host" service on client
using
>> > default keytab: Cannot contact any KDC for requested realm.
>> >
>> > After restarting certmonger, I have:
>> >
>> > Request ID '20150316184508':
>> > status: MONITORING
>> > ca-error: Server at
https://spinque04.hq.spinque.com/ipa/xml
>> > denied our request, giving up: 2100 (RPC failed at server.
>> Insufficient
>> > access: Principal 'ldap/spinque04.hq.spinque.com(a)HQ.SPINQUE.COM
>> > <mailto:spinque04.hq.spinque.com@HQ.SPINQUE.COM>' is not permitted
to
>> > use CA '.' with profile 'caIPAserviceCert' for certificate
issuance.).
>> >
>> > Request ID '20150316184529':
>> > status: MONITORING
>> > ca-error: Server at
https://spinque04.hq.spinque.com/ipa/xml
>> > denied our request, giving up: 2100 (RPC failed at server.
>> Insufficient
>> > access: Principal 'HTTP/spinque04.hq.spinque.com(a)HQ.SPINQUE.COM
>> > <mailto:spinque04.hq.spinque.com@HQ.SPINQUE.COM>' is not permitted
to
>> > use CA '.' with profile 'caIPAserviceCert' for certificate
issuance.).
>>
>> I think this is a side-effect of updating the packages with expired
>> certs (you are about half upgraded right now). The new CA ACL rules
>> don't seem to have been applied. I'm not sure what the safest course in,
>> maybe Flo or Fraser know.
>>
>> rob
>>
>> >
>> > The journal shows continuous Tomcat errors such as this:
>> >
>> > Mar 01 00:09:28
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> named-pkcs11[5692]: validating
>> >
docs.oracle.com/CNAME <
http://docs.oracle.com/CNAME>: bad cache hit
>> > (
oracle.com/DS <
http://oracle.com/DS>)
>> > Mar 01 00:09:28
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> named-pkcs11[5692]: broken trust
>> chain
>> > resolving 'docs.oracle.com/A/IN
<
http://docs.oracle.com/A/IN>';:
>> 8.8.8.8#53
>> > Mar 01 00:09:28
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> named-pkcs11[5692]: validating
>> >
docs.oracle.com/CNAME <
http://docs.oracle.com/CNAME>: bad cache hit
>> > (
oracle.com/DS <
http://oracle.com/DS>)
>> > Mar 01 00:09:28
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> named-pkcs11[5692]: broken trust
>> chain
>> > resolving 'docs.oracle.com/AAAA/IN
<
http://docs.oracle.com/AAAA/IN>';:
>> > 8.8.8.8#53
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]: Mar 01, 2017 12:09:32
>> AM
>> > org.apache.catalina.core.ContainerBase backgroundProcess
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]: WARNING: Exception
>> > processing realm com.netscape.cms.tomcat.ProxyRealm@4077b502
>> background
>> > process
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]:
>> > java.lang.NullPointerException
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]: at
>> >
>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:109)
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]: at
>> >
>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1154)
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]: at
>> >
>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5697)
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]: at
>> >
>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]: at
>> >
>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]: at
>> >
>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]: at
>> >
>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
>> > Mar 01 00:09:32
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com> server[5912]: at
>> > java.lang.Thread.run(Thread.java:745)
>> >
>> > On Wed, 7 Jun 2017 at 16:22 Rob Crittenden <rcritten(a)redhat.com
>> > <mailto:rcritten@redhat.com>> wrote:
>> >
>> > Adding freeipa-users back.
>> >
>> > Roberto Cornacchia wrote:
>> > > Just to confirm, this is indeed the expired certificate
>> > >
>> > > $ getcert list -d /etc/httpd/alias -n Server-Cert
>> > > Number of certificates and requests being tracked: 8.
>> > > Request ID '20150316184529':
>> > > status: CA_UNREACHABLE
>> > > ca-error: Error setting up ccache for "host" service on
client
>> using
>> > > default keytab: Cannot contact any KDC for requested realm.
>> > > stuck: no
>> > > key pair storage:
>> > >
>> >
>>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>> > > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>> > > certificate:
>> > >
>> >
>>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>> > > Certificate DB'
>> > > CA: IPA
>> > > issuer: CN=Certificate
Authority,O=HQ.SPINQUE.COM
>> > <
http://HQ.SPINQUE.COM> <
http://HQ.SPINQUE.COM>
>> > > subject:
CN=spinque04.hq.spinque.com <
>>
http://spinque04.hq.spinque.com>
>> > > <
http://spinque04.hq.spinque.com>,O=HQ.SPINQUE.COM
>> > <
http://HQ.SPINQUE.COM> <
http://HQ.SPINQUE.COM>
>> > > expires: 2017-03-16 18:45:29 UTC
>> > > key usage:
>> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> > > eku: id-kp-serverAuth,id-kp-clientAuth
>> > > pre-save command:
>> > > post-save command: /usr/lib64/ipa/certmonger/restart_httpd
>> > > track: yes
>> > > auto-renew: yes
>> >
>> > Since you have Apache up and limping along I'd first try:
>> >
>> > # getcert resubmit -i 20150316184529
>> >
>> > It is very possible that the tool will reject the server cert
>> since it
>> > is expired. If that happens then you'll need to set time back.
>> >
>> > I'd encourage you to run: getcert list | grep expires
>> >
>> > This will show you what certs need to be renewed and should give
>> some
>> > insight into the window of opportunity for setting the date back.
>> >
>> > Assuming it's just the Apache cert that is expired I'd try
setting
>> the
>> > date back to March 15, restart IPA, then I'd restart certmonger or
>> try
>> > the about resubmit command again.
>> >
>> > Things can get tricky when some certs have renewed and some
>> haven't.
>> >
>> > rob
>> >
>> > >
>> > >
>> > > On Wed, 7 Jun 2017 at 15:36 Roberto Cornacchia
>> > > <roberto.cornacchia(a)gmail.com
>> > <mailto:roberto.cornacchia@gmail.com>
>> > <mailto:roberto.cornacchia@gmail.com
>> > <mailto:roberto.cornacchia@gmail.com>>> wrote:
>> > >
>> > > Thanks Rob,
>> > >
>> > > I've seen in similar posts that you recommend to go back
in
>> > time and
>> > > restart certmonger. Would it work in this case?
>> > >
>> > >
>> > > On Wed, 7 Jun 2017 at 15:30 Rob Crittenden
>> > <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
>> > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>
>> wrote:
>> > >
>> > > Roberto Cornacchia via FreeIPA-users wrote:
>> > > > OK, I did so and httpd restarts.
>> > > >
>> > > > $ openssl s_client -connect 127.0.0.1:443
>> > <
http://127.0.0.1:443>
>> > > <
http://127.0.0.1:443> <
http://127.0.0.1:443>
-showcerts
>> > > > CONNECTED(00000003)
>> > > > depth=1 O =
HQ.SPINQUE.COM
<
http://HQ.SPINQUE.COM>
>> > <
http://HQ.SPINQUE.COM>
>> > > <
http://HQ.SPINQUE.COM>, CN = Certificate
>> > > > Authority
>> > > > verify return:1
>> > > > depth=0 O =
HQ.SPINQUE.COM
<
http://HQ.SPINQUE.COM>
>> > <
http://HQ.SPINQUE.COM>
>> > > <
http://HQ.SPINQUE.COM>, CN =
>> > > >
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com>
<
http://spinque04.hq.spinque.com
>> >
>> > > <
http://spinque04.hq.spinque.com>
>> > > > verify error:num=10:certificate has expired
>> > > > notAfter=Mar 16 18:45:29 2017 GMT
>> > > > verify return:1
>> > > > depth=0 O =
HQ.SPINQUE.COM
<
http://HQ.SPINQUE.COM>
>> > <
http://HQ.SPINQUE.COM>
>> > > <
http://HQ.SPINQUE.COM>, CN =
>> > > >
spinque04.hq.spinque.com
>> > <
http://spinque04.hq.spinque.com>
<
http://spinque04.hq.spinque.com
>> >
>> > > <
http://spinque04.hq.spinque.com>
>> > > > notAfter=Mar 16 18:45:29 2017 GMT
>> > > > verify return:1
>> > > > ---
>> > > > Certificate chain
>> > > > 0
s:/O=HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com
>> > <
http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
>> > > <
http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
>> > > >
<
http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
>> > > >
i:/O=HQ.SPINQUE.COM/CN=Certificate
>> > <
http://HQ.SPINQUE.COM/CN=Certificate>
>> > > <
http://HQ.SPINQUE.COM/CN=Certificate>
>> > > > <
http://HQ.SPINQUE.COM/CN=Certificate>
Authority
>> > > > ...
>> > > >
>> > > > Fair enough, but why does this say it expires in
2019?
>> Are
>> > > they two
>> > > > different certificates?
>> > > >
>> > > > $ getcert list -d /etc/httpd/alias -n ipaCert
>> > > > Number of certificates and requests being tracked: 8.
>> > > > Request ID '20160501114633':
>> > > > status: MONITORING
>> > > > stuck: no
>> > > > key pair storage:
>> > > >
>> > >
>> >
>>
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>> > > > Certificate
DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>> > > > certificate:
>> > > >
>> > >
>> >
>>
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>> > > > Certificate DB'
>> > > > CA: dogtag-ipa-ca-renew-agent
>> > > > issuer: CN=Certificate
Authority,O=HQ.SPINQUE.COM
>> > <
http://HQ.SPINQUE.COM>
>> > > <
http://HQ.SPINQUE.COM>
<
http://HQ.SPINQUE.COM>
>> > > > subject: CN=IPA
RA,O=HQ.SPINQUE.COM
>> > <
http://HQ.SPINQUE.COM> <
http://HQ.SPINQUE.COM>
>> > > <
http://HQ.SPINQUE.COM>
>> > > > expires: 2019-01-26 19:41:51 UTC
>> > > > key usage:
>> > >
>> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> > > > eku: id-kp-serverAuth,id-kp-clientAuth
>> > > > pre-save command:
>> > /usr/lib64/ipa/certmonger/renew_ra_cert_pre
>> > > > post-save command:
>> /usr/lib64/ipa/certmonger/renew_ra_cert
>> > > > track: yes
>> > > > auto-renew: yes
>> > > >
>> > > > What's the right way to solve this?
>> > >
>> > > You're looking at the wrong cert.
>> > >
>> > > # getcert list -d /etc/httpd/alias -n Server-Cert
>> > >
>> > > And really, you should examine all certificate status,
>> not
>> > just
>> > > a single
>> > > one.
>> > >
>> > > I was also strongly urge you to wait until all problems
>> > are resolved
>> > > before attempting to update packages in the future
>> (unless
>> > a package
>> > > claims to fix a specific problem), particularly when it
>> > comes to
>> > > certificates.
>> > >
>> > > rob
>> > >
>> >
>> >
>> >
>> > _______________________________________________
>> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> > To unsubscribe send an email to
>> freeipa-users-leave(a)lists.fedorahosted.org
>> >
>>
>>