Kees Bakker via FreeIPA-users wrote:
> On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
>> On 26-10-18 18:20, Florence Blanc-Renaud wrote:
>>> On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
>>>> On 26-10-18 18:00, Timo Aaltonen wrote:
>>>>> On 26.10.2018 18.59, Kees Bakker wrote:
>>>>>> On 26-10-18 14:55, Timo Aaltonen wrote:
>>>>>>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
>>>>>>>> On 25-10-18 20:46, Timo Aaltonen wrote:
>>>>>>>>> On 25.10.2018 21.44, Rob Crittenden wrote:
>>>>>>>>>> Kees Bakker wrote:
>>>>>>>>>>> On 25-10-18 16:11, Rob Crittenden wrote:
>>>>>>>>>>>> Kees Bakker via FreeIPA-users wrote:
>>>>>>>>>>>>> On 25-10-18 14:18, Rob Crittenden
wrote:
>>>>>>>>>>>>>> Kees Bakker via FreeIPA-users
wrote:
>>>>>>>>>>>>>>> Could it be that this error
already existed since we started? Notice
>>>>>>>>>>>>>>> the Request ID of 2016...,
and the expires: 2018-10-24.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # getcert list -n ipaCert |
sed blabla
>>>>>>>>>>>>>>> Number of certificates and
requests being tracked: 8.
>>>>>>>>>>>>>>> Request ID
'20161103094546':
>>>>>>>>>>>>>>> status: CA_UNREACHABLE
>>>>>>>>>>>>>>> ca-error: Error 77
connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL
CA cert (path? access rights?).
>>>>>>>>>>>>>>> stuck: no
>>>>>>>>>>>>>>> key pair storage:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt'
>>>>>>>>>>>>>>> certificate:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB'
>>>>>>>>>>>>>>> CA:
dogtag-ipa-ca-renew-agent
>>>>>>>>>>>>>>> issuer: CN=Certificate
Authority,O=MYDOMAIN
>>>>>>>>>>>>>>> subject: CN=IPA
RA,O=MYDOMAIN
>>>>>>>>>>>>>>> expires: 2018-10-24
08:45:40 UTC
>>>>>>>>>>>>>>> key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>>>>>>>>>>>>>> eku:
id-kp-serverAuth,id-kp-clientAuth
>>>>>>>>>>>>>>> pre-save command:
/usr/lib/ipa/certmonger/renew_ra_cert_pre
>>>>>>>>>>>>>>> post-save command:
/usr/lib/ipa/certmonger/renew_ra_cert
>>>>>>>>>>>>>>> track: yes
>>>>>>>>>>>>>>> auto-renew: yes
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In other words, is this the
same issue as
https://pagure.io/freeipa/issue/7422 ?
>>>>>>>>>>>>>> The problem is your certs expired
yesterday so connections won't work
>>>>>>>>>>>>>> (the code and message don't
come from within certmonger).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> certmonger _should_ have renewed
them. Try killing ntpd, going back a
>>>>>>>>>>>>>> few days, restart krb5kdc,
dirsrv, httpd and the CA then certmonger and
>>>>>>>>>>>>>> see what happens.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Easy for you to say. You know what
you're doing :-)
>>>>>>>>>>>>> For me it's all magic.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyway, I'll try it. I'm just
scared to set the clock back, because there may
>>>>>>>>>>>>> be clients in the network that use
this server as a NTP server.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Another thing I want to mention is
that the error started showing up two days
>>>>>>>>>>>>> ago, on Oct 22, while the expiration
is today, Oct 24.
>>>>>>>>>>>>>
>>>>>>>>>>>> It shouldn't take more than a few
minutes to roll back time, restart
>>>>>>>>>>>> services and see what happens. I think
your NTP clients will be able to
>>>>>>>>>>>> recover ok if the server is not available
for a few minutes.
>>>>>>>>>>>>
>>>>>>>>>>>> certmonger logs to syslog so you probably
want to look at that to see if
>>>>>>>>>>>> you can find a reason the certs
weren't renewed automatically.
>>>>>>>>>>>>
>>>>>>>>>>> No, that didn't help.
>>>>>>>>>>> And in the syslog there was nothing more than
this. (I had to stop the
>>>>>>>>>>> nameserver because it was spitting out lots
of messages.)
>>>>>>>>>>>
>>>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has
been changed
>>>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time
has been changed
>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping
Certificate monitoring and PKI enrollment...
>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped
Certificate monitoring and PKI enrollment.
>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting
Certificate monitoring and PKI enrollment...
>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started
Certificate monitoring and PKI enrollment.
>>>>>>>>>>> Oct 11 06:00:05 ipasrv certmonger[131018]:
2018-10-11 06:00:05 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profile
>>>>>>>>>>> Review: Problem with the SSL CA cert (path?
access rights?).
>>>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>>>> Oct 11 06:00:07 ipasrv certmonger[131018]:
2018-10-11 06:00:07 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>>>> Oct 11 06:00:17 ipasrv certmonger[131018]:
2018-10-11 06:00:17 [131018] Error 77 connecting to
https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access
rights?).
>>>>>>>>>>>
>>>>>>>>>> Ok, I think I know what is going on. This is
Ubuntu which AFAIK still
>>>>>>>>>> lacks nss-pem. That is probably why it can't
connect to renew the certs.
>>>>>>>>>>
>>>>>>>>>> I don't know if there is a workaround. Timo,
do you know?
>>>>>>>>> Ubuntu 18.04 and up have libnsspem, and certmonger
depends on it. I've
>>>>>>>>> never tested cert renewal though.
>>>>>>>>>
>>>>>>>> Does that mean, I'm screwed? What options do I have?
>>>>>>>> Live with it?
>>>>>>>> Migrate to, say Centos?
>>>>>>>> Try to upgrade the server to Ubuntu 18.04 (with
uncertainty whether it will work)?
>>>>>>>> Something else?
>>>>>>> Stock 18.04 has other issues, there's an updated version
on
>>>>>>> ppa:freeipa/staging which is backported from 18.10 and should
be fine
>>>>>>> and hopefully provided as a stable update on 18.04 later on.
>>>>>>>
>>>>>>> But you could try pulling libnsspem from 18.04, and *then*
roll back time?
>>>>>>>
>>>>>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
>>>>>>
>>>>>> Then I stopped ntp (and bind).
>>>>>> Set the time back to Oct 11
>>>>>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd,
certmonger
>>>>>> (in that order).
>>>>>>
>>>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit:
Forwarding request to dogtag-ipa-renew-agent
>>>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit:
dogtag-ipa-renew-agent returned 3
>>>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03
[168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview:
Peer certificate cannot be authenticated with given CA certificates.
>>>>>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12
[168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview:
Peer certificate cannot be authenticated with given CA certificates.
>>>>>>
>>>>>> :-(
>>>>>>
>>>>>> Rob said also to restart CA.
>>>>>> "restart krb5kdc, dirsrv, httpd and the CA then
certmonger"
>>>>>> I don't know which service that is. Does that matter?
>>>>> systemctl restart ipa?
>>>>>
>>>>>
>>>> I'm a bit scared to restart service ipa, because it also restarts
several other services,
>>>> link bind, and perhaps ntp. The latter is the one that I want to be
absolutely in control
>>>> of not starting.
>>> And you're right! The CA is pki-tomcatd, so you already restarted it.
>>>
>>>> It's getting too late now, time for weekend. I'll give it another
try on Monday.
>>>> Meanwhile I want to point at the changed message. In case that rings a
bell for
>>>> someone.
>>>>
>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327]
Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer
certificate cannot be authenticated with given CA certificates.
>>>>
>>> You can have a look at Rob's blog for additional items to check:
>>>
https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-auth...
>> Thanks, I just stumbled on it myself. Interesting read, although I don't
quite
>> understand all details.
>>
>> I really need some guidance what to do next. I tried the date trick, I installed
>> libnsspem (from Ubunu 18.04). The certmonger error message changed from
>> Error 77 into Error 60, but the problem remained.
>>
>> Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
>>
>> Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase
backgroundProcess
>> WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d
background process
>> java.lang.NullPointerException
>> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113)
>> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> I could do the date trick again, but then the question is, why didn't it work
last time?
> I tried it once more to set the date back and to restart the renewal process.
> I keep getting Error 60 with
> Peer certificate cannot be authenticated with given CA certificates.
>
> Also, I had a look at Rob's blog. But I'm lost at what to do with
> "... the fix was to reset the NSS trust flags in the Apache NSS
database"
> The curl command (with the date set back) seems to connect but it gives a
> gnutls_handshake() failed: Illegal parameter
>
> And then I don't know what it looks like if
> " You should get client certificate not found."
> So I didn't try the certutil -M commands.
>
> BTW. While setting the date back I have bind (DNS) switched off because it
> gave a lot of messages in my /var/log/syslog.
>
Right, this is for a previous version of IPA on RHEL/Fedora. The RA
agent cert used to be stored in the Apache NSS database.
That's a problem in
general with notes and blogs. When you write
things you don't know what to write about possible validness in
the future.
curl linked aganst gnu_tls? I have no idea, but it isn't this.
This error message originates in curl.
What is the validity of your CA certificate itself? Is it still valid
back in time?
Yes it is. Again, this is Ubuntu 16.04, I'm looking at ...
certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca'
certutil -L -d /etc/apache2/nssdb/ -n 'GHS.NL IPA CA'
certutil -L -d /etc/dirsrv/slapd-GHS-NL -n 'GHS.NL IPA CA'
Validity:
Not Before: Thu Nov 03 09:45:18 2016
Not After : Mon Nov 03 09:45:18 2036
--
Kees