Kees Bakker wrote:
> On 29-10-18 19:30, Rob Crittenden wrote:
>> Kees Bakker via FreeIPA-users wrote:
>>> On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
>>>> On 26-10-18 18:20, Florence Blanc-Renaud wrote:
>>>>> On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
>>>>>> On 26-10-18 18:00, Timo Aaltonen wrote:
>>>>>>> On 26.10.2018 18.59, Kees Bakker wrote:
>>>>>>>> On 26-10-18 14:55, Timo Aaltonen wrote:
>>>>>>>>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users
wrote:
>>>>>>>>>> On 25-10-18 20:46, Timo Aaltonen wrote:
>>>>>>>>>>> On 25.10.2018 21.44, Rob Crittenden wrote:
>>>>>>>>>>>> Kees Bakker wrote:
>>>>>>>>>>>>> On 25-10-18 16:11, Rob Crittenden
wrote:
>>>>>>>>>>>>>> Kees Bakker via FreeIPA-users
wrote:
>>>>>>>>>>>>>>> On 25-10-18 14:18, Rob
Crittenden wrote:
>>>>>>>>>>>>>>>> Kees Bakker via
FreeIPA-users wrote:
>>>>>>>>>>>>>>>>> Could it be that this
error already existed since we started? Notice
>>>>>>>>>>>>>>>>> the Request ID of
2016..., and the expires: 2018-10-24.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> # getcert list -n
ipaCert | sed blabla
>>>>>>>>>>>>>>>>> Number of
certificates and requests being tracked: 8.
>>>>>>>>>>>>>>>>> Request ID
'20161103094546':
>>>>>>>>>>>>>>>>> status:
CA_UNREACHABLE
>>>>>>>>>>>>>>>>> ca-error: Error
77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the
SSL CA cert (path? access rights?).
>>>>>>>>>>>>>>>>> stuck: no
>>>>>>>>>>>>>>>>> key pair
storage:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt'
>>>>>>>>>>>>>>>>> certificate:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB'
>>>>>>>>>>>>>>>>> CA:
dogtag-ipa-ca-renew-agent
>>>>>>>>>>>>>>>>> issuer:
CN=Certificate Authority,O=MYDOMAIN
>>>>>>>>>>>>>>>>> subject: CN=IPA
RA,O=MYDOMAIN
>>>>>>>>>>>>>>>>> expires:
2018-10-24 08:45:40 UTC
>>>>>>>>>>>>>>>>> key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>>>>>>>>>>>>>>>> eku:
id-kp-serverAuth,id-kp-clientAuth
>>>>>>>>>>>>>>>>> pre-save
command: /usr/lib/ipa/certmonger/renew_ra_cert_pre
>>>>>>>>>>>>>>>>> post-save
command: /usr/lib/ipa/certmonger/renew_ra_cert
>>>>>>>>>>>>>>>>> track: yes
>>>>>>>>>>>>>>>>> auto-renew: yes
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In other words, is
this the same issue as
https://pagure.io/freeipa/issue/7422 ?
>>>>>>>>>>>>>>>> The problem is your certs
expired yesterday so connections won't work
>>>>>>>>>>>>>>>> (the code and message
don't come from within certmonger).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> certmonger _should_ have
renewed them. Try killing ntpd, going back a
>>>>>>>>>>>>>>>> few days, restart
krb5kdc, dirsrv, httpd and the CA then certmonger and
>>>>>>>>>>>>>>>> see what happens.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Easy for you to say. You know
what you're doing :-)
>>>>>>>>>>>>>>> For me it's all magic.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Anyway, I'll try it.
I'm just scared to set the clock back, because there may
>>>>>>>>>>>>>>> be clients in the network
that use this server as a NTP server.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Another thing I want to
mention is that the error started showing up two days
>>>>>>>>>>>>>>> ago, on Oct 22, while the
expiration is today, Oct 24.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It shouldn't take more than a
few minutes to roll back time, restart
>>>>>>>>>>>>>> services and see what happens. I
think your NTP clients will be able to
>>>>>>>>>>>>>> recover ok if the server is not
available for a few minutes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> certmonger logs to syslog so you
probably want to look at that to see if
>>>>>>>>>>>>>> you can find a reason the certs
weren't renewed automatically.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> No, that didn't help.
>>>>>>>>>>>>> And in the syslog there was nothing
more than this. (I had to stop the
>>>>>>>>>>>>> nameserver because it was spitting
out lots of messages.)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[1]:
Time has been changed
>>>>>>>>>>>>> Oct 11 06:00:00 ipasrv
systemd[52167]: Time has been changed
>>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]:
Stopping Certificate monitoring and PKI enrollment...
>>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]:
Stopped Certificate monitoring and PKI enrollment.
>>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]:
Starting Certificate monitoring and PKI enrollment...
>>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]:
Started Certificate monitoring and PKI enrollment.
>>>>>>>>>>>>> Oct 11 06:00:05 ipasrv
certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profile
>>>>>>>>>>>>> Review: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>>>>>> Oct 11 06:00:07 ipasrv
certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>>>>>> Oct 11 06:00:17 ipasrv
certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to
https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access
rights?).
>>>>>>>>>>>>>
>>>>>>>>>>>> Ok, I think I know what is going on. This
is Ubuntu which AFAIK still
>>>>>>>>>>>> lacks nss-pem. That is probably why it
can't connect to renew the certs.
>>>>>>>>>>>>
>>>>>>>>>>>> I don't know if there is a
workaround. Timo, do you know?
>>>>>>>>>>> Ubuntu 18.04 and up have libnsspem, and
certmonger depends on it. I've
>>>>>>>>>>> never tested cert renewal though.
>>>>>>>>>>>
>>>>>>>>>> Does that mean, I'm screwed? What options do
I have?
>>>>>>>>>> Live with it?
>>>>>>>>>> Migrate to, say Centos?
>>>>>>>>>> Try to upgrade the server to Ubuntu 18.04 (with
uncertainty whether it will work)?
>>>>>>>>>> Something else?
>>>>>>>>> Stock 18.04 has other issues, there's an updated
version on
>>>>>>>>> ppa:freeipa/staging which is backported from 18.10
and should be fine
>>>>>>>>> and hopefully provided as a stable update on 18.04
later on.
>>>>>>>>>
>>>>>>>>> But you could try pulling libnsspem from 18.04, and
*then* roll back time?
>>>>>>>>>
>>>>>>>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
>>>>>>>>
>>>>>>>> Then I stopped ntp (and bind).
>>>>>>>> Set the time back to Oct 11
>>>>>>>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2,
pki-tomcatd, certmonger
>>>>>>>> (in that order).
>>>>>>>>
>>>>>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit:
Forwarding request to dogtag-ipa-renew-agent
>>>>>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit:
dogtag-ipa-renew-agent returned 3
>>>>>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11
06:08:03 [168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be
authenticated with given CA certificates.
>>>>>>>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11
06:08:12 [168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be
authenticated with given CA certificates.
>>>>>>>>
>>>>>>>> :-(
>>>>>>>>
>>>>>>>> Rob said also to restart CA.
>>>>>>>> "restart krb5kdc, dirsrv, httpd and the CA then
certmonger"
>>>>>>>> I don't know which service that is. Does that
matter?
>>>>>>> systemctl restart ipa?
>>>>>>>
>>>>>>>
>>>>>> I'm a bit scared to restart service ipa, because it also
restarts several other services,
>>>>>> link bind, and perhaps ntp. The latter is the one that I want to
be absolutely in control
>>>>>> of not starting.
>>>>> And you're right! The CA is pki-tomcatd, so you already restarted
it.
>>>>>
>>>>>> It's getting too late now, time for weekend. I'll give it
another try on Monday.
>>>>>> Meanwhile I want to point at the changed message. In case that
rings a bell for
>>>>>> someone.
>>>>>>
>>>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03
[168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview:
Peer certificate cannot be authenticated with given CA certificates.
>>>>>>
>>>>> You can have a look at Rob's blog for additional items to check:
>>>>>
https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-auth...
>>>> Thanks, I just stumbled on it myself. Interesting read, although I
don't quite
>>>> understand all details.
>>>>
>>>> I really need some guidance what to do next. I tried the date trick, I
installed
>>>> libnsspem (from Ubunu 18.04). The certmonger error message changed from
>>>> Error 77 into Error 60, but the problem remained.
>>>>
>>>> Futhermore I noticed that pki-tomcat spits out a warning every 10
seconds
>>>>
>>>> Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase
backgroundProcess
>>>> WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@5417a64d background process
>>>> java.lang.NullPointerException
>>>> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113)
>>>> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
>>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
>>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
>>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
>>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>>
>>>> I could do the date trick again, but then the question is, why didn't
it work last time?
>>> I tried it once more to set the date back and to restart the renewal
process.
>>> I keep getting Error 60 with
>>> Peer certificate cannot be authenticated with given CA certificates.
>>>
>>> Also, I had a look at Rob's blog. But I'm lost at what to do with
>>> "... the fix was to reset the NSS trust flags in the Apache NSS
database"
>>> The curl command (with the date set back) seems to connect but it gives a
>>> gnutls_handshake() failed: Illegal parameter
>>>
>>> And then I don't know what it looks like if
>>> " You should get client certificate not found."
>>> So I didn't try the certutil -M commands.
>>>
>>> BTW. While setting the date back I have bind (DNS) switched off because it
>>> gave a lot of messages in my /var/log/syslog.
>>>
>> Right, this is for a previous version of IPA on RHEL/Fedora. The RA
>> agent cert used to be stored in the Apache NSS database.
> That's a problem in general with notes and blogs. When you write
> things you don't know what to write about possible validness in
> the future.
>
>> curl linked aganst gnu_tls? I have no idea, but it isn't this.
>>
>> This error message originates in curl.
>>
>> What is the validity of your CA certificate itself? Is it still valid
>> back in time?
> Yes it is. Again, this is Ubuntu 16.04, I'm looking at ...
>
> certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca'
> certutil -L -d /etc/apache2/nssdb/ -n 'GHS.NL IPA CA'
> certutil -L -d /etc/dirsrv/slapd-GHS-NL -n 'GHS.NL IPA CA'
>
> Validity:
> Not Before: Thu Nov 03 09:45:18 2016
> Not After : Mon Nov 03 09:45:18 2036
>
Ok I'm just flying blind right now, reading curl code, but in the gnutls
code it throws this error if the server cert validation fails.
I don't know where gnutls reads its CA trust from or whether it honors
the system-wide trust, or even if your IPA CA is in the global trust
properly.
When back in time do IPA operations work? ipa user-show admin, ipa
cert-show 1, etc?
I'll do that maybe later this week. I don't want to mess with the IPA
server while it is still working (sort of).
Meanwhile, here is a new question, a different approach.
Can I manually generate a new certificate?
For example the "Server-Cert cert-pki-ca" of the pki-tomcat server.
And if so, do you have a suggestion about the commands to use?
--
Kees