4:59 a.m.
Hello,
I'm running a freeipa server over a cloudera cluster, on 2020-12-31 all the certs
expired and did not renew by itself.
After I set the system date before the expiration date, I tried ipa-cacert-renew but
returns an error saying that ca cert are not managed by certmonger so I did a getcert
resubmit for every cert.
Almos all went on "Monitoring" state, except for one that says
"NEED_CSR_GEN_PIN".
If I try to do 'ipactl start', it starts to first upgrade IPA and fails because of
the pki-tomcat service:
```
2019-12-31T19:12:01Z DEBUG response body '<html><head><title>Apache
Tomcat/7.0.76 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color
: black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem
unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b> Exception
report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server
encountered an internal error that prevented it from fulfilling this requ
est.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea
d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76
logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
2019-12-31T19:12:01Z DEBUG The CA status is: check interrupted due to error: Retrieving CA
status failed with status 500
2019-12-31T19:12:01Z DEBUG Waiting for CA to start...
```
I also looked for the previous threads listed on this forum, but none of them provided a
solution
6:53 a.m.
On 2/8/21 11:59 AM, Manuel Gugliucci via FreeIPA-users wrote:
Hello,
I'm running a freeipa server over a cloudera cluster, on 2020-12-31 all the certs
expired and did not renew by itself.
After I set the system date before the expiration date, I tried ipa-cacert-renew but
returns an error saying that ca cert are not managed by certmonger so I did a getcert
resubmit for every cert.
Hi,
the command "ipa-cacert-manage renew" is used to renew the CA
certificate (in case IPA was installed with an embedded CA), not the
other certs.
Before giving any advice, I would like to know more about the
deployment. Are there a single or multiple IPA servers? Is the CA role
deployed on multiple servers? Which version is installed?
# kinit admin; ipa server-role-find
# rpm -qa *ipa-server
Which certificates are valid or expired?
# getcert list
Thanks,
flo
Almos all went on "Monitoring" state, except for one that says
"NEED_CSR_GEN_PIN".
If I try to do 'ipactl start', it starts to first upgrade IPA and fails because
of the pki-tomcat service:
```
2019-12-31T19:12:01Z DEBUG response body '<html><head><title>Apache
Tomcat/7.0.76 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color
: black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem
unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b> Exception
report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server
encountered an internal error that prevented it from fulfilling this requ
est.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea
d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76
logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
2019-12-31T19:12:01Z DEBUG The CA status is: check interrupted due to error: Retrieving
CA status failed with status 500
2019-12-31T19:12:01Z DEBUG Waiting for CA to start...
```
I also looked for the previous threads listed on this forum, but none of them provided a
solution
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
7:03 a.m.
Hi Florence, thanks for the answer
it's a single IPA server, VERSION: 4.6.8, API_VERSION: 2.237
I kinit as admin without problems, then:
[root@ipa1 ~]# ipa server-role-find
ipa: ERROR: cannot connect to 'https://ipa1.itec.lab/ipa/json': Internal Server
Error
[root@ipa1 ~]# rpm -qa *ipa-server
ipa-server-4.6.8-5.el7.centos.x86_64
# getcert list
Number of certificates and requests being tracked: 7.
Request ID '20191231201955':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=ipa1.itec.lab,O=ITEC.LAB
subject: CN=ipa1.itec.lab,O=ITEC.LAB
expires: 2020-12-31 20:19:55 UTC
principal name: krbtgt/ITEC.LAB(a)ITEC.LAB
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20201102185036':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=CA Audit,O=ITEC.LAB
expires: 2020-12-08 09:35:14 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201102185037':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=OCSP Subsystem,O=ITEC.LAB
expires: 2020-12-08 09:38:07 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201102185038':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=CA Subsystem,O=ITEC.LAB
expires: 2020-12-08 09:37:36 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201102185039':
status: NEED_CSR_GEN_PIN
stuck: yes
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent-selfsigned
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=Certificate Authority,O=ITEC.LAB
expires: 2037-01-25 14:22:25 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201102185040':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=IPA RA,O=ITEC.LAB
expires: 2020-12-08 09:37:47 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20201102185042':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=ipa1.itec.lab,O=ITEC.LAB
expires: 2020-12-08 09:35:05 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert
cert-pki-ca"
track: yes
auto-renew: yes
I had to set my date in several weeks before the expiring to renew them via certmonger,
but it does not auto-renew past 30-12-2020
Thanks for the support,
Manuel
7:47 a.m.
On 2/8/21 2:03 PM, Manuel Gujo via FreeIPA-users wrote:
Hi Florence, thanks for the answer
it's a single IPA server, VERSION: 4.6.8, API_VERSION: 2.237
Hi,
The CA is self-signed and still valid, and you are lucky because this
ipa version already provides a new tool called ipa-cert-fix that should
be able to help renew the certificates.
For more information please refer to the doc [1]. ipa-cert-fix analyzes
the existing certificates and lists the ones that need to be renewed,
then prompts you for confirmation and proceeds. No need to move the date
in the past or do manual steps.
HTH,
flo
[1]
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/...
I kinit as admin without problems, then:
[root@ipa1 ~]# ipa server-role-find
ipa: ERROR: cannot connect to 'https://ipa1.itec.lab/ipa/json': Internal Server
Error
[root@ipa1 ~]# rpm -qa *ipa-server
ipa-server-4.6.8-5.el7.centos.x86_64
# getcert list
Number of certificates and requests being tracked: 7.
Request ID '20191231201955':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=ipa1.itec.lab,O=ITEC.LAB
subject: CN=ipa1.itec.lab,O=ITEC.LAB
expires: 2020-12-31 20:19:55 UTC
principal name: krbtgt/ITEC.LAB(a)ITEC.LAB
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20201102185036':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=CA Audit,O=ITEC.LAB
expires: 2020-12-08 09:35:14 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201102185037':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=OCSP Subsystem,O=ITEC.LAB
expires: 2020-12-08 09:38:07 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201102185038':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=CA Subsystem,O=ITEC.LAB
expires: 2020-12-08 09:37:36 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201102185039':
status: NEED_CSR_GEN_PIN
stuck: yes
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent-selfsigned
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=Certificate Authority,O=ITEC.LAB
expires: 2037-01-25 14:22:25 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201102185040':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=IPA RA,O=ITEC.LAB
expires: 2020-12-08 09:37:47 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20201102185042':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=ipa1.itec.lab,O=ITEC.LAB
expires: 2020-12-08 09:35:05 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert
cert-pki-ca"
track: yes
auto-renew: yes
I had to set my date in several weeks before the expiring to renew them via certmonger,
but it does not auto-renew past 30-12-2020
Thanks for the support,
Manuel
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
7:56 a.m.
Hi,
I re-sync the date to today and ran ipa-cert-fix but it returns an error
[root@ipa1 ~]# ipa-cert-fix
WARNING
ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA. It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.
The following certificates will be renewed:
Dogtag sslserver certificate:
Subject: CN=ipa1.itec.lab,O=ITEC.LAB
Serial: 17
Expires: 2020-12-08 09:35:05
Dogtag subsystem certificate:
Subject: CN=CA Subsystem,O=ITEC.LAB
Serial: 19
Expires: 2020-12-08 09:37:36
Dogtag ca_ocsp_signing certificate:
Subject: CN=OCSP Subsystem,O=ITEC.LAB
Serial: 21
Expires: 2020-12-08 09:38:07
Dogtag ca_audit_signing certificate:
Subject: CN=CA Audit,O=ITEC.LAB
Serial: 18
Expires: 2020-12-08 09:35:14
IPA IPA RA certificate:
Subject: CN=IPA RA,O=ITEC.LAB
Serial: 20
Expires: 2020-12-08 09:37:47
IPA Apache HTTPS certificate:
Subject: CN=ipa1.itec.lab,O=ITEC.LAB
Serial: 24
Expires: 2020-12-30 09:35:04
IPA LDAP certificate:
Subject: CN=ipa1.itec.lab,O=ITEC.LAB
Serial: 25
Expires: 2020-12-30 09:35:16
IPA KDC certificate:
Subject: CN=ipa1.itec.lab,O=ITEC.LAB
Serial: 1
Expires: 2020-12-31 20:19:55
Enter "yes" to proceed: yes
Proceeding.
[Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.crt'
The ipa-cert-fix command failed.
2:31 a.m.
On 2/8/21 2:56 PM, Manuel Gujo via FreeIPA-users wrote:
Hi,
I re-sync the date to today and ran ipa-cert-fix but it returns an error
[root@ipa1 ~]# ipa-cert-fix
WARNING
ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA. It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.
The following certificates will be renewed:
Dogtag sslserver certificate:
Subject: CN=ipa1.itec.lab,O=ITEC.LAB
Serial: 17
Expires: 2020-12-08 09:35:05
Dogtag subsystem certificate:
Subject: CN=CA Subsystem,O=ITEC.LAB
Serial: 19
Expires: 2020-12-08 09:37:36
Dogtag ca_ocsp_signing certificate:
Subject: CN=OCSP Subsystem,O=ITEC.LAB
Serial: 21
Expires: 2020-12-08 09:38:07
Dogtag ca_audit_signing certificate:
Subject: CN=CA Audit,O=ITEC.LAB
Serial: 18
Expires: 2020-12-08 09:35:14
IPA IPA RA certificate:
Subject: CN=IPA RA,O=ITEC.LAB
Serial: 20
Expires: 2020-12-08 09:37:47
IPA Apache HTTPS certificate:
Subject: CN=ipa1.itec.lab,O=ITEC.LAB
Serial: 24
Expires: 2020-12-30 09:35:04
IPA LDAP certificate:
Subject: CN=ipa1.itec.lab,O=ITEC.LAB
Serial: 25
Expires: 2020-12-30 09:35:16
IPA KDC certificate:
Subject: CN=ipa1.itec.lab,O=ITEC.LAB
Serial: 1
Expires: 2020-12-31 20:19:55
Enter "yes" to proceed: yes
Proceeding.
[Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.crt'
The ipa-cert-fix command failed.
Hi,
which version of pki-server is installed? You may be hitting
https://bugzilla.redhat.com/show_bug.cgi?id=1897120
Looks like you will need to manually fix the renewal issue by following
the old good method with changing date etc...
The first expiration date is 2020-12-08, the system date needs to be
moved before that date. Please try and let me know if there are issues.
flo
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
3:06 a.m.
Hi,
pki-server version is 10.5.17, if I hit pki-server-upgrade it says this:
# pki-server-upgrade
Upgrading from version 10.5.17 to 10.5.18:
1. Fix EC admin certificate profile (Yes/No) [Y]:
could be helpful?
I also found this: https://www.dogtagpki.org/wiki/Tomcat_SSL_Configuration_with_OpenSSL
could generating a self signed sslserver.crt a good idea?
in the bugzilla link you sent, do I have to download and run that pyscript?
About the old method with turn the date before expiration, what I have to do precisely?
Because I already done this:
#getcert list
//list all the cert IDs
#ipa-getcert resubmit -i <ID>
//for every cert listed before
#systemctl certmonger restart
Thanks for the support Flo
Manuel
3:05 a.m.
Hi,
I've retried to move date three weeks before 2020-12-08 and renew cert manually
# ipa-getcert resubmit -i "ID"
Resubmitting "20201102185036" to "dogtag-ipa-ca-renew-agent".
Here's one of the output log from journalctl -xe
# journalctl -xe
nov 17 18:08:27 ipa1.itec.lab certmonger[27108]: 2020-11-17 18:08:27 [27108] Internal
error
nov 17 18:08:29 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[28053]: Traceback (most
recent call last):
File
"/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, in
<module>
sys.exit(main())
File
"/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 507, in main
kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)
File
"/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in
kinit_keytab
cred =
gssapi.Credentials(name=name, store=store, usage='initiate')
File
"/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__
store=store)
File
"/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire
usage)
File
"ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from
(gssapi/raw/ext_cred
GSSError: Major
(851968): Unspecified GSS failure. Minor code may provide more information, Minor
(252963
now all the certs (except from kerberos and CA ones) are status: CA_UNREACHABLE.
CA cert is status: NEED_CSR_GEN_PIN
9:37 a.m.
Manuel Gujo via FreeIPA-users wrote:
Hi,
I've retried to move date three weeks before 2020-12-08 and renew cert manually
# ipa-getcert resubmit -i "ID"
Resubmitting "20201102185036" to "dogtag-ipa-ca-renew-agent".
Here's one of the output log from journalctl -xe
# journalctl -xe
nov 17 18:08:27 ipa1.itec.lab certmonger[27108]: 2020-11-17 18:08:27 [27108] Internal
error
nov 17 18:08:29 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[28053]: Traceback (most
recent call last):
File
"/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, in
<module>
sys.exit(main())
File
"/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 507, in main
kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)
File
"/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in
kinit_keytab
cred =
gssapi.Credentials(name=name, store=store, usage='initiate')
File
"/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__
store=store)
File
"/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire
usage)
File
"ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from
(gssapi/raw/ext_cred
GSSError: Major
(851968): Unspecified GSS failure. Minor code may provide more information, Minor
(252963
now all the certs (except from kerberos and CA ones) are status: CA_UNREACHABLE.
CA cert is status: NEED_CSR_GEN_PIN
When you are moving back in time are you bringing the IPA services back
up? You need to do this manually if you have an NTP server enabled
(which it is by default).
Minimum you need to restart, in order, dirsrv.target, krb5kdc, named,
httpd, pki-tomcatd. If you restart certmonger it may kick off the
renewals for you (or it might not).
If you can get the services running back in time then runipa config-show
to determine whether this server is configured as the CA renewal server.
Only one in the cluster will have this role and the renewals need to
take place on that server. If this one isn't it and none of the others
report it then you can run: ipa config-mod --ca-renewal-master-server=<fqdn>
As Flo said the NEED_CSR_GEN_PIN is from your using ipa-cacert-manage.
It doesn't affect anything in the short term.
rob
9:42 a.m.
Hi Rob,
do I have to stop all the IPA services before i move back the date? Now I'm only
moving back date and restarting certmonger.
pki-tomcatd is failed so i can't stop/restart it
10:04 a.m.
Manuel Gujo via FreeIPA-users wrote:
Hi Rob,
do I have to stop all the IPA services before i move back the date? Now I'm only
moving back date and restarting certmonger.
It wouldn't hurt.
You absolutely need to restart things in the past because they can't run
in current time with expired certs.
pki-tomcatd is failed so i can't stop/restart it
Then go back in the past and we can try to figure out why it won't start
then. It won't start now due to expired certs.
You can't renew the certs without a working CA.
rob
10:11 a.m.
I moved the date before the expiring and restarted the services one by one as you listed
(systemctl restart dirsrv@my-domain, systemctl restart krb5kdc etc.)
then:
[root@ipa1 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: STOPPED (if I do systemctl status named it says running)
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: STOPPED
ipa-dnskeysyncd Service: STOPPED
ipa: INFO: The ipactl command was successful
pki-tomcatd failed to start:
# systemctl restart pki-tomcatd@ITEC-LAB
Job for pki-tomcatd(a)ITEC-LAB.service failed because the control process exited with error
code. See "systemctl status pki-tomcatd(a)ITEC-LAB.service" and "journalctl
-xe" for details.
# journalctl -xe
nov 17 18:22:31 ipa1.itec.lab systemd[1]: Unit pki-tomcatd(a)ITEC-LAB.service entered failed
state.
nov 17 18:22:31 ipa1.itec.lab audispd[24456]: node=ipa1.itec.lab type=SERVICE_START
msg=audit(1605637351.916:7091): pid=1 uid=0 auid=4294967295 ses=4294967295
msg='unit=pki-tomc
nov 17 18:22:31 ipa1.itec.lab systemd[1]: pki-tomcatd(a)ITEC-LAB.service failed.
nov 17 18:22:31 ipa1.itec.lab polkitd[30556]: Unregistered Authentication Agent for
unix-process:17970:71502359 (system bus name :1.1719, object path
/org/freedesktop/PolicyKit1
10:39 a.m.
Manuel Gujo via FreeIPA-users wrote:
I moved the date before the expiring and restarted the services one
by one as you listed (systemctl restart dirsrv@my-domain, systemctl restart krb5kdc etc.)
then:
[root@ipa1 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: STOPPED (if I do systemctl status named it says running)
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: STOPPED
ipa-dnskeysyncd Service: STOPPED
ipa: INFO: The ipactl command was successful
pki-tomcatd failed to start:
# systemctl restart pki-tomcatd@ITEC-LAB
Job for pki-tomcatd(a)ITEC-LAB.service failed because the control process exited with error
code. See "systemctl status pki-tomcatd(a)ITEC-LAB.service" and "journalctl
-xe" for details.
# journalctl -xe
nov 17 18:22:31 ipa1.itec.lab systemd[1]: Unit pki-tomcatd(a)ITEC-LAB.service entered
failed state.
nov 17 18:22:31 ipa1.itec.lab audispd[24456]: node=ipa1.itec.lab type=SERVICE_START
msg=audit(1605637351.916:7091): pid=1 uid=0 auid=4294967295 ses=4294967295
msg='unit=pki-tomc
nov 17 18:22:31 ipa1.itec.lab systemd[1]: pki-tomcatd(a)ITEC-LAB.service failed.
nov 17 18:22:31 ipa1.itec.lab polkitd[30556]: Unregistered Authentication Agent for
unix-process:17970:71502359 (system bus name :1.1719, object path
/org/freedesktop/PolicyKit1
Look in /var/log/pki/pki-tomcat/ca/debug. Find where it tries to start
the service and go down from there. Looking at the end of the log is
almost always fruitless.
rob
12:07 p.m.
Here's what I found in /var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host ipa1.itec.lab port 636 Error
netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException:
Connection refused (Connection refused) (-1)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:667)
at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1056)
at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:962)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:568)
at com.netscape.certsrv.apps.CMS.init(CMS.java:191)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1458)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Internal Database Error encountered: Could not connect to LDAP server host ipa1.itec.lab
port 636 Error netscape.ldap.LDAPException: Unable to create socket:
java.net.ConnectException: Connection refused (Connection refused) (-1)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689)
at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1056)
at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:962)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:568)
at com.netscape.certsrv.apps.CMS.init(CMS.java:191)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1458)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[31/Dec/2019:19:55:52][localhost-startStop-1]: CMS.start(): shutdown server
[31/Dec/2019:19:55:52][localhost-startStop-1]: CMSEngine.shutdown()
12:23 p.m.
Manuel Gujo via FreeIPA-users wrote:
Here's what I found in /var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host ipa1.itec.lab port 636 Error
netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException:
Connection refused (Connection refused) (-1)
Double-check that dirsrv is started and listening on port 636. I assume
the host name is correct.
rob
2:33 a.m.
# systemctl status dirsrv@ITEC-LAB
● dirsrv(a)ITEC-LAB.service - 389 Directory Server ITEC-LAB.
Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset:
disabled)
Active: active (running) since mar 2020-11-17 18:00:26 UTC; 2 months 28 days ago
Main PID: 15817 (ns-slapd)
Status: "slapd started: Ready to process requests"
CGroup: /system.slice/system-dirsrv.slice/dirsrv(a)ITEC-LAB.service
└─15817 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ITEC-LAB -i
/var/run/dirsrv/slapd-ITEC-LAB.pid
feb 12 17:26:03 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3
# netstat -tulpn | grep LISTEN
tcp 0 0 0.0.0.0:749 0.0.0.0:* LISTEN
1222/kadmind
tcp 0 0 0.0.0.0:464 0.0.0.0:* LISTEN
1222/kadmind
tcp 0 0 192.168.20.3:53 0.0.0.0:* LISTEN
17818/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
17818/named
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 24121/sshd
tcp 0 0 0.0.0.0:88 0.0.0.0:* LISTEN
15869/krb5kdc
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN
17818/named
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
24232/master
tcp6 0 0 :::749 :::* LISTEN
1222/kadmind
tcp6 0 0 :::80 :::* LISTEN
16122/httpd
tcp6 0 0 :::464 :::* LISTEN
1222/kadmind
tcp6 0 0 :::53 :::* LISTEN
17818/named
tcp6 0 0 :::22 :::* LISTEN 24121/sshd
tcp6 0 0 :::88 :::* LISTEN
15869/krb5kdc
tcp6 0 0 ::1:953 :::* LISTEN
17818/named
tcp6 0 0 ::1:25 :::* LISTEN
24232/master
tcp6 0 0 :::443 :::* LISTEN
16122/httpd
dirsrv is started but I don't see port 636 on this list, how can I open it for dirsrv?
8:30 a.m.
Manuel Gujo via FreeIPA-users wrote:
# systemctl status dirsrv@ITEC-LAB
● dirsrv(a)ITEC-LAB.service - 389 Directory Server ITEC-LAB.
Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset:
disabled)
Active: active (running) since mar 2020-11-17 18:00:26 UTC; 2 months 28 days ago
Main PID: 15817 (ns-slapd)
Status: "slapd started: Ready to process requests"
CGroup: /system.slice/system-dirsrv.slice/dirsrv(a)ITEC-LAB.service
└─15817 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ITEC-LAB -i
/var/run/dirsrv/slapd-ITEC-LAB.pid
feb 12 17:26:03 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2
feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3
# netstat -tulpn | grep LISTEN
tcp 0 0 0.0.0.0:749 0.0.0.0:* LISTEN
1222/kadmind
tcp 0 0 0.0.0.0:464 0.0.0.0:* LISTEN
1222/kadmind
tcp 0 0 192.168.20.3:53 0.0.0.0:* LISTEN
17818/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
17818/named
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
24121/sshd
tcp 0 0 0.0.0.0:88 0.0.0.0:* LISTEN
15869/krb5kdc
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN
17818/named
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
24232/master
tcp6 0 0 :::749 :::* LISTEN
1222/kadmind
tcp6 0 0 :::80 :::* LISTEN
16122/httpd
tcp6 0 0 :::464 :::* LISTEN
1222/kadmind
tcp6 0 0 :::53 :::* LISTEN
17818/named
tcp6 0 0 :::22 :::* LISTEN
24121/sshd
tcp6 0 0 :::88 :::* LISTEN
15869/krb5kdc
tcp6 0 0 ::1:953 :::* LISTEN
17818/named
tcp6 0 0 ::1:25 :::* LISTEN
24232/master
tcp6 0 0 :::443 :::* LISTEN
16122/httpd
dirsrv is started but I don't see port 636 on this list, how can I open it for
dirsrv?
In fact it isn't listening on any port.
Stop dirsrv and look in /etc/dirsrv/slapd-ITEC-LAB/dse.ldif for
nsslapd-port which should be 389 and nsslapd-security which should be
on. If not then fix it and restart dirsrv. That should fix it.
rob
9:38 a.m.
Hi Rob,
so in "/etc/dirsrv/slapd-ITEC-LAB/dse.ldif", nsslapd-port was '0' and
nsslapd-security was off, I fixed it and now it's listening on port 389 and 636:
# netstat -tulpn | grep LISTEN | grep ns-slapd
tcp6 0 0 :::636 :::* LISTEN
30606/ns-slapd
tcp6 0 0 :::389 :::* LISTEN
30606/ns-slapd
Then I tried to restart all the ipactl services one by one. pki-tomcatd keeps failing and
/var/log/pki/pki-tomcat/ca/debug does not log anymore (last log is the one i sent you
above, 31 Dec 2019)
I resubmitted all the expired certs and restarting cermonger but certs keep being
unreachable.
from certmonger logs:
nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]: Forwarding request
to dogtag-ipa-renew-agent
nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: GET
http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit?profileId=caServerCert&a...
nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: (null)
nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]:
dogtag-ipa-renew-agent returned 3
nov 17 18:11:47 ipa1.itec.lab certmonger[30685]: 2020-11-17 18:11:47 [30685] Error 7
connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: Couldn't connect to
server.
in certmonger's log I also saw these:
nov 17 18:11:01 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30741]: Traceback (most
recent call last):
File
"/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, in
<module>
sys.exit(main())
File
"/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 495, in main
api.finalize()
File
"/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 740, in finalize
self.__do_if_not_done('load_plugins')
File
"/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 431, in
__do_if_not_done
getattr(self,
name)()
File
"/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 620, in
load_plugins
self.add_package(package)
File
"/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 647, in add_package
module =
importlib.import_module(name)
File
"/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File
"/usr/lib/python2.7/site-packages/ipaserver/plugins/server.py", line 32, in
<module>
from
ipaserver.install import bindinstance, dnskeysyncinstance
File
"/usr/lib/python2.7/site-packages/ipaserver/install/dnskeysyncinstance.py", line
17, in <module>
from ipaserver
import p11helper as _ipap11helper
File
"/usr/lib/python2.7/site-packages/ipaserver/p11helper.py", line 342, in
<module>
_libp11_kit =
_ffi.dlopen(ctypes.util.find_library('p11-kit'))
File
"/usr/lib64/python2.7/ctypes/util.py", line 244, in find_library
return
_findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))
File
"/usr/lib64/python2.7/ctypes/util.py", line 233, in _findSoname_ldconfig
f =
os.popen('/sbin/ldconfig -p 2>/dev/null')
OSError: [Errno 12]
Cannot allocate memory
2:02 p.m.
Manuel Gujo via FreeIPA-users wrote:
Hi Rob,
so in "/etc/dirsrv/slapd-ITEC-LAB/dse.ldif", nsslapd-port was '0' and
nsslapd-security was off, I fixed it and now it's listening on port 389 and 636:
# netstat -tulpn | grep LISTEN | grep ns-slapd
tcp6 0 0 :::636 :::* LISTEN
30606/ns-slapd
tcp6 0 0 :::389 :::* LISTEN
30606/ns-slapd
Then I tried to restart all the ipactl services one by one. pki-tomcatd keeps failing and
/var/log/pki/pki-tomcat/ca/debug does not log anymore (last log is the one i sent you
above, 31 Dec 2019)
I resubmitted all the expired certs and restarting cermonger but certs keep being
unreachable.
If the CA isn't running then there is no point in resubmitting the
certmonger requests. It is guaranteed to fail with UNREACHABLE.
Check the journalctl output and the other logs, like catalina, in
/var/log/pki/pki-tomcat for more information on why it failed to start.
from certmonger logs:
nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]: Forwarding request
to dogtag-ipa-renew-agent
nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: GET
http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit?profileId=caServerCert&a...
nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: (null)
nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]:
dogtag-ipa-renew-agent returned 3
nov 17 18:11:47 ipa1.itec.lab certmonger[30685]: 2020-11-17 18:11:47 [30685] Error 7
connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: Couldn't connect to
server.
in certmonger's log I also saw these:
nov 17 18:11:01 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30741]: Traceback (most
recent call last):
File
"/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, in
<module>
sys.exit(main())
File
"/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 495, in main
api.finalize()
File
"/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 740, in finalize
self.__do_if_not_done('load_plugins')
File
"/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 431, in
__do_if_not_done
getattr(self,
name)()
File
"/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 620, in
load_plugins
self.add_package(package)
File
"/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 647, in add_package
module =
importlib.import_module(name)
File
"/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File
"/usr/lib/python2.7/site-packages/ipaserver/plugins/server.py", line 32, in
<module>
from
ipaserver.install import bindinstance, dnskeysyncinstance
File
"/usr/lib/python2.7/site-packages/ipaserver/install/dnskeysyncinstance.py", line
17, in <module>
from ipaserver
import p11helper as _ipap11helper
File
"/usr/lib/python2.7/site-packages/ipaserver/p11helper.py", line 342, in
<module>
_libp11_kit =
_ffi.dlopen(ctypes.util.find_library('p11-kit'))
File
"/usr/lib64/python2.7/ctypes/util.py", line 244, in find_library
return
_findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))
File
"/usr/lib64/python2.7/ctypes/util.py", line 233, in _findSoname_ldconfig
f =
os.popen('/sbin/ldconfig -p 2>/dev/null')
OSError: [Errno
12] Cannot allocate memory
Is this host memory-constrained? How much RAM does it have?
rob
2:58 a.m.
Manuel Gujo via FreeIPA-users wrote:
If the CA isn't running then there is no point in resubmitting the
certmonger requests. It is guaranteed to fail with UNREACHABLE.
Check the journalctl output and the other logs, like catalina, in
/var/log/pki/pki-tomcat for more information on why it failed to start.
Is this host memory-constrained? How much RAM does it have?
rob
there's new log on debug. Catalina does not log anything (0kb per file).
in debug:
Could not connect to LDAP server host ipa1.itec.lab port 636 Error
netscape.ldap.LDAPException: Unable to create socket:
org.mozilla.jss.ssl.SSLSocketException: org.mozilla.jss.ssl.SSLSocketExc
eption: SSL_ForceHandshake failed: (-8181) Peer's Certificate has expired. (-1)
in "system" logs says the same thing of debugs'
When I try to run 'ipactl start' without -f option, it says this:
# ipactl start
IPA version error: data needs to be upgraded (expected version
'4.6.8-5.el7.centos', current version '4.4.0-14.el7.centos.4')
then after a while it fails and in /var/log/ipaupgrade.log says:
2020-11-17T18:25:05Z DEBUG httplib request failed:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 220, in
_httplib_request
conn.request(method, path, body=request_body, headers=headers)
File "/usr/lib64/python2.7/httplib.py", line 1056, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 1090, in _send_request
self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 1052, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 890, in _send_output
self.send(msg)
File "/usr/lib64/python2.7/httplib.py", line 852, in send
self.connect()
File "/usr/lib64/python2.7/httplib.py", line 1266, in connect
HTTPConnection.connect(self)
File "/usr/lib64/python2.7/httplib.py", line 833, in connect
self.timeout, self.source_address)
File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 111] Connection refused
2020-11-17T18:25:05Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
run command ipa-server-upgrade manually.
2020-11-17T18:25:05Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line
54, in run
server.upgrade()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2176, in upgrade
upgrade_configuration()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2059, in upgrade_configuration
cainstance.repair_profile_caIPAserviceCert()
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
1949, in repair_profile_caIPAserviceCert
with api.Backend.ra_certprofile as profile_api:
File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line
1311, in __enter__
method='GET'
File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 167, in
https_request
method=method, headers=headers)
File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 229, in
_httplib_request
raise NetworkError(uri=uri, error=str(e))
2020-11-17T18:25:05Z DEBUG The ipa-server-upgrade command failed, exception: NetworkError:
cannot connect to 'https://ipa1.itec.lab:8443/ca/rest/account/login': [Errno 111]
Connection refused
2020-11-17T18:25:05Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
After this run, I noticed that some of the certs went on Monitoring state
# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20191231201955':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=ipa1.itec.lab,O=ITEC.LAB
subject: CN=ipa1.itec.lab,O=ITEC.LAB
expires: 2022-02-08 15:59:12 UTC
principal name: krbtgt/ITEC.LAB(a)ITEC.LAB
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20201117182331':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit:
Couldn't connect to server.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=CA Audit,O=ITEC.LAB
expires: 2020-12-08 09:35:14 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201117182333':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit:
Couldn't connect to server.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=OCSP Subsystem,O=ITEC.LAB
expires: 2020-12-08 09:38:07 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201117182335':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=CA Subsystem,O=ITEC.LAB
expires: 2022-11-07 18:24:47 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201117182336':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=Certificate Authority,O=ITEC.LAB
expires: 2037-01-25 14:22:25 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201117182338':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit:
Couldn't connect to server.
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=IPA RA,O=ITEC.LAB
expires: 2020-12-08 09:37:47 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20201117182339':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=ipa1.itec.lab,O=ITEC.LAB
expires: 2022-11-07 18:24:56 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20201117182342':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-ITEC-LAB',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/dirsrv/slapd-ITEC-LAB/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-ITEC-LAB',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=ipa1.itec.lab,O=ITEC.LAB
expires: 2020-12-30 09:35:16 UTC
principal name: ldap/ipa1.itec.lab(a)ITEC.LAB
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv ITEC-LAB
track: yes
auto-renew: yes
Request ID '20201117182351':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=ITEC.LAB
subject: CN=ipa1.itec.lab,O=ITEC.LAB
expires: 2020-12-30 09:35:04 UTC
principal name: HTTP/ipa1.itec.lab(a)ITEC.LAB
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: STOPPED
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: STOPPED
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
but pki-tomcatd still fails if I try to restart it and in the debug logs:
[17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet:service() uri =
/ca/admin/ca/getStatus
[17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet: caGetStatus start to service.
[17/Nov/2020:18:32:34][http-bio-8080-exec-7]: Failed to read product version String.
java.io.FileNotFoundException: /usr/share/pki/CS_SERVER_VERSION (No such file or
directory)
[17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet: curDate=Tue Nov 17 18:32:34 UTC
2020 id=caGetStatus time=9
IPA VM has 2 CPU and 4GB of RAM, it never goes up to 90% of the usage
1180
days inactive
1188
days old
freeipa-users@lists.fedorahosted.org
19 comments
4 participants
participants (4)
-
Florence Blanc-Renaud -
Manuel Gugliucci -
Manuel Gujo
-
Rob Crittenden