Hello,
I'm running a freeipa server over a cloudera cluster, on 2020-12-31 all the certs expired and did not renew by itself.
After I set the system date before the expiration date, I tried ipa-cacert-renew but returns an error saying that ca cert are not managed by certmonger so I did a getcert resubmit for every cert.
Almos all went on "Monitoring" state, except for one that says "NEED_CSR_GEN_PIN".
If I try to do 'ipactl start', it starts to first upgrade IPA and fails because of the pki-tomcat service:
``` 2019-12-31T19:12:01Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this requ est.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-12-31T19:12:01Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-12-31T19:12:01Z DEBUG Waiting for CA to start... ``` I also looked for the previous threads listed on this forum, but none of them provided a solution
On 2/8/21 11:59 AM, Manuel Gugliucci via FreeIPA-users wrote:
Hello,
I'm running a freeipa server over a cloudera cluster, on 2020-12-31 all the certs expired and did not renew by itself.
After I set the system date before the expiration date, I tried ipa-cacert-renew but returns an error saying that ca cert are not managed by certmonger so I did a getcert resubmit for every cert.
Hi,
the command "ipa-cacert-manage renew" is used to renew the CA certificate (in case IPA was installed with an embedded CA), not the other certs.
Before giving any advice, I would like to know more about the deployment. Are there a single or multiple IPA servers? Is the CA role deployed on multiple servers? Which version is installed?
# kinit admin; ipa server-role-find # rpm -qa *ipa-server
Which certificates are valid or expired? # getcert list
Thanks, flo
Almos all went on "Monitoring" state, except for one that says "NEED_CSR_GEN_PIN".
If I try to do 'ipactl start', it starts to first upgrade IPA and fails because of the pki-tomcat service:
2019-12-31T19:12:01Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this requ est.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-12-31T19:12:01Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-12-31T19:12:01Z DEBUG Waiting for CA to start...I also looked for the previous threads listed on this forum, but none of them provided a solution _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Florence, thanks for the answer
it's a single IPA server, VERSION: 4.6.8, API_VERSION: 2.237
I kinit as admin without problems, then:
[root@ipa1 ~]# ipa server-role-find ipa: ERROR: cannot connect to 'https://ipa1.itec.lab/ipa/json': Internal Server Error [root@ipa1 ~]# rpm -qa *ipa-server ipa-server-4.6.8-5.el7.centos.x86_64
# getcert list Number of certificates and requests being tracked: 7. Request ID '20191231201955': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: SelfSign issuer: CN=ipa1.itec.lab,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2020-12-31 20:19:55 UTC principal name: krbtgt/ITEC.LAB@ITEC.LAB certificate template/profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20201102185036': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=CA Audit,O=ITEC.LAB expires: 2020-12-08 09:35:14 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185037': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=OCSP Subsystem,O=ITEC.LAB expires: 2020-12-08 09:38:07 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185038': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=CA Subsystem,O=ITEC.LAB expires: 2020-12-08 09:37:36 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185039': status: NEED_CSR_GEN_PIN stuck: yes key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent-selfsigned issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=Certificate Authority,O=ITEC.LAB expires: 2037-01-25 14:22:25 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185040': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=IPA RA,O=ITEC.LAB expires: 2020-12-08 09:37:47 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20201102185042': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2020-12-08 09:35:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes
I had to set my date in several weeks before the expiring to renew them via certmonger, but it does not auto-renew past 30-12-2020
Thanks for the support, Manuel
On 2/8/21 2:03 PM, Manuel Gujo via FreeIPA-users wrote:
Hi Florence, thanks for the answer
it's a single IPA server, VERSION: 4.6.8, API_VERSION: 2.237
Hi,
The CA is self-signed and still valid, and you are lucky because this ipa version already provides a new tool called ipa-cert-fix that should be able to help renew the certificates.
For more information please refer to the doc [1]. ipa-cert-fix analyzes the existing certificates and lists the ones that need to be renewed, then prompts you for confirmation and proceeds. No need to move the date in the past or do manual steps.
HTH, flo
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
I kinit as admin without problems, then:
[root@ipa1 ~]# ipa server-role-find ipa: ERROR: cannot connect to 'https://ipa1.itec.lab/ipa/json': Internal Server Error [root@ipa1 ~]# rpm -qa *ipa-server ipa-server-4.6.8-5.el7.centos.x86_64
# getcert list Number of certificates and requests being tracked: 7. Request ID '20191231201955': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: SelfSign issuer: CN=ipa1.itec.lab,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2020-12-31 20:19:55 UTC principal name: krbtgt/ITEC.LAB@ITEC.LAB certificate template/profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20201102185036': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=CA Audit,O=ITEC.LAB expires: 2020-12-08 09:35:14 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185037': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=OCSP Subsystem,O=ITEC.LAB expires: 2020-12-08 09:38:07 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185038': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=CA Subsystem,O=ITEC.LAB expires: 2020-12-08 09:37:36 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185039': status: NEED_CSR_GEN_PIN stuck: yes key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent-selfsigned issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=Certificate Authority,O=ITEC.LAB expires: 2037-01-25 14:22:25 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185040': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=IPA RA,O=ITEC.LAB expires: 2020-12-08 09:37:47 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20201102185042': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2020-12-08 09:35:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes
I had to set my date in several weeks before the expiring to renew them via certmonger, but it does not auto-renew past 30-12-2020
Thanks for the support, Manuel _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi,
I re-sync the date to today and ran ipa-cert-fix but it returns an error
[root@ipa1 ~]# ipa-cert-fix
WARNING
ipa-cert-fix is intended for recovery when expired certificates prevent the normal operation of IPA. It should ONLY be used in such scenarios, and backup of the system, especially certificates and keys, is STRONGLY RECOMMENDED.
The following certificates will be renewed:
Dogtag sslserver certificate: Subject: CN=ipa1.itec.lab,O=ITEC.LAB Serial: 17 Expires: 2020-12-08 09:35:05
Dogtag subsystem certificate: Subject: CN=CA Subsystem,O=ITEC.LAB Serial: 19 Expires: 2020-12-08 09:37:36
Dogtag ca_ocsp_signing certificate: Subject: CN=OCSP Subsystem,O=ITEC.LAB Serial: 21 Expires: 2020-12-08 09:38:07
Dogtag ca_audit_signing certificate: Subject: CN=CA Audit,O=ITEC.LAB Serial: 18 Expires: 2020-12-08 09:35:14
IPA IPA RA certificate: Subject: CN=IPA RA,O=ITEC.LAB Serial: 20 Expires: 2020-12-08 09:37:47
IPA Apache HTTPS certificate: Subject: CN=ipa1.itec.lab,O=ITEC.LAB Serial: 24 Expires: 2020-12-30 09:35:04
IPA LDAP certificate: Subject: CN=ipa1.itec.lab,O=ITEC.LAB Serial: 25 Expires: 2020-12-30 09:35:16
IPA KDC certificate: Subject: CN=ipa1.itec.lab,O=ITEC.LAB Serial: 1 Expires: 2020-12-31 20:19:55
Enter "yes" to proceed: yes Proceeding. [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.crt' The ipa-cert-fix command failed.
On 2/8/21 2:56 PM, Manuel Gujo via FreeIPA-users wrote:
Hi,
I re-sync the date to today and ran ipa-cert-fix but it returns an error
[root@ipa1 ~]# ipa-cert-fix
WARNINGipa-cert-fix is intended for recovery when expired certificates prevent the normal operation of IPA. It should ONLY be used in such scenarios, and backup of the system, especially certificates and keys, is STRONGLY RECOMMENDED.
The following certificates will be renewed:
Dogtag sslserver certificate: Subject: CN=ipa1.itec.lab,O=ITEC.LAB Serial: 17 Expires: 2020-12-08 09:35:05
Dogtag subsystem certificate: Subject: CN=CA Subsystem,O=ITEC.LAB Serial: 19 Expires: 2020-12-08 09:37:36
Dogtag ca_ocsp_signing certificate: Subject: CN=OCSP Subsystem,O=ITEC.LAB Serial: 21 Expires: 2020-12-08 09:38:07
Dogtag ca_audit_signing certificate: Subject: CN=CA Audit,O=ITEC.LAB Serial: 18 Expires: 2020-12-08 09:35:14
IPA IPA RA certificate: Subject: CN=IPA RA,O=ITEC.LAB Serial: 20 Expires: 2020-12-08 09:37:47
IPA Apache HTTPS certificate: Subject: CN=ipa1.itec.lab,O=ITEC.LAB Serial: 24 Expires: 2020-12-30 09:35:04
IPA LDAP certificate: Subject: CN=ipa1.itec.lab,O=ITEC.LAB Serial: 25 Expires: 2020-12-30 09:35:16
IPA KDC certificate: Subject: CN=ipa1.itec.lab,O=ITEC.LAB Serial: 1 Expires: 2020-12-31 20:19:55
Enter "yes" to proceed: yes Proceeding. [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.crt' The ipa-cert-fix command failed.
Hi, which version of pki-server is installed? You may be hitting https://bugzilla.redhat.com/show_bug.cgi?id=1897120
Looks like you will need to manually fix the renewal issue by following the old good method with changing date etc... The first expiration date is 2020-12-08, the system date needs to be moved before that date. Please try and let me know if there are issues.
flo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi,
pki-server version is 10.5.17, if I hit pki-server-upgrade it says this:
# pki-server-upgrade Upgrading from version 10.5.17 to 10.5.18: 1. Fix EC admin certificate profile (Yes/No) [Y]:
could be helpful?
I also found this: https://www.dogtagpki.org/wiki/Tomcat_SSL_Configuration_with_OpenSSL
could generating a self signed sslserver.crt a good idea?
in the bugzilla link you sent, do I have to download and run that pyscript?
About the old method with turn the date before expiration, what I have to do precisely? Because I already done this:
#getcert list //list all the cert IDs
#ipa-getcert resubmit -i <ID> //for every cert listed before
#systemctl certmonger restart
Thanks for the support Flo
Manuel
Hi,
I've retried to move date three weeks before 2020-12-08 and renew cert manually
# ipa-getcert resubmit -i "ID" Resubmitting "20201102185036" to "dogtag-ipa-ca-renew-agent".
Here's one of the output log from journalctl -xe
# journalctl -xe nov 17 18:08:27 ipa1.itec.lab certmonger[27108]: 2020-11-17 18:08:27 [27108] Internal error nov 17 18:08:29 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[28053]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 507, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (252963
now all the certs (except from kerberos and CA ones) are status: CA_UNREACHABLE.
CA cert is status: NEED_CSR_GEN_PIN
Manuel Gujo via FreeIPA-users wrote:
Hi,
I've retried to move date three weeks before 2020-12-08 and renew cert manually
# ipa-getcert resubmit -i "ID" Resubmitting "20201102185036" to "dogtag-ipa-ca-renew-agent".
Here's one of the output log from journalctl -xe
# journalctl -xe nov 17 18:08:27 ipa1.itec.lab certmonger[27108]: 2020-11-17 18:08:27 [27108] Internal error nov 17 18:08:29 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[28053]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 507, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (252963
now all the certs (except from kerberos and CA ones) are status: CA_UNREACHABLE.
CA cert is status: NEED_CSR_GEN_PIN
When you are moving back in time are you bringing the IPA services back up? You need to do this manually if you have an NTP server enabled (which it is by default).
Minimum you need to restart, in order, dirsrv.target, krb5kdc, named, httpd, pki-tomcatd. If you restart certmonger it may kick off the renewals for you (or it might not).
If you can get the services running back in time then runipa config-show to determine whether this server is configured as the CA renewal server. Only one in the cluster will have this role and the renewals need to take place on that server. If this one isn't it and none of the others report it then you can run: ipa config-mod --ca-renewal-master-server=<fqdn>
As Flo said the NEED_CSR_GEN_PIN is from your using ipa-cacert-manage. It doesn't affect anything in the short term.
rob
Hi Rob,
do I have to stop all the IPA services before i move back the date? Now I'm only moving back date and restarting certmonger.
pki-tomcatd is failed so i can't stop/restart it
Manuel Gujo via FreeIPA-users wrote:
Hi Rob,
do I have to stop all the IPA services before i move back the date? Now I'm only moving back date and restarting certmonger.
It wouldn't hurt.
You absolutely need to restart things in the past because they can't run in current time with expired certs.
pki-tomcatd is failed so i can't stop/restart it
Then go back in the past and we can try to figure out why it won't start then. It won't start now due to expired certs.
You can't renew the certs without a working CA.
rob
I moved the date before the expiring and restarted the services one by one as you listed (systemctl restart dirsrv@my-domain, systemctl restart krb5kdc etc.)
then: [root@ipa1 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: STOPPED (if I do systemctl status named it says running) httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: STOPPED ipa-dnskeysyncd Service: STOPPED ipa: INFO: The ipactl command was successful
pki-tomcatd failed to start:
# systemctl restart pki-tomcatd@ITEC-LAB Job for pki-tomcatd@ITEC-LAB.service failed because the control process exited with error code. See "systemctl status pki-tomcatd@ITEC-LAB.service" and "journalctl -xe" for details.
# journalctl -xe nov 17 18:22:31 ipa1.itec.lab systemd[1]: Unit pki-tomcatd@ITEC-LAB.service entered failed state. nov 17 18:22:31 ipa1.itec.lab audispd[24456]: node=ipa1.itec.lab type=SERVICE_START msg=audit(1605637351.916:7091): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=pki-tomc nov 17 18:22:31 ipa1.itec.lab systemd[1]: pki-tomcatd@ITEC-LAB.service failed. nov 17 18:22:31 ipa1.itec.lab polkitd[30556]: Unregistered Authentication Agent for unix-process:17970:71502359 (system bus name :1.1719, object path /org/freedesktop/PolicyKit1
Manuel Gujo via FreeIPA-users wrote:
I moved the date before the expiring and restarted the services one by one as you listed (systemctl restart dirsrv@my-domain, systemctl restart krb5kdc etc.)
then: [root@ipa1 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: STOPPED (if I do systemctl status named it says running) httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: STOPPED ipa-dnskeysyncd Service: STOPPED ipa: INFO: The ipactl command was successful
pki-tomcatd failed to start:
# systemctl restart pki-tomcatd@ITEC-LAB Job for pki-tomcatd@ITEC-LAB.service failed because the control process exited with error code. See "systemctl status pki-tomcatd@ITEC-LAB.service" and "journalctl -xe" for details.
# journalctl -xe nov 17 18:22:31 ipa1.itec.lab systemd[1]: Unit pki-tomcatd@ITEC-LAB.service entered failed state. nov 17 18:22:31 ipa1.itec.lab audispd[24456]: node=ipa1.itec.lab type=SERVICE_START msg=audit(1605637351.916:7091): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=pki-tomc nov 17 18:22:31 ipa1.itec.lab systemd[1]: pki-tomcatd@ITEC-LAB.service failed. nov 17 18:22:31 ipa1.itec.lab polkitd[30556]: Unregistered Authentication Agent for unix-process:17970:71502359 (system bus name :1.1719, object path /org/freedesktop/PolicyKit1
Look in /var/log/pki/pki-tomcat/ca/debug. Find where it tries to start the service and go down from there. Looking at the end of the log is almost always fruitless.
rob
Here's what I found in /var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host ipa1.itec.lab port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:667) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1056) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:962) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:568) at com.netscape.certsrv.apps.CMS.init(CMS.java:191) at com.netscape.certsrv.apps.CMS.start(CMS.java:1458) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Internal Database Error encountered: Could not connect to LDAP server host ipa1.itec.lab port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1056) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:962) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:568) at com.netscape.certsrv.apps.CMS.init(CMS.java:191) at com.netscape.certsrv.apps.CMS.start(CMS.java:1458) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [31/Dec/2019:19:55:52][localhost-startStop-1]: CMS.start(): shutdown server [31/Dec/2019:19:55:52][localhost-startStop-1]: CMSEngine.shutdown()
Manuel Gujo via FreeIPA-users wrote:
Here's what I found in /var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host ipa1.itec.lab port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)
Double-check that dirsrv is started and listening on port 636. I assume the host name is correct.
rob
# systemctl status dirsrv@ITEC-LAB ● dirsrv@ITEC-LAB.service - 389 Directory Server ITEC-LAB. Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled) Active: active (running) since mar 2020-11-17 18:00:26 UTC; 2 months 28 days ago Main PID: 15817 (ns-slapd) Status: "slapd started: Ready to process requests" CGroup: /system.slice/system-dirsrv.slice/dirsrv@ITEC-LAB.service └─15817 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ITEC-LAB -i /var/run/dirsrv/slapd-ITEC-LAB.pid
feb 12 17:26:03 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3
# netstat -tulpn | grep LISTEN tcp 0 0 0.0.0.0:749 0.0.0.0:* LISTEN 1222/kadmind tcp 0 0 0.0.0.0:464 0.0.0.0:* LISTEN 1222/kadmind tcp 0 0 192.168.20.3:53 0.0.0.0:* LISTEN 17818/named tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 17818/named tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 24121/sshd tcp 0 0 0.0.0.0:88 0.0.0.0:* LISTEN 15869/krb5kdc tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 17818/named tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 24232/master tcp6 0 0 :::749 :::* LISTEN 1222/kadmind tcp6 0 0 :::80 :::* LISTEN 16122/httpd tcp6 0 0 :::464 :::* LISTEN 1222/kadmind tcp6 0 0 :::53 :::* LISTEN 17818/named tcp6 0 0 :::22 :::* LISTEN 24121/sshd tcp6 0 0 :::88 :::* LISTEN 15869/krb5kdc tcp6 0 0 ::1:953 :::* LISTEN 17818/named tcp6 0 0 ::1:25 :::* LISTEN 24232/master tcp6 0 0 :::443 :::* LISTEN 16122/httpd
dirsrv is started but I don't see port 636 on this list, how can I open it for dirsrv?
Manuel Gujo via FreeIPA-users wrote:
# systemctl status dirsrv@ITEC-LAB ● dirsrv@ITEC-LAB.service - 389 Directory Server ITEC-LAB. Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled) Active: active (running) since mar 2020-11-17 18:00:26 UTC; 2 months 28 days ago Main PID: 15817 (ns-slapd) Status: "slapd started: Ready to process requests" CGroup: /system.slice/system-dirsrv.slice/dirsrv@ITEC-LAB.service └─15817 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ITEC-LAB -i /var/run/dirsrv/slapd-ITEC-LAB.pid
feb 12 17:26:03 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3
# netstat -tulpn | grep LISTEN tcp 0 0 0.0.0.0:749 0.0.0.0:* LISTEN 1222/kadmind tcp 0 0 0.0.0.0:464 0.0.0.0:* LISTEN 1222/kadmind tcp 0 0 192.168.20.3:53 0.0.0.0:* LISTEN 17818/named tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 17818/named tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 24121/sshd tcp 0 0 0.0.0.0:88 0.0.0.0:* LISTEN 15869/krb5kdc tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 17818/named tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 24232/master tcp6 0 0 :::749 :::* LISTEN 1222/kadmind tcp6 0 0 :::80 :::* LISTEN 16122/httpd tcp6 0 0 :::464 :::* LISTEN 1222/kadmind tcp6 0 0 :::53 :::* LISTEN 17818/named tcp6 0 0 :::22 :::* LISTEN 24121/sshd tcp6 0 0 :::88 :::* LISTEN 15869/krb5kdc tcp6 0 0 ::1:953 :::* LISTEN 17818/named tcp6 0 0 ::1:25 :::* LISTEN 24232/master tcp6 0 0 :::443 :::* LISTEN 16122/httpd
dirsrv is started but I don't see port 636 on this list, how can I open it for dirsrv?
In fact it isn't listening on any port.
Stop dirsrv and look in /etc/dirsrv/slapd-ITEC-LAB/dse.ldif for nsslapd-port which should be 389 and nsslapd-security which should be on. If not then fix it and restart dirsrv. That should fix it.
rob
Hi Rob,
so in "/etc/dirsrv/slapd-ITEC-LAB/dse.ldif", nsslapd-port was '0' and nsslapd-security was off, I fixed it and now it's listening on port 389 and 636:
# netstat -tulpn | grep LISTEN | grep ns-slapd tcp6 0 0 :::636 :::* LISTEN 30606/ns-slapd tcp6 0 0 :::389 :::* LISTEN 30606/ns-slapd
Then I tried to restart all the ipactl services one by one. pki-tomcatd keeps failing and /var/log/pki/pki-tomcat/ca/debug does not log anymore (last log is the one i sent you above, 31 Dec 2019)
I resubmitted all the expired certs and restarting cermonger but certs keep being unreachable.
from certmonger logs:
nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]: Forwarding request to dogtag-ipa-renew-agent nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: GET http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit?profileId=caServerCert&... nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: (null) nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]: dogtag-ipa-renew-agent returned 3 nov 17 18:11:47 ipa1.itec.lab certmonger[30685]: 2020-11-17 18:11:47 [30685] Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.
in certmonger's log I also saw these:
nov 17 18:11:01 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30741]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 495, in main api.finalize() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 740, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 431, in __do_if_not_done getattr(self, name)() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 620, in load_plugins self.add_package(package) File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 647, in add_package module = importlib.import_module(name) File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/usr/lib/python2.7/site-packages/ipaserver/plugins/server.py", line 32, in <module> from ipaserver.install import bindinstance, dnskeysyncinstance File "/usr/lib/python2.7/site-packages/ipaserver/install/dnskeysyncinstance.py", line 17, in <module> from ipaserver import p11helper as _ipap11helper File "/usr/lib/python2.7/site-packages/ipaserver/p11helper.py", line 342, in <module> _libp11_kit = _ffi.dlopen(ctypes.util.find_library('p11-kit')) File "/usr/lib64/python2.7/ctypes/util.py", line 244, in find_library return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)) File "/usr/lib64/python2.7/ctypes/util.py", line 233, in _findSoname_ldconfig f = os.popen('/sbin/ldconfig -p 2>/dev/null') OSError: [Errno 12] Cannot allocate memory
Manuel Gujo via FreeIPA-users wrote:
Hi Rob,
so in "/etc/dirsrv/slapd-ITEC-LAB/dse.ldif", nsslapd-port was '0' and nsslapd-security was off, I fixed it and now it's listening on port 389 and 636:
# netstat -tulpn | grep LISTEN | grep ns-slapd tcp6 0 0 :::636 :::* LISTEN 30606/ns-slapd tcp6 0 0 :::389 :::* LISTEN 30606/ns-slapd
Then I tried to restart all the ipactl services one by one. pki-tomcatd keeps failing and /var/log/pki/pki-tomcat/ca/debug does not log anymore (last log is the one i sent you above, 31 Dec 2019)
I resubmitted all the expired certs and restarting cermonger but certs keep being unreachable.
If the CA isn't running then there is no point in resubmitting the certmonger requests. It is guaranteed to fail with UNREACHABLE.
Check the journalctl output and the other logs, like catalina, in /var/log/pki/pki-tomcat for more information on why it failed to start.
from certmonger logs:
nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]: Forwarding request to dogtag-ipa-renew-agent nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: GET http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit?profileId=caServerCert&... nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: (null) nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]: dogtag-ipa-renew-agent returned 3 nov 17 18:11:47 ipa1.itec.lab certmonger[30685]: 2020-11-17 18:11:47 [30685] Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.
in certmonger's log I also saw these:
nov 17 18:11:01 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30741]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 495, in main api.finalize() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 740, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 431, in __do_if_not_done getattr(self, name)() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 620, in load_plugins self.add_package(package) File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 647, in add_package module = importlib.import_module(name) File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/usr/lib/python2.7/site-packages/ipaserver/plugins/server.py", line 32, in <module> from ipaserver.install import bindinstance, dnskeysyncinstance File "/usr/lib/python2.7/site-packages/ipaserver/install/dnskeysyncinstance.py", line 17, in <module> from ipaserver import p11helper as _ipap11helper File "/usr/lib/python2.7/site-packages/ipaserver/p11helper.py", line 342, in <module> _libp11_kit = _ffi.dlopen(ctypes.util.find_library('p11-kit')) File "/usr/lib64/python2.7/ctypes/util.py", line 244, in find_library return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)) File "/usr/lib64/python2.7/ctypes/util.py", line 233, in _findSoname_ldconfig f = os.popen('/sbin/ldconfig -p 2>/dev/null') OSError: [Errno 12] Cannot allocate memory
Is this host memory-constrained? How much RAM does it have?
rob
Manuel Gujo via FreeIPA-users wrote:
If the CA isn't running then there is no point in resubmitting the certmonger requests. It is guaranteed to fail with UNREACHABLE.
Check the journalctl output and the other logs, like catalina, in /var/log/pki/pki-tomcat for more information on why it failed to start.
Is this host memory-constrained? How much RAM does it have?
rob
there's new log on debug. Catalina does not log anything (0kb per file). in debug:
Could not connect to LDAP server host ipa1.itec.lab port 636 Error netscape.ldap.LDAPException: Unable to create socket: org.mozilla.jss.ssl.SSLSocketException: org.mozilla.jss.ssl.SSLSocketExc eption: SSL_ForceHandshake failed: (-8181) Peer's Certificate has expired. (-1)
in "system" logs says the same thing of debugs'
When I try to run 'ipactl start' without -f option, it says this: # ipactl start IPA version error: data needs to be upgraded (expected version '4.6.8-5.el7.centos', current version '4.4.0-14.el7.centos.4')
then after a while it fails and in /var/log/ipaupgrade.log says:
2020-11-17T18:25:05Z DEBUG httplib request failed: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 220, in _httplib_request conn.request(method, path, body=request_body, headers=headers) File "/usr/lib64/python2.7/httplib.py", line 1056, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1090, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 1052, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 890, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 852, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 1266, in connect HTTPConnection.connect(self) File "/usr/lib64/python2.7/httplib.py", line 833, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused 2020-11-17T18:25:05Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2020-11-17T18:25:05Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2176, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2059, in upgrade_configuration cainstance.repair_profile_caIPAserviceCert() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1949, in repair_profile_caIPAserviceCert with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1311, in __enter__ method='GET' File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 167, in https_request method=method, headers=headers) File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 229, in _httplib_request raise NetworkError(uri=uri, error=str(e))
2020-11-17T18:25:05Z DEBUG The ipa-server-upgrade command failed, exception: NetworkError: cannot connect to 'https://ipa1.itec.lab:8443/ca/rest/account/login': [Errno 111] Connection refused 2020-11-17T18:25:05Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
After this run, I noticed that some of the certs went on Monitoring state
# getcert list Number of certificates and requests being tracked: 9. Request ID '20191231201955': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: SelfSign issuer: CN=ipa1.itec.lab,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2022-02-08 15:59:12 UTC principal name: krbtgt/ITEC.LAB@ITEC.LAB certificate template/profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20201117182331': status: CA_UNREACHABLE ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=CA Audit,O=ITEC.LAB expires: 2020-12-08 09:35:14 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182333': status: CA_UNREACHABLE ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=OCSP Subsystem,O=ITEC.LAB expires: 2020-12-08 09:38:07 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182335': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=CA Subsystem,O=ITEC.LAB expires: 2022-11-07 18:24:47 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182336': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=Certificate Authority,O=ITEC.LAB expires: 2037-01-25 14:22:25 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182338': status: CA_UNREACHABLE ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=IPA RA,O=ITEC.LAB expires: 2020-12-08 09:37:47 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20201117182339': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2022-11-07 18:24:56 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182342': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-ITEC-LAB',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-ITEC-LAB/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-ITEC-LAB',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2020-12-30 09:35:16 UTC principal name: ldap/ipa1.itec.lab@ITEC.LAB key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv ITEC-LAB track: yes auto-renew: yes Request ID '20201117182351': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=ITEC.LAB subject: CN=ipa1.itec.lab,O=ITEC.LAB expires: 2020-12-30 09:35:04 UTC principal name: HTTP/ipa1.itec.lab@ITEC.LAB key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: STOPPED httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: STOPPED pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
but pki-tomcatd still fails if I try to restart it and in the debug logs:
[17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet:service() uri = /ca/admin/ca/getStatus [17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet: caGetStatus start to service. [17/Nov/2020:18:32:34][http-bio-8080-exec-7]: Failed to read product version String. java.io.FileNotFoundException: /usr/share/pki/CS_SERVER_VERSION (No such file or directory) [17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet: curDate=Tue Nov 17 18:32:34 UTC 2020 id=caGetStatus time=9
IPA VM has 2 CPU and 4GB of RAM, it never goes up to 90% of the usage
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Manuel Gugliucci -
Manuel Gujo
-
Rob Crittenden