Hi,

it seems that PKI is not happy with the subject name of the certificates.

The failing certs are for KDC, dirsrv and httpd and they all use the same subject name constraint in their profile.

1. Was any certificate profile modified (caIPAserviceCert or KDCs_PKINIT_Certs)? You can use

ipa certprofile-show <name> --out /dev/stdout

And then check the part related to Subject Name Constraint. In my default installation, I have

policyset.serverCertSet.1.constraint.class_id=subjectNameConstraintImpl
policyset.serverCertSet.1.constraint.name=Subject Name Constraint
policyset.serverCertSet.1.constraint.params.accept=true
policyset.serverCertSet.1.constraint.params.pattern=CN=[^,]+,.+
policyset.serverCertSet.1.default.class_id=subjectNameDefaultImpl
policyset.serverCertSet.1.default.name=Subject Name Default
policyset.serverCertSet.1.default.params.name=CN=$request.req_subject_name.cn$, O=IPA.TEST

which means that the subject name should match CN= followed by (anything except a comma) multiple times then a comma and any char multiple times.

2. If the profile wasn't changed, can you check in /var/log/pki/pki-tomcat/ca/debug.$DATE.log the received certificate request? Does its subject match the pattern? The error message java.lang.StringIndexOutOfBoundsException: String index out of range: -1 hints that an expected pattern was not found.

flo

On Thu, Sep 14, 2023 at 4:11 PM Evan G via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:

Hi Rob,

When we start tomcat with the date rolled back, we are not seeing any errors at all. All of the ipa services start up without issue. The problem is in actually renewing the certs, when we do so we have seen many different errors as we've been troubleshooting -- mostly this one: `ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will retry: 4035 (RPC failed at server.  Request failed with status 500: Non-2xx response from CA REST API: 500. String index out of range: -1).[02/Aug/2023:00:00:31][ajp-bio-127.0.0.1-8009-exec-2]: EnrollProfile: populate: begins`

When I restart certmonger after all services up, these are the errors that I am seeing in the tomcat debug logs:
```
[02/Aug/2023:00:00:31][ajp-bio-127.0.0.1-8009-exec-2]: BasicProfile: populate: policy setid =serverCertSet
[02/Aug/2023:00:00:31][ajp-bio-127.0.0.1-8009-exec-2]: EnrollDefault: populate: SubjectNameDefault: start
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
        at java.lang.String.substring(String.java:1967)
        at com.netscape.certsrv.pattern.Pattern.substitute2(Pattern.java:132)
        at com.netscape.cms.profile.def.EnrollDefault.mapPattern(EnrollDefault.java:815)
        at com.netscape.cms.profile.def.SubjectNameDefault.populate(SubjectNameDefault.java:160)
        at com.netscape.cms.profile.def.EnrollDefault.populate(EnrollDefault.java:226)
        at com.netscape.cms.profile.common.BasicProfile.populate(BasicProfile.java:1114)
        at com.netscape.cms.profile.common.EnrollProfile.populate(EnrollProfile.java:2626)
        at com.netscape.cms.servlet.cert.CertProcessor.populateRequests(CertProcessor.java:379)
        at com.netscape.cms.servlet.cert.EnrollmentProcessor.processEnrollment(EnrollmentProcessor.java:188)
        at com.netscape.cms.servlet.cert.EnrollmentProcessor.processEnrollment(EnrollmentProcessor.java:96)
        at com.netscape.cms.servlet.cert.CertRequestDAO.submitRequest(CertRequestDAO.java:197)
        at org.dogtagpki.server.ca.rest.CertRequestService.enrollCert(CertRequestService.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297)
        at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
        at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
        at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
        at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
        at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
        at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:750)
```

This is what we see when we run `getcert list` and `ipa-getcert list` respectively:

```
Number of certificates and requests being tracked: 9.
Request ID '20190920201259':
        status: CA_UNREACHABLE
        ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will retry: 4035 (RPC failed at server.  Request failed with status 500: Non-2xx response from CA REST API: 500. String index out of range: -1).
        stuck: no
        key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
        certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=<HOSTNAME>,O=<OU>
        expires: 2023-08-25 18:05:07 UTC
        principal name: krbtgt/<OU>@<OU>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-pkinit-KPKdc
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
        track: yes
        auto-renew: yes
Request ID '20210908000050':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS FIPS 140-2 Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS FIPS 140-2 Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=CA Audit,O=<OU>
        expires: 2025-07-21 02:36:57 UTC
        key usage: digitalSignature,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20210908000051':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS FIPS 140-2 Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS FIPS 140-2 Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=OCSP Subsystem,O=<OU>
        expires: 2025-07-21 02:36:17 UTC
        key usage: digitalSignature,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20210908000052':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS FIPS 140-2 Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS FIPS 140-2 Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=CA Subsystem,O=<OU>
        expires: 2025-07-21 02:37:17 UTC
        key usage: digitalSignature,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20210908000053':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS FIPS 140-2 Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS FIPS 140-2 Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=Certificate Authority,O=<OU>
        expires: 2039-09-20 20:11:25 UTC
        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20210908000054':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=IPA RA,O=<OU>
        expires: 2025-06-26 02:36:15 UTC
        key usage: digitalSignature,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
Request ID '20210908000055':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS FIPS 140-2 Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS FIPS 140-2 Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=<HOSTNAME>,O=<OU>
        expires: 2025-07-21 02:36:37 UTC
        dns: <HOSTNAME>
        key usage: digitalSignature,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20210908000056':
        status: CA_UNREACHABLE
        ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will retry: 4035 (RPC failed at server.  Request failed with status 500: Non-2xx response from CA REST API: 500. String index out of range: -1).
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-<OU>',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS FIPS 140-2 Certificate DB',pinfile='/etc/dirsrv/slapd-<OU>/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-<OU>',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS FIPS 140-2 Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=<HOSTNAME>,O=<OU>
        expires: 2023-09-03 18:30:48 UTC
        dns: <HOSTNAME>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv <OU>
        track: yes
        auto-renew: yes
Request ID '20210908000057':
        status: CA_UNREACHABLE
        ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will retry: 4035 (RPC failed at server.  Request failed with status 500: Non-2xx response from CA REST API: 500. String index out of range: -1).
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS FIPS 140-2 Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS FIPS 140-2 Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=<HOSTNAME>,O=<OU>
        expires: 2023-09-03 18:30:48 UTC
        dns: <HOSTNAME>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
```

```
Number of certificates and requests being tracked: 9.
Request ID '20190920201259':
        status: CA_UNREACHABLE
        ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will retry: 4035 (RPC failed at server.  Request failed with status 500: Non-2xx response from CA REST API: 500. String index out of range: -1).
        stuck: no
        key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
        certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=<HOSTNAME>,O=<OU>
        expires: 2023-08-25 18:05:07 UTC
        principal name: krbtgt/<OU>@<OU>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-pkinit-KPKdc
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
        track: yes
        auto-renew: yes
Request ID '20210908000056':
        status: CA_UNREACHABLE
        ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will retry: 4035 (RPC failed at server.  Request failed with status 500: Non-2xx response from CA REST API: 500. String index out of range: -1).
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-<OU>',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS FIPS 140-2 Certificate DB',pinfile='/etc/dirsrv/slapd-<OU>/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-<OU>',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS FIPS 140-2 Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=<HOSTNAME>,O=<OU>
        expires: 2023-09-03 18:30:48 UTC
        dns: <HOSTNAME>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv <OU>
        track: yes
        auto-renew: yes
Request ID '20210908000057':
        status: CA_UNREACHABLE
        ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will retry: 4035 (RPC failed at server.  Request failed with status 500: Non-2xx response from CA REST API: 500. String index out of range: -1).
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS FIPS 140-2 Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS FIPS 140-2 Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=<OU>
        subject: CN=<HOSTNAME>,O=<OU>
        expires: 2023-09-03 18:30:48 UTC
        dns: <HOSTNAME>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
```
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue