Similar to a number of other posts, I have a server on which pki-tomcatd won't start.
I just have two servers; a master and replica. I haven't upgraded anything recently. The problem started two days ago when the server certificates renewed. The renewal appears to have been executed successfully and getcert list on both machines shows valid certs (many recently renewed). I believe it was the certificate renewal that triggered the problem because the replica's localhost_access log shows this transition:
10.1.5.8 - - [16/Oct/2024:19:30:05 -0600] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 119 10.1.5.8 - - [16/Oct/2024:20:24:10 -0600] "GET /ca/admin/ca/getStatus HTTP/1.1" 404 784
the same day as the renewal (everything before was successful; everything after failed). Subsequently last night, the server restarted and pki-tomcatd would not restart. I don't think that the master has restarted since the renewal and honestly I'm afraid to try it...
I started troubleshooting with https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat... and the cert on both machines is identical and valid. Like others, this command fails on both master and replica:
$ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
However, this is successful on both:
$ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt
and I saw mention somewhere that if the latter is successful, then everything must be fine.
/var/log/pki/pki-tomcat/ca/debug log shows:
2024-10-18 12:30:38 [main] INFO: CMSEngine: initializing password store 2024-10-18 12:30:38 [main] INFO: CMSEngine: initializing password store for internaldb 2024-10-18 12:30:38 [main] INFO: CMSEngine: initializing password store for replicationdb 2024-10-18 12:30:38 [main] INFO: CMSEngine: Initializing subsystem listeners 2024-10-18 12:30:38 [main] INFO: CMSEngine: Java version: 17.0.5 2024-10-18 12:30:38 [main] INFO: CMSEngine: security providers: 2024-10-18 12:30:38 [main] INFO: PluginRegistry: Loading plugin registry from /var/lib/pki/pki-tomcat/conf/ca/registry.cfg 2024-10-18 12:30:38 [main] SEVERE: LdapBoundConnFactory: Unable to connect to LDAP server: Authentication failed netscape.ldap.LDAPException: Authentication failed (49) at netscape.ldap.LDAPSaslBind.checkForSASLBindCompletion(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.checkClientAuth(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at com.netscape.cmscore.ldapconn.LdapBoundConnection.<init>(LdapBoundConnection.java:108) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:287) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:263) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:226) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:195) at org.dogtagpki.server.ca.CAEngine.initDatabase(CAEngine.java:199) at com.netscape.cmscore.apps.CMSEngine.start(CMSEngine.java:1105) at com.netscape.cmscore.apps.CMSEngine.contextInitialized(CMSEngine.java:1690) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:129) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:150) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:140) at java.base/java.security.AccessController.doPrivileged(AccessController.java:318) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:583) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:473) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
2024-10-18 12:30:38 [main] SEVERE: Unable to start CA engine: Unable to connect to LDAP server: Authentication failed Unable to connect to LDAP server: Authentication failed at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:305) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:263) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:226) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:195) at org.dogtagpki.server.ca.CAEngine.initDatabase(CAEngine.java:199) at com.netscape.cmscore.apps.CMSEngine.start(CMSEngine.java:1105) at com.netscape.cmscore.apps.CMSEngine.contextInitialized(CMSEngine.java:1690) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:129) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:150) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:140) at java.base/java.security.AccessController.doPrivileged(AccessController.java:318) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:583) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:473) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476) Caused by: netscape.ldap.LDAPException: Authentication failed (49) at netscape.ldap.LDAPSaslBind.checkForSASLBindCompletion(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.checkClientAuth(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at com.netscape.cmscore.ldapconn.LdapBoundConnection.<init>(LdapBoundConnection.java:108) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:287) ... 51 more
2024-10-18 12:30:38 [main] INFO: Shutting down CA subsystem 2024-10-18 12:30:38 [main] INFO: RequestSubsystem: Request subsystem stopped 2024-10-18 12:30:38 [main] SEVERE: Exception sending context initialized event to listener instance of class [org.dogtagpki.server.ca.CAEngine] java.lang.RuntimeException: Unable to start CA engine: Unable to connect to LDAP server: Authentication failed at com.netscape.cmscore.apps.CMSEngine.contextInitialized(CMSEngine.java:1695) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:129) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:150) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:140) at java.base/java.security.AccessController.doPrivileged(AccessController.java:318) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:583) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:473) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476) Caused by: Unable to connect to LDAP server: Authentication failed at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:305) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:263) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:226) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:195) at org.dogtagpki.server.ca.CAEngine.initDatabase(CAEngine.java:199) at com.netscape.cmscore.apps.CMSEngine.start(CMSEngine.java:1105) at com.netscape.cmscore.apps.CMSEngine.contextInitialized(CMSEngine.java:1690) ... 45 more Caused by: netscape.ldap.LDAPException: Authentication failed (49) at netscape.ldap.LDAPSaslBind.checkForSASLBindCompletion(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.checkClientAuth(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at com.netscape.cmscore.ldapconn.LdapBoundConnection.<init>(LdapBoundConnection.java:108) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:287) ... 51 more
2024-10-18 12:30:38 [main] INFO: Shutting down CA subsystem 2024-10-18 12:30:38 [main] INFO: RequestSubsystem: Request subsystem stopped
I'm not sure if this is the same issue as other threads have requested help with or not—most of them seem to fizzle out without reporting the resolution. Any suggestions?
Sean McLennan via FreeIPA-users wrote:
Similar to a number of other posts, I have a server on which pki-tomcatd won't start.
I just have two servers; a master and replica. I haven't upgraded anything recently. The problem started two days ago when the server certificates renewed. The renewal appears to have been executed successfully and getcert list on both machines shows valid certs (many recently renewed). I believe it was the certificate renewal that triggered the problem because the replica's localhost_access log shows this transition:
10.1.5.8 - - [16/Oct/2024:19:30:05 -0600] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 119 10.1.5.8 - - [16/Oct/2024:20:24:10 -0600] "GET /ca/admin/ca/getStatus HTTP/1.1" 404 784
the same day as the renewal (everything before was successful; everything after failed). Subsequently last night, the server restarted and pki-tomcatd would not restart. I don't think that the master has restarted since the renewal and honestly I'm afraid to try it...
I started troubleshooting with https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat... and the cert on both machines is identical and valid. Like others, this command fails on both master and replica:
$ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
However, this is successful on both:
$ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt
and I saw mention somewhere that if the latter is successful, then everything must be fine.
It doesn't mean everything is fine. This just lists the CA private keys.
You don't say what distribution or release you are running. I'd recommend installing {free}ipa-healthcheck and seeing if that detects any issues.
rob
/var/log/pki/pki-tomcat/ca/debug log shows:
2024-10-18 12:30:38 [main] INFO: CMSEngine: initializing password store 2024-10-18 12:30:38 [main] INFO: CMSEngine: initializing password store for internaldb 2024-10-18 12:30:38 [main] INFO: CMSEngine: initializing password store for replicationdb 2024-10-18 12:30:38 [main] INFO: CMSEngine: Initializing subsystem listeners 2024-10-18 12:30:38 [main] INFO: CMSEngine: Java version: 17.0.5 2024-10-18 12:30:38 [main] INFO: CMSEngine: security providers: 2024-10-18 12:30:38 [main] INFO: PluginRegistry: Loading plugin registry from /var/lib/pki/pki-tomcat/conf/ca/registry.cfg 2024-10-18 12:30:38 [main] SEVERE: LdapBoundConnFactory: Unable to connect to LDAP server: Authentication failed netscape.ldap.LDAPException: Authentication failed (49) at netscape.ldap.LDAPSaslBind.checkForSASLBindCompletion(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.checkClientAuth(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at com.netscape.cmscore.ldapconn.LdapBoundConnection.<init>(LdapBoundConnection.java:108) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:287) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:263) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:226) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:195) at org.dogtagpki.server.ca.CAEngine.initDatabase(CAEngine.java:199) at com.netscape.cmscore.apps.CMSEngine.start(CMSEngine.java:1105) at com.netscape.cmscore.apps.CMSEngine.contextInitialized(CMSEngine.java:1690) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:129) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:150) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:140) at java.base/java.security.AccessController.doPrivileged(AccessController.java:318) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:583) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:473) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
2024-10-18 12:30:38 [main] SEVERE: Unable to start CA engine: Unable to connect to LDAP server: Authentication failed Unable to connect to LDAP server: Authentication failed at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:305) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:263) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:226) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:195) at org.dogtagpki.server.ca.CAEngine.initDatabase(CAEngine.java:199) at com.netscape.cmscore.apps.CMSEngine.start(CMSEngine.java:1105) at com.netscape.cmscore.apps.CMSEngine.contextInitialized(CMSEngine.java:1690) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:129) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:150) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:140) at java.base/java.security.AccessController.doPrivileged(AccessController.java:318) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:583) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:473) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476) Caused by: netscape.ldap.LDAPException: Authentication failed (49) at netscape.ldap.LDAPSaslBind.checkForSASLBindCompletion(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.checkClientAuth(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at com.netscape.cmscore.ldapconn.LdapBoundConnection.<init>(LdapBoundConnection.java:108) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:287) ... 51 more
2024-10-18 12:30:38 [main] INFO: Shutting down CA subsystem 2024-10-18 12:30:38 [main] INFO: RequestSubsystem: Request subsystem stopped 2024-10-18 12:30:38 [main] SEVERE: Exception sending context initialized event to listener instance of class [org.dogtagpki.server.ca.CAEngine] java.lang.RuntimeException: Unable to start CA engine: Unable to connect to LDAP server: Authentication failed at com.netscape.cmscore.apps.CMSEngine.contextInitialized(CMSEngine.java:1695) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:129) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:150) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:140) at java.base/java.security.AccessController.doPrivileged(AccessController.java:318) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:583) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:473) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476) Caused by: Unable to connect to LDAP server: Authentication failed at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:305) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:263) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:226) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:195) at org.dogtagpki.server.ca.CAEngine.initDatabase(CAEngine.java:199) at com.netscape.cmscore.apps.CMSEngine.start(CMSEngine.java:1105) at com.netscape.cmscore.apps.CMSEngine.contextInitialized(CMSEngine.java:1690) ... 45 more Caused by: netscape.ldap.LDAPException: Authentication failed (49) at netscape.ldap.LDAPSaslBind.checkForSASLBindCompletion(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPSaslBind.bind(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.authenticate(Unknown Source) at netscape.ldap.LDAPConnection.checkClientAuth(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at netscape.ldap.LDAPConnection.connect(Unknown Source) at com.netscape.cmscore.ldapconn.LdapBoundConnection.<init>(LdapBoundConnection.java:108) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:287) ... 51 more
2024-10-18 12:30:38 [main] INFO: Shutting down CA subsystem 2024-10-18 12:30:38 [main] INFO: RequestSubsystem: Request subsystem stopped
I'm not sure if this is the same issue as other threads have requested help with or not—most of them seem to fizzle out without reporting the resolution. Any suggestions?
You don't say what distribution or release you are running.
Apologies—I meant to add that and then got distracted. They are both CentoOS Stream 9 running 4.9.8 (master) and 4.10.0 (replica). I was actually surprised they aren't the same version—I'm not sure how that happened TBH.
I'd recommend installing {free}ipa-healthcheck and seeing if that detects any issues.
Thank you! I didn't know it existed—that's very useful.
On the master it only identifies that the replica is not functioning correctly. On the replica, the first thing it identified was the ldap / NSS DB mismatch on 'subsystemCert cert-pki-ca'—that I fixed with the instruction on https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat... and pki-tomcatd now starts. Although oddly ipa-healthcheck is still showing it as a error. And it seems other renewed certificates were not updated in the replica's ldap either:
[ { "source": "ipahealthcheck.dogtag.ca", "check": "DogtagCertsConnectivityCheck", "result": "ERROR", "uuid": "b64ca4ed-889b-4707-88f8-f6231adaaecb", "when": "20241019190638Z", "duration": "0.095653", "kw": { "key": "cert_show_ra", "error": "Certificate operation cannot be completed: Request failed with status 404: Non-2xx response from CA REST API: 404. Certificate ID 0x13 not found (404)", "serial": "19", "msg": "Serial number not found: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPADogtagCertsMatchCheck", "result": "ERROR", "uuid": "63e59ed6-42f8-4c7c-b884-1ff18aa879db", "when": "20241019190644Z", "duration": "0.654190", "kw": { "key": "ocspSigningCert cert-pki-ca", "nickname": "ocspSigningCert cert-pki-ca", "dbdir": "/etc/pki/pki-tomcat/alias", "msg": "{nickname} certificate in NSS DB {dbdir} does not match entry in LDAP" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPADogtagCertsMatchCheck", "result": "ERROR", "uuid": "f09f4527-a9fa-4b28-8433-7c802cf6e6b7", "when": "20241019190644Z", "duration": "0.725190", "kw": { "key": "subsystemCert cert-pki-ca", "nickname": "subsystemCert cert-pki-ca", "dbdir": "/etc/pki/pki-tomcat/alias", "msg": "{nickname} certificate in NSS DB {dbdir} does not match entry in LDAP" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPADogtagCertsMatchCheck", "result": "ERROR", "uuid": "32caa7a6-1eec-469b-9ba8-9e75a76f0a7f", "when": "20241019190644Z", "duration": "0.792584", "kw": { "key": "auditSigningCert cert-pki-ca", "nickname": "auditSigningCert cert-pki-ca", "dbdir": "/etc/pki/pki-tomcat/alias", "msg": "{nickname} certificate in NSS DB {dbdir} does not match entry in LDAP" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPARAAgent", "result": "ERROR", "uuid": "7f7b4a56-6c93-4b11-90f1-86a658c6259a", "when": "20241019190645Z", "duration": "0.021523", "kw": { "key": "description_mismatch", "expected": "2;19;CN=Certificate Authority,O=SIMPLYWS.COM;CN=IPA RA,O=SIMPLYWS.COM", "got": "2;7;CN=Certificate Authority,O=SIMPLYWS.COM;CN=IPA RA,O=SIMPLYWS.COM", "msg": "RA agent description does not match. Found {got} in LDAP and expected {expected}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "22545043-3a4d-43d7-af30-c6da0432ce8b", "when": "20241019190645Z", "duration": "0.120478", "kw": { "key": "20221129200204", "serial": 19, "error": "Certificate operation cannot be completed: Request failed with status 404: Non-2xx response from CA REST API: 404. Certificate ID 0x13 not found (404)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "9100d431-8156-4951-99f2-04bae21f8044", "when": "20241019190645Z", "duration": "0.294161", "kw": { "key": "20221129200205", "serial": 17, "error": "Certificate operation cannot be completed: Request failed with status 404: Non-2xx response from CA REST API: 404. Certificate ID 0x11 not found (404)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "9f30d1ea-03f5-44b5-abf8-7e369df23fb5", "when": "20241019190646Z", "duration": "0.468931", "kw": { "key": "20221129200209", "serial": 15, "error": "Certificate operation cannot be completed: Request failed with status 404: Non-2xx response from CA REST API: 404. Certificate ID 0xf not found (404)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "05d6e5cf-25e4-47c3-ac54-142e8f0a79cd", "when": "20241019190646Z", "duration": "0.644094", "kw": { "key": "20221129200212", "serial": 16, "error": "Certificate operation cannot be completed: Request failed with status 404: Non-2xx response from CA REST API: 404. Certificate ID 0x10 not found (404)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "f73d24c3-ecd2-45bb-81a8-95430d31ebd4", "when": "20241019190646Z", "duration": "1.138467", "kw": { "key": "20221129200213", "serial": 1, "error": "Failed to authenticate to CA REST API", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "a7d3a152-ce3a-4240-8e70-c959a11a710a", "when": "20241019190646Z", "duration": "1.367480", "kw": { "key": "20221129200214", "serial": 13, "error": "Failed to authenticate to CA REST API", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "7fb7d331-667b-4010-8678-a824bbc4bd02", "when": "20241019190647Z", "duration": "1.548316", "kw": { "key": "20221129195719", "serial": 12, "error": "Failed to authenticate to CA REST API", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "3728dfb8-0e7b-4ff7-b1fe-bc09ba6f58cf", "when": "20241019190647Z", "duration": "1.797597", "kw": { "key": "20221129195645", "serial": 11, "error": "Failed to authenticate to CA REST API", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "9b9afa55-06f6-4794-a6e8-7abb1c55b8ca", "when": "20241019190647Z", "duration": "1.981098", "kw": { "key": "20221129200256", "serial": 14, "error": "Failed to authenticate to CA REST API", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } } ]
All of which I guess points to a replication problem? Although changes that I've made to users have replicated fine, including ones after the certificate renewal and ipa-replica-conncheck doesn't report any problems...
Would 'ipa-replica-manage re-initialize' be a reasonable approach to resolving those missing entries?
Thank you for your help—it's much appreciated.
Sean McLennan via FreeIPA-users wrote:
You don't say what distribution or release you are running.
Apologies—I meant to add that and then got distracted. They are both CentoOS Stream 9 running 4.9.8 (master) and 4.10.0 (replica). I was actually surprised they aren't the same version—I'm not sure how that happened TBH.
I'd recommend installing {free}ipa-healthcheck and seeing if that detects any issues.
Thank you! I didn't know it existed—that's very useful.
On the master it only identifies that the replica is not functioning correctly. On the replica, the first thing it identified was the ldap / NSS DB mismatch on 'subsystemCert cert-pki-ca'—that I fixed with the instruction on https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat... and pki-tomcatd now starts. Although oddly ipa-healthcheck is still showing it as a error. And it seems other renewed certificates were not updated in the replica's ldap either:
Yes, it sure looks that way.
The 404's you see are because the CA didn't successfully start but tomcat did. So there is no registered servlet associated with the URI.
[snip]
All of which I guess points to a replication problem? Although changes that I've made to users have replicated fine, including ones after the certificate renewal and ipa-replica-conncheck doesn't report any problems...
Would 'ipa-replica-manage re-initialize' be a reasonable approach to resolving those missing entries?
There are two replication agreements. One for the IPA data (ipa-replica-manage) and one for the PKI data (ipa-csreplica-manage). It looks like the IPA agreement is working fine.
I'd try the force-sync command first to see if you can kickstart replication. If that fails then a re-init is probably in order.
rob
The 404's you see are because the CA didn't successfully start but tomcat did. So there is no registered servlet associated with the URI.
Oh—just for future reference, is there somewhere that can be seen (aside from ipa-healthcheck)? ipactl status shows everything running...
There are two replication agreements. One for the IPA data (ipa-replica-manage) and one for the PKI data (ipa-csreplica-manage).
Ah! Also good to know—I did not realize.
I'd try the force-sync command first to see if you can kickstart replication. If that fails then a re-init is probably in order.
The force-sync did not work, but re-init did and everything seems to be OK now.
Thanks for your help!
Sean McLennan via FreeIPA-users wrote:
The 404's you see are because the CA didn't successfully start but tomcat did. So there is no registered servlet associated with the URI.
Oh—just for future reference, is there somewhere that can be seen (aside from ipa-healthcheck)? ipactl status shows everything running...
Right. We can't easily test whether the applications within tomcat are running via systemd IIRC. Best we can do is see if tomcat is running. A simple way is: ipa cert-show 1. As long as it doesn't return an error the communication is working. This also exercises the RA cert.
There are two replication agreements. One for the IPA data (ipa-replica-manage) and one for the PKI data (ipa-csreplica-manage).
Ah! Also good to know—I did not realize.
I'd try the force-sync command first to see if you can kickstart replication. If that fails then a re-init is probably in order.
The force-sync did not work, but re-init did and everything seems to be OK now.
OK. Glad you're back up and running again.
rob
freeipa-users@lists.fedorahosted.org
-
Rob Crittenden -
Sean McLennan