[Freeipa-users] Re: Internal Database Error encountered: Could not connect to LDAP server host idmipa02.nix.mds.xyz port 636 Error netscape.ldap.LDAPException: Authentication failed (48)

Hi, On Wed, Sep 28, 2022 at 2:17 AM TomK via FreeIPA-users <freeipa-users(a)lists.fedorahosted.org&gt; wrote: On 2022-09-26 9:13 a.m., TomK via FreeIPA-users wrote: > On 2022-09-26 8:50 a.m., Rob Crittenden via FreeIPA-users wrote: >> TomK via FreeIPA-users wrote: >>> On 2022-09-25 12:42 a.m., TomK via FreeIPA-users wrote: >>>> On 2022-09-25 12:38 a.m., TomK via FreeIPA-users wrote: >>>>> Hey Everyone! >>>>> >>>>> Wondering if anyone could help nudge me along in the right direction >>>>> on this one.  Getting the following on my FreeIPA master and replica: >>>>> >>>>> Internal Database Error encountered: Could not connect to LDAP server >>>>> host idmipa01.nix.mds.xyz <http://idmipa01.nix.mds.xyz> port 636 Error netscape.ldap.LDAPException: >>>>> Authentication failed (48) >>>>> >>>>> Internal Database Error encountered: Could not connect to LDAP server >>>>> host idmipa02.nix.mds.xyz <http://idmipa02.nix.mds.xyz> port 636 Error netscape.ldap.LDAPException: >>>>> Authentication failed (48) >>>>> >>>>> These appeared after some power outages occurred 2-3 times and both >>>>> hosts were affected.  Went over a few pages online to try to get to >>>>> the bottom of these errors on these VM's however no luck so far: >>>>> >>>>> >>>>> https://access.redhat.com/solutions/3081821 >>>>> https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom... >>>>> >>>>> >>>>> and about a dozen other pages with little luck. >>>>> >>>>> >>>>> Here's what I tried. First, wanted to and did kick off the following >>>>> on idmipa02: >>>>> >>>>> ipa-cacert-manage renew >>>>> >>>>> I've read on a few posts that command will cause the running server >>>>> to become the renewal master, so was cautious to check first: >>>>> >>>>> [idmipa01] >>>>> # ipa config-show | grep 'IPA CA renewal master' >>>>>    IPA CA renewal master: idmipa02.nix.mds.xyz <http://idmipa02.nix.mds.xyz> >>>>> >>>>> >>>>> [idmipa02] >>>>> # ipa config-show | grep 'IPA CA renewal master' >>>>>    IPA CA renewal master: idmipa02.nix.mds.xyz <http://idmipa02.nix.mds.xyz> >>>>> >>>>> >>>>> Checked the certs and indeed the serial was different: >>>>> >>>>> # ldapsearch -D 'cn=directory manager' -W -b >>>>> uid=pkidbuser,ou=people,o=ipaca >>>>> Enter LDAP Password: >>>>> # extended LDIF >>>>> # >>>>> # LDAPv3 >>>>> # base <uid=pkidbuser,ou=people,o=ipaca> with scope subtree >>>>> # filter: (objectclass=*) >>>>> # requesting: ALL >>>>> # >>>>> >>>>> # pkidbuser, people, ipaca >>>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>>> userPassword:: >>>>> e1NTSEE1MTJ9NUs3N......................................g4 >>>>> description: 2;26;CN=Certificate Authority,O=NIX.MDS.XYZ <http://NIX.MDS.XYZ>;CN=CA >>>>> Subsystem,O=NIX >>>>>   .MDS.XYZ <http://MDS.XYZ> >>>>> seeAlso: CN=CA Subsystem,O=NIX.MDS.XYZ <http://NIX.MDS.XYZ> >>>>> userCertificate:: >>>>> MIIDdjCCAl6............................IYL9mJQXhHIxpc= >>>>> userCertificate:: >>>>> MIIDcTCCAlmgAwIBAg.........Mdr8SvD9uWfMPwUE4Tf2csf0z+Z >>>>> userCertificate:: >>>>> MIIDcTCCAlmgA..............yShSmujM9PJrJPBBjLmTCIle9Xl >>>>> userCertificate:: >>>>> MIIDdDCCAlygAwIBAg......................cgDVlPYm3LmKk+ >>>>> userstate: 1 >>>>> usertype: agentType >>>>> mail: >>>>> cn: pkidbuser >>>>> sn: pkidbuser >>>>> uid: pkidbuser >>>>> objectClass: top >>>>> objectClass: person >>>>> objectClass: organizationalPerson >>>>> objectClass: inetOrgPerson >>>>> objectClass: cmsuser >>>>> >>>>> # search result >>>>> search: 2 >>>>> result: 0 Success >>>>> >>>>> # numResponses: 2 >>>>> # numEntries: 1 >>>>> >>>>> >>>>> >>>>> # certutil -d /etc/pki/pki-tomcat/alias/ -L -n 'subsystemCert >>>>> cert-pki-ca' -a >>>>> -----BEGIN CERTIFICATE----- >>>>> MIIDdDC..........................................dJmcMKreZ7cgDVlPYm3LmKk+ >>>>> >>>>> -----END CERTIFICATE----- >>>>> >>>>> >>>>> # certutil -d /etc/pki/pki-tomcat/alias/ -L -n 'subsystemCert >>>>> cert-pki-ca' |grep -i serial >>>>>          Serial Number: 268369925 (0xfff0005) >>>>> >>>>> So updated it using: >>>>> >>>>> ldapmodify -x -h localhost -p 389 -D "cn=Directory Manager" -W << EOF >>>>> dn:uid=pkidbuser,ou=people,o=ipaca >>>>> changetype: modify >>>>> replace: description >>>>> description: 2;268369925;CN=Certificate Authority,O=NIX.MDS.XYZ <http://NIX.MDS.XYZ>;CN=CA >>>>> Subsystem,O=NIX.MDS.XYZ <http://NIX.MDS.XYZ> >>>>> EOF >>>>> >>>>> >>>>> Then verified that only the serial changed (the cert was already in >>>>> the list anyway so did not need to change) by comparing the before >>>>> and after: >>>>> >>>>> >>>>> # diff 1.txt 2.txt >>>>> 11a12,13 >>>>>> description: 2;268369925;CN=Certificate >>>>> Authority,O=NIX.MDS.XYZ <http://NIX.MDS.XYZ>;CN=CA Subsyste >>>>>>    m,O=NIX.MDS.XYZ <http://NIX.MDS.XYZ> >>>>> 14,15d15 >>>>> < description: 2;26;CN=Certificate Authority,O=NIX.MDS.XYZ <http://NIX.MDS.XYZ>;CN=CA >>>>> Subsystem,O=NIX >>>>> <  .MDS.XYZ <http://MDS.XYZ> >>>>> >>>>> >>>>> Confirmed trust attributes are fine: >>>>> >>>>> >>>>> certutil -d /etc/dirsrv/slapd-NIX-MDS-XYZ/ -L >>>>> >>>>> Certificate Nickname Trust Attributes >>>>> >>>>> SSL,S/MIME,JAR/XPI >>>>> >>>>> Server-Cert u,u,u >>>>> NIX.MDS.XYZ <http://NIX.MDS.XYZ> IPA CA CT,C,C >>>>> >>>>> >>>>> Yet on restart on idmipa02, still the same issue: >>>>> >>>>> >>>>> # ipactl restart >>>>> Restarting Directory Service >>>>> Restarting krb5kdc Service >>>>> Restarting kadmin Service >>>>> Restarting named Service >>>>> Restarting httpd Service >>>>> Restarting ipa-custodia Service >>>>> Restarting ntpd Service >>>>> Restarting pki-tomcatd Service >>>>> Failed to restart pki-tomcatd Service >>>>> Shutting down >>>>> Hint: You can use --ignore-service-failure option for forced start in >>>>> case that a non-critical service failed >>>>> Aborting ipactl >>>>> >>>>> >>>>> I have dated snapshots of both servers however, they both are with >>>>> the above mentioned issue.  These hosts were also offline for a >>>>> couple of months meaning cert expiration could be an issue. Likewise, >>>>> I could have caused a slight mess myself trying various online >>>>> solutions that don't always match 100%. >>>>> >>>>> In regards to the certificate expiration, below are the expiration >>>>> dates for various certs though admittedly, I can't be sure of how >>>>> impacting any of these dates are since I don't yet understand the >>>>> usage of each of these certs as much as I would like to, which the >>>>> exception of the subsystemCert: >>>>> >>>>> # getcert list|grep -Ei "expires|status|key pair storage" >>>>>          status: CA_UNREACHABLE >>>>>          key pair storage: >>>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >>>>> cert-pki-ca',token='NSS Certificate DB',pin set >>>>>          expires: 2022-09-10 22:14:56 UTC >>>>>          status: CA_UNREACHABLE >>>>>          key pair storage: >>>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert >>>>> cert-pki-ca',token='NSS Certificate DB',pin set >>>>>          expires: 2022-09-10 22:13:56 UTC >>>>>          status: CA_UNREACHABLE >>>>>          key pair storage: >>>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert >>>>> cert-pki-ca',token='NSS Certificate DB',pin set >>>>>          expires: 2022-09-10 22:13:54 UTC >>>>>          status: MONITORING >>>>>          key pair storage: >>>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert >>>>> cert-pki-ca',token='NSS Certificate DB',pin set >>>>>          expires: 2036-11-21 07:32:02 UTC >>>>>          status: CA_UNREACHABLE >>>>>          key pair storage: >>>>> type=FILE,location='/var/lib/ipa/ra-agent.key' >>>>>          expires: 2022-09-21 22:13:57 UTC >>>>>          status: CA_UNREACHABLE >>>>>          key pair storage: >>>>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert >>>>> cert-pki-ca',token='NSS Certificate DB',pin set >>>>>          expires: 2022-08-27 17:23:10 UTC >>>>>          status: CA_UNREACHABLE >>>>>          key pair storage: >>>>> type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS >>>>> Certificate DB',pinfile='/etc/dirsrv/slapd-NIX-MDS-XYZ/pwdfile.txt' >>>>>          expires: 2022-09-29 17:22:58 UTC >>>>>          status: CA_UNREACHABLE >>>>>          key pair storage: >>>>> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS >>>>> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>>>>          expires: 2022-09-29 17:22:45 UTC >>>>>          status: MONITORING >>>>>          key pair storage: >>>>> type=FILE,location='/var/kerberos/krb5kdc/kdc.key' >>>>>          expires: 2023-09-25 02:17:17 UTC >>>>> >>>>> Both hosts are reachable from each other.  Verified a couple of ports >>>>> to be sure. F/W is off on both, for the moment and both hosts exist >>>>> on the same VLAN. >>>>> >>>>> >>>> >>>> FreeIPA Version: >>>> >>>> # ipa --version >>>> VERSION: 4.6.6, API_VERSION: 2.231 >>>> >>>> Plus the pki-tomcat debug log entry on restart: >>>> >>>> >>>> # tail -f /var/log/pki/pki-tomcat/ca/debug -n 100 >>>> >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> ============================================ >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: =====  DEBUG SUBSYSTEM >>>> INITIALIZED   ======= >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> ============================================ >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: restart at >>>> autoShutdown? false >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: autoShutdown >>>> crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: about to >>>> look for cert for auto-shutdown support:auditSigningCert cert-pki-ca >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: found >>>> cert:auditSigningCert cert-pki-ca >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: done init >>>> id=debug >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: initialized >>>> debug >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: >>>> initSubsystem id=log >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: ready to >>>> init id=log >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Event filters: >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Creating >>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Event filters: >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Creating >>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Event filters: >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Creating >>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: restart at >>>> autoShutdown? false >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: autoShutdown >>>> crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: about to >>>> look for cert for auto-shutdown support:auditSigningCert cert-pki-ca >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: found >>>> cert:auditSigningCert cert-pki-ca >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: done init >>>> id=log >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: >>>> initialized log >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: >>>> initSubsystem id=jss >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: ready to >>>> init id=jss >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: >>>> initializing JSS subsystem >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: enabled: >>>> true >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: NSS >>>> database: /var/lib/pki/pki-tomcat/alias/ >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: >>>> initializing CryptoManager >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: >>>> initializing SSL >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: random: >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: - >>>> algorithm: pkcs11prng >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: - >>>> provider: Mozilla-JSS >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: >>>> initialization complete >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: restart at >>>> autoShutdown? false >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: autoShutdown >>>> crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: about to >>>> look for cert for auto-shutdown support:auditSigningCert cert-pki-ca >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: found >>>> cert:auditSigningCert cert-pki-ca >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: done init >>>> id=jss >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: >>>> initialized jss >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: >>>> initSubsystem id=dbs >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: ready to >>>> init id=dbs >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: DBSubsystem: init() >>>> mEnableSerialMgmt=true >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Creating >>>> LdapBoundConnFactor(DBSubsystem) >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: LdapBoundConnFactory: >>>> init >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> LdapBoundConnFactory:doCloning true >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: LdapAuthInfo: init() >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: LdapAuthInfo: init >>>> begins >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: LdapAuthInfo: init ends >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: init: before >>>> makeConnection errorIfDown is true >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: makeConnection: >>>> errorIfDown true >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: TCP Keep-Alive: true >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> ldapconn/PKISocketFactory.makeSocket: begins >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> ldapconn/PKISocketFactory.makeSSLSocket: begins >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> SSLClientCertificateSelectionCB: Setting desired cert nickname to: >>>> subsystemCert cert-pki-ca >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> ldapconn/PKISocketFactory.makeSSLSocket: set client auth cert >>>> nickname subsystemCert cert-pki-ca >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> SSLClientCertificatSelectionCB: Entering! >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Candidate cert: >>>> caSigningCert cert-pki-ca >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> SSLClientCertificateSelectionCB: returning: null >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> PKIClientSocketListener.handshakeCompleted: begins >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: SignedAuditLogger: >>>> event CLIENT_ACCESS_SESSION_ESTABLISH >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: LogFile: event type not >>>> selected: CLIENT_ACCESS_SESSION_ESTABLISH >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> PKIClientSocketListener.handshakeCompleted: >>>> CS_CLIENT_ACCESS_SESSION_ESTABLISH_SUCCESS >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: >>>> PKIClientSocketListener.handshakeCompleted: clientIP=192.168.0.45 >>>> serverIP=192.168.0.45 serverPort=31746 >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: SSL handshake happened >>>> Could not connect to LDAP server host idmipa02.nix.mds.xyz <http://idmipa02.nix.mds.xyz> port 636 >>>> Error netscape.ldap.LDAPException: Authentication failed (48) >>>>          at >>>> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) >>>> >>>>          at >>>> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) >>>> >>>>          at >>>> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) >>>> >>>>          at >>>> com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:667) >>>>          at >>>> com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1056) >>>>          at >>>> com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:962) >>>>          at >>>> com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:568) >>>>          at com.netscape.certsrv.apps.CMS.init(CMS.java:191) >>>>          at com.netscape.certsrv.apps.CMS.start(CMS.java:1458) >>>>          at >>>> com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) >>>> >>>>          at javax.servlet.GenericServlet.init(GenericServlet.java:158) >>>>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>          at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> >>>>          at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>>          at java.lang.reflect.Method.invoke(Method.java:498) >>>>          at >>>> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >>>>          at >>>> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >>>>          at java.security.AccessController.doPrivileged(Native Method) >>>>          at >>>> javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >>>>          at >>>> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >>>>          at >>>> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) >>>> >>>>          at >>>> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) >>>> >>>>          at >>>> org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218) >>>> >>>>          at >>>> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174) >>>> >>>>          at >>>> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066) >>>>          at >>>> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377) >>>> >>>>          at >>>> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669) >>>> >>>>          at >>>> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) >>>>          at >>>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >>>> >>>>          at >>>> org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) >>>>          at >>>> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) >>>> >>>>          at >>>> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) >>>> >>>>          at java.security.AccessController.doPrivileged(Native Method) >>>>          at >>>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) >>>>          at >>>> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) >>>>          at >>>> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) >>>> >>>>          at >>>> org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) >>>> >>>>          at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>          at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>          at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>> >>>>          at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>> >>>>          at java.lang.Thread.run(Thread.java:748) >>>> Internal Database Error encountered: Could not connect to LDAP server >>>> host idmipa02.nix.mds.xyz <http://idmipa02.nix.mds.xyz> port 636 Error netscape.ldap.LDAPException: >>>> Authentication failed (48) >>>>          at >>>> com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689) >>>>          at >>>> com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1056) >>>>          at >>>> com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:962) >>>>          at >>>> com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:568) >>>>          at com.netscape.certsrv.apps.CMS.init(CMS.java:191) >>>>          at com.netscape.certsrv.apps.CMS.start(CMS.java:1458) >>>>          at >>>> com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) >>>> >>>>          at javax.servlet.GenericServlet.init(GenericServlet.java:158) >>>>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>          at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> >>>>          at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>>          at java.lang.reflect.Method.invoke(Method.java:498) >>>>          at >>>> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >>>>          at >>>> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >>>>          at java.security.AccessController.doPrivileged(Native Method) >>>>          at >>>> javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >>>>          at >>>> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >>>>          at >>>> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) >>>> >>>>          at >>>> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) >>>> >>>>          at >>>> org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218) >>>> >>>>          at >>>> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174) >>>> >>>>          at >>>> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066) >>>>          at >>>> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377) >>>> >>>>          at >>>> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669) >>>> >>>>          at >>>> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) >>>>          at >>>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >>>> >>>>          at >>>> org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) >>>>          at >>>> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) >>>> >>>>          at >>>> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) >>>> >>>>          at java.security.AccessController.doPrivileged(Native Method) >>>>          at >>>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) >>>>          at >>>> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) >>>>          at >>>> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) >>>> >>>>          at >>>> org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) >>>> >>>>          at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>          at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>          at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>> >>>>          at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>> >>>>          at java.lang.Thread.run(Thread.java:748) >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMS.start(): shutdown >>>> server >>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine.shutdown() >>>> >>>> >>> >>> Decided to start fresh and work off of idmipa01 (first host and the >>> master) instead. >>> >>> Eventually I got success (I'll write up a more detailed procedure in the >>> next few days from all the RH and FLo's and Fraser's blogs ): >>> >>> getcert list|grep -Ei "Request ID|status:|stuck:|expires" >>> Request ID '20180122053031': >>>          status: MONITORING >>>          stuck: no >>>          expires: 2024-09-15 05:15:58 UTC >>> Request ID '20180122053032': >>>          status: MONITORING >>>          stuck: no >>>          expires: 2024-09-15 05:09:34 UTC >>> Request ID '20180122053033': >>>          status: MONITORING >>>          stuck: no >>>          expires: 2024-09-15 05:14:47 UTC >>> Request ID '20180122053034': >>>          status: MONITORING >>>          stuck: no >>>          expires: 2042-09-11 09:07:22 UTC >>> Request ID '20180122053035': >>>          status: MONITORING >>>          stuck: no >>>          expires: 2024-08-31 09:03:44 UTC >>> Request ID '20180122053036': >>>          status: MONITORING >>>          stuck: no >>>          expires: 2024-08-31 09:03:43 UTC >>> Request ID '20180122053037': >>>          status: MONITORING >>>          stuck: no >>>          expires: 2024-09-26 05:16:52 UTC >>> Request ID '20180122053042': >>>          status: MONITORING >>>          stuck: no >>>          expires: 2024-09-26 05:16:38 UTC >>> Request ID '20180122053135': >>>          status: MONITORING >>>          stuck: no >>>          expires: 2023-09-26 00:54:45 UTC >>> >>> My question is now how do I replciate to the secondary master or would I >>> have to regenerate all certs there? >>> >>> # ipa-replica-manage list -v >>> idmipa01.nix.mds.xyz <http://idmipa01.nix.mds.xyz>;: master >>> idmipa02.nix.mds.xyz <http://idmipa02.nix.mds.xyz>;: master >>> >>> # ipa-replica-manage list -v idmipa02.nix.mds.xyz <http://idmipa02.nix.mds.xyz> >>> idmipa01.nix.mds.xyz <http://idmipa01.nix.mds.xyz>;: replica >>>    last update status: Error (18) Replication error acquiring replica: >>> Incremental update transient warning.  Backing off, will retry update >>> later. (transient warning) >>>    last update ended: 1970-01-01 00:00:00+00:00 >>> >>> # ipa-replica-manage list -v idmipa01.nix.mds.xyz <http://idmipa01.nix.mds.xyz> >>> idmipa02.nix.mds.xyz <http://idmipa02.nix.mds.xyz>;: replica >>>    last update status: Error (0) Replica acquired successfully: >>> Incremental update succeeded >>>    last update ended: 2022-09-26 05:40:34+00:00 >>> >> >> You need to get the replication issue resolved first. It may come down >> to re-initializing 02 from 01. >> >> The CA uses the same certificates, minus Server-Cert cert-pki-ca, on its >> clones so there is no re-generating them per-server. >> >> rob >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org >> To unsubscribe send an email to >> freeipa-users-leave(a)lists.fedorahosted.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... >> Do not reply to spam, report it: >> https://pagure.io/fedora-infrastructure/new_issue > > > And that is exactly what I did to get it working and all synced up. > Seems I'm ready for an upgrade. > Spoke too soon it seems :( [idmipa01] - (Primary) Master # getcert list|grep -Ei "Request ID|status:|stuck:|expires" Request ID '20180122053031':          status: MONITORING          stuck: no          expires: 2024-09-15 05:15:58 UTC Request ID '20180122053032':          status: MONITORING          stuck: no          expires: 2024-09-15 05:09:34 UTC Request ID '20180122053033':          status: MONITORING          stuck: no          expires: 2024-09-15 05:14:47 UTC Request ID '20180122053034':          status: MONITORING          stuck: no          expires: 2042-09-11 09:07:22 UTC Request ID '20180122053035':          status: MONITORING          stuck: no          expires: 2024-08-31 09:03:44 UTC Request ID '20180122053036':          status: MONITORING          stuck: no          expires: 2024-08-31 09:03:43 UTC Request ID '20180122053037':          status: MONITORING          stuck: no          expires: 2024-09-26 05:16:52 UTC Request ID '20180122053042':          status: MONITORING          stuck: no          expires: 2024-09-26 05:16:38 UTC Request ID '20180122053135':          status: MONITORING          stuck: no          expires: 2023-09-26 00:54:45 UTC # ipa config-show | grep 'IPA CA renewal master'    IPA CA renewal master: idmipa01.nix.mds.xyz <http://idmipa01.nix.mds.xyz> [idmipa02] - (Secondary) Master getcert list|grep -Ei "Request ID|status:|stuck:|expires" Request ID '20180122053638':          status: MONITORING          stuck: no          expires: 2024-09-15 05:15:58 UTC Request ID '20180122053639':          status: MONITORING          stuck: no          expires: 2024-09-15 05:09:34 UTC Request ID '20180122053640':          status: MONITORING          stuck: no          expires: 2024-09-15 05:14:47 UTC Request ID '20180122053641':          status: MONITORING          stuck: no          expires: 2036-11-21 07:32:02 UTC Request ID '20180122053642':          status: MONITORING          stuck: no          expires: 2024-08-31 09:03:44 UTC Request ID '20180122053643':          status: CA_UNREACHABLE          stuck: no          expires: 2022-08-27 17:23:10 UTC Request ID '20180122053644':          status: CA_UNREACHABLE          stuck: no          expires: 2022-09-29 17:22:58 UTC Request ID '20180122053649':          status: CA_UNREACHABLE          stuck: no          expires: 2022-09-29 17:22:45 UTC Request ID '20180122053742':          status: MONITORING          stuck: no          expires: 2023-09-26 00:54:54 UTC # ipa config-show | grep 'IPA CA renewal master'    IPA CA renewal master: idmipa01.nix.mds.xyz <http://idmipa01.nix.mds.xyz> Which commands can I safely run on idmipa02 to resolve the above? ipa-cacert-manage renew         # Definitely not as it will switch the CA renewal master, and apparently updates only the CA. ipa-certupdate                  # Ran this, no luck. This got me a bit further on idmipa02: getcert list|grep -Ei "Request ID|status:|stuck:|expires" Request ID '20180122053638':          status: MONITORING          stuck: no          expires: 2024-09-15 05:15:58 UTC Request ID '20180122053639':          status: MONITORING          stuck: no          expires: 2024-09-15 05:09:34 UTC Request ID '20180122053640':          status: MONITORING          stuck: no          expires: 2024-09-15 05:14:47 UTC Request ID '20180122053641':          status: MONITORING          stuck: no          expires: 2036-11-21 07:32:02 UTC Request ID '20180122053642':          status: MONITORING          stuck: no          expires: 2024-08-31 09:03:44 UTC Request ID '20180122053643':          status: CA_UNREACHABLE          stuck: no          expires: 2022-08-27 17:23:10 UTC Request ID '20180122053644':          status: MONITORING          stuck: no          expires: 2024-09-27 23:42:10 UTC Request ID '20180122053649':          status: MONITORING          stuck: no          expires: 2024-09-27 23:41:58 UTC Request ID '20180122053742':          status: MONITORING          stuck: no          expires: 2023-09-26 00:54:54 UTC This left me with only one issue on this host: Request ID '20180122053643':          status: CA_UNREACHABLE          ca-error: Error 7 connecting to https://idmipa02.nix.mds.xyz:8443/ca/agent/ca/profileReview: Couldn't connect to server. Which I thought was due to the pki-tomcat being offline: # ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful Started it up: # ipactl restart Restarting Directory Service Restarting krb5kdc Service Restarting kadmin Service Restarting named Service Restarting httpd Service Restarting ipa-custodia Service Restarting ntpd Service Restarting pki-tomcatd Service Restarting smb Service Restarting winbind Service Restarting ipa-otpd Service Restarting ipa-dnskeysyncd Service ipa: INFO: The ipactl command was successful Which changed the error to: Request ID '20180122053643':          status: CA_UNREACHABLE          ca-error: Error 60 connecting to https://idmipa02.nix.mds.xyz:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. On idmipa02 I receive: # ipa ca-show ipa ipa: ERROR: Failed to authenticate to CA REST API Not so on idmipa01: # ipa ca-show ipa    Name: ipa    Description: IPA CA    Authority ID: 338e27c3-3325-4b89-9a39-8ad7fd3f01b7    Subject DN: CN=Certificate Authority,O=NIX.MDS.XYZ <http://NIX.MDS.XYZ>    Issuer DN: CN=Certificate Authority,O=NIX.MDS.XYZ <http://NIX.MDS.XYZ>    Certificate: MIIDizCCAnOg...................6AzRwvlw= Could you please let me know what is the right course of action here? From your output it's difficult to know which cert is expired but I guess it is the Server-Cert cert-pki-ca in /etc/pki/pki-tomcat/alias. If you have ipa 4.6.6+, you can use the ipa-cert-fix command. It is documented here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/... Before launching the command, backup your NSS DBs. You will need to run the command on idmipa02 only (as I understand that idmipa01 is already fixed and fully working), and note that the command will also switch the CA master role to idmipa02. You may switch it back to idmipa01 later if you want.

HTH, flo -- Thx, TK. _______________________________________________ FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue _______________________________________________ FreeIPA-users mailing list --freeipa-users(a)lists.fedorahosted.org To unsubscribe send an email tofreeipa-users-leave(a)lists.fedorahosted.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedorahosted.org/archives/list/freeipa-users@lists... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue