Hi,
On Wed, Sep 28, 2022 at 2:17 AM TomK via FreeIPA-users
<freeipa-users(a)lists.fedorahosted.org> wrote:
On 2022-09-26 9:13 a.m., TomK via FreeIPA-users wrote:
> On 2022-09-26 8:50 a.m., Rob Crittenden via FreeIPA-users wrote:
>> TomK via FreeIPA-users wrote:
>>> On 2022-09-25 12:42 a.m., TomK via FreeIPA-users wrote:
>>>> On 2022-09-25 12:38 a.m., TomK via FreeIPA-users wrote:
>>>>> Hey Everyone!
>>>>>
>>>>> Wondering if anyone could help nudge me along in the right
direction
>>>>> on this one. Getting the following on my FreeIPA master and
replica:
>>>>>
>>>>> Internal Database Error encountered: Could not connect to
LDAP server
>>>>> host idmipa01.nix.mds.xyz <
http://idmipa01.nix.mds.xyz>
port
636 Error netscape.ldap.LDAPException:
>>>>> Authentication failed (48)
>>>>>
>>>>> Internal Database Error encountered: Could not connect to
LDAP server
>>>>> host idmipa02.nix.mds.xyz <
http://idmipa02.nix.mds.xyz>
port
636 Error netscape.ldap.LDAPException:
>>>>> Authentication failed (48)
>>>>>
>>>>> These appeared after some power outages occurred 2-3 times
and both
>>>>> hosts were affected. Went over a few pages online to try to
get to
>>>>> the bottom of these errors on these VM's however no luck so
far:
>>>>>
>>>>>
>>>>>
https://access.redhat.com/solutions/3081821
>>>>>
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
>>>>>
>>>>>
>>>>> and about a dozen other pages with little luck.
>>>>>
>>>>>
>>>>> Here's what I tried. First, wanted to and did kick off the
following
>>>>> on idmipa02:
>>>>>
>>>>> ipa-cacert-manage renew
>>>>>
>>>>> I've read on a few posts that command will cause the
running
server
>>>>> to become the renewal master, so was cautious to check first:
>>>>>
>>>>> [idmipa01]
>>>>> # ipa config-show | grep 'IPA CA renewal master'
>>>>> IPA CA renewal master: idmipa02.nix.mds.xyz
<
http://idmipa02.nix.mds.xyz>
>>>>>
>>>>>
>>>>> [idmipa02]
>>>>> # ipa config-show | grep 'IPA CA renewal master'
>>>>> IPA CA renewal master: idmipa02.nix.mds.xyz
<
http://idmipa02.nix.mds.xyz>
>>>>>
>>>>>
>>>>> Checked the certs and indeed the serial was different:
>>>>>
>>>>> # ldapsearch -D 'cn=directory manager' -W -b
>>>>> uid=pkidbuser,ou=people,o=ipaca
>>>>> Enter LDAP Password:
>>>>> # extended LDIF
>>>>> #
>>>>> # LDAPv3
>>>>> # base <uid=pkidbuser,ou=people,o=ipaca> with scope
subtree
>>>>> # filter: (objectclass=*)
>>>>> # requesting: ALL
>>>>> #
>>>>>
>>>>> # pkidbuser, people, ipaca
>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>> userPassword::
>>>>> e1NTSEE1MTJ9NUs3N......................................g4
>>>>> description: 2;26;CN=Certificate Authority,O=NIX.MDS.XYZ
<
http://NIX.MDS.XYZ>;CN=CA
>>>>> Subsystem,O=NIX
>>>>> .MDS.XYZ <
http://MDS.XYZ>
>>>>> seeAlso: CN=CA Subsystem,O=NIX.MDS.XYZ
<
http://NIX.MDS.XYZ>
>>>>> userCertificate::
>>>>> MIIDdjCCAl6............................IYL9mJQXhHIxpc=
>>>>> userCertificate::
>>>>> MIIDcTCCAlmgAwIBAg.........Mdr8SvD9uWfMPwUE4Tf2csf0z+Z
>>>>> userCertificate::
>>>>> MIIDcTCCAlmgA..............yShSmujM9PJrJPBBjLmTCIle9Xl
>>>>> userCertificate::
>>>>> MIIDdDCCAlygAwIBAg......................cgDVlPYm3LmKk+
>>>>> userstate: 1
>>>>> usertype: agentType
>>>>> mail:
>>>>> cn: pkidbuser
>>>>> sn: pkidbuser
>>>>> uid: pkidbuser
>>>>> objectClass: top
>>>>> objectClass: person
>>>>> objectClass: organizationalPerson
>>>>> objectClass: inetOrgPerson
>>>>> objectClass: cmsuser
>>>>>
>>>>> # search result
>>>>> search: 2
>>>>> result: 0 Success
>>>>>
>>>>> # numResponses: 2
>>>>> # numEntries: 1
>>>>>
>>>>>
>>>>>
>>>>> # certutil -d /etc/pki/pki-tomcat/alias/ -L -n
'subsystemCert
>>>>> cert-pki-ca' -a
>>>>> -----BEGIN CERTIFICATE-----
>>>>>
MIIDdDC..........................................dJmcMKreZ7cgDVlPYm3LmKk+
>>>>>
>>>>> -----END CERTIFICATE-----
>>>>>
>>>>>
>>>>> # certutil -d /etc/pki/pki-tomcat/alias/ -L -n
'subsystemCert
>>>>> cert-pki-ca' |grep -i serial
>>>>> Serial Number: 268369925 (0xfff0005)
>>>>>
>>>>> So updated it using:
>>>>>
>>>>> ldapmodify -x -h localhost -p 389 -D "cn=Directory
Manager"
-W << EOF
>>>>> dn:uid=pkidbuser,ou=people,o=ipaca
>>>>> changetype: modify
>>>>> replace: description
>>>>> description: 2;268369925;CN=Certificate
Authority,O=NIX.MDS.XYZ <
http://NIX.MDS.XYZ>;CN=CA
>>>>> Subsystem,O=NIX.MDS.XYZ <
http://NIX.MDS.XYZ>
>>>>> EOF
>>>>>
>>>>>
>>>>> Then verified that only the serial changed (the cert was
already in
>>>>> the list anyway so did not need to change) by comparing the
before
>>>>> and after:
>>>>>
>>>>>
>>>>> # diff 1.txt 2.txt
>>>>> 11a12,13
>>>>>> description: 2;268369925;CN=Certificate
>>>>> Authority,O=NIX.MDS.XYZ <
http://NIX.MDS.XYZ>;CN=CA
Subsyste
>>>>>> m,O=NIX.MDS.XYZ <
http://NIX.MDS.XYZ>
>>>>> 14,15d15
>>>>> < description: 2;26;CN=Certificate Authority,O=NIX.MDS.XYZ
<
http://NIX.MDS.XYZ>;CN=CA
>>>>> Subsystem,O=NIX
>>>>> < .MDS.XYZ <
http://MDS.XYZ>
>>>>>
>>>>>
>>>>> Confirmed trust attributes are fine:
>>>>>
>>>>>
>>>>> certutil -d /etc/dirsrv/slapd-NIX-MDS-XYZ/ -L
>>>>>
>>>>> Certificate Nickname Trust Attributes
>>>>>
>>>>> SSL,S/MIME,JAR/XPI
>>>>>
>>>>> Server-Cert u,u,u
>>>>> NIX.MDS.XYZ <
http://NIX.MDS.XYZ> IPA CA CT,C,C
>>>>>
>>>>>
>>>>> Yet on restart on idmipa02, still the same issue:
>>>>>
>>>>>
>>>>> # ipactl restart
>>>>> Restarting Directory Service
>>>>> Restarting krb5kdc Service
>>>>> Restarting kadmin Service
>>>>> Restarting named Service
>>>>> Restarting httpd Service
>>>>> Restarting ipa-custodia Service
>>>>> Restarting ntpd Service
>>>>> Restarting pki-tomcatd Service
>>>>> Failed to restart pki-tomcatd Service
>>>>> Shutting down
>>>>> Hint: You can use --ignore-service-failure option for forced
start in
>>>>> case that a non-critical service failed
>>>>> Aborting ipactl
>>>>>
>>>>>
>>>>> I have dated snapshots of both servers however, they both
are with
>>>>> the above mentioned issue. These hosts were also offline for a
>>>>> couple of months meaning cert expiration could be an issue.
Likewise,
>>>>> I could have caused a slight mess myself trying various online
>>>>> solutions that don't always match 100%.
>>>>>
>>>>> In regards to the certificate expiration, below are the
expiration
>>>>> dates for various certs though admittedly, I can't be sure
of how
>>>>> impacting any of these dates are since I don't yet
understand the
>>>>> usage of each of these certs as much as I would like to,
which the
>>>>> exception of the subsystemCert:
>>>>>
>>>>> # getcert list|grep -Ei "expires|status|key pair
storage"
>>>>> status: CA_UNREACHABLE
>>>>> key pair storage:
>>>>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>>>>> cert-pki-ca',token='NSS Certificate DB',pin set
>>>>> expires: 2022-09-10 22:14:56 UTC
>>>>> status: CA_UNREACHABLE
>>>>> key pair storage:
>>>>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>>>>> cert-pki-ca',token='NSS Certificate DB',pin set
>>>>> expires: 2022-09-10 22:13:56 UTC
>>>>> status: CA_UNREACHABLE
>>>>> key pair storage:
>>>>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>>>>> cert-pki-ca',token='NSS Certificate DB',pin set
>>>>> expires: 2022-09-10 22:13:54 UTC
>>>>> status: MONITORING
>>>>> key pair storage:
>>>>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>>>>> cert-pki-ca',token='NSS Certificate DB',pin set
>>>>> expires: 2036-11-21 07:32:02 UTC
>>>>> status: CA_UNREACHABLE
>>>>> key pair storage:
>>>>> type=FILE,location='/var/lib/ipa/ra-agent.key'
>>>>> expires: 2022-09-21 22:13:57 UTC
>>>>> status: CA_UNREACHABLE
>>>>> key pair storage:
>>>>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>>>>> cert-pki-ca',token='NSS Certificate DB',pin set
>>>>> expires: 2022-08-27 17:23:10 UTC
>>>>> status: CA_UNREACHABLE
>>>>> key pair storage:
>>>>>
type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS
>>>>> Certificate
DB',pinfile='/etc/dirsrv/slapd-NIX-MDS-XYZ/pwdfile.txt'
>>>>> expires: 2022-09-29 17:22:58 UTC
>>>>> status: CA_UNREACHABLE
>>>>> key pair storage:
>>>>>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>>>>> Certificate
DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>>>>> expires: 2022-09-29 17:22:45 UTC
>>>>> status: MONITORING
>>>>> key pair storage:
>>>>> type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>>>>> expires: 2023-09-25 02:17:17 UTC
>>>>>
>>>>> Both hosts are reachable from each other. Verified a couple
of ports
>>>>> to be sure. F/W is off on both, for the moment and both
hosts exist
>>>>> on the same VLAN.
>>>>>
>>>>>
>>>>
>>>> FreeIPA Version:
>>>>
>>>> # ipa --version
>>>> VERSION: 4.6.6, API_VERSION: 2.231
>>>>
>>>> Plus the pki-tomcat debug log entry on restart:
>>>>
>>>>
>>>> # tail -f /var/log/pki/pki-tomcat/ca/debug -n 100
>>>>
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> ============================================
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: ===== DEBUG
SUBSYSTEM
>>>> INITIALIZED =======
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> ============================================
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
restart at
>>>> autoShutdown? false
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
autoShutdown
>>>> crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
about to
>>>> look for cert for auto-shutdown support:auditSigningCert
cert-pki-ca
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: found
>>>> cert:auditSigningCert cert-pki-ca
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
done init
>>>> id=debug
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
initialized
>>>> debug
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
>>>> initSubsystem id=log
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
ready to
>>>> init id=log
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Event filters:
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Creating
>>>>
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Event filters:
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Creating
>>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Event filters:
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Creating
>>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
restart at
>>>> autoShutdown? false
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
autoShutdown
>>>> crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
about to
>>>> look for cert for auto-shutdown support:auditSigningCert
cert-pki-ca
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: found
>>>> cert:auditSigningCert cert-pki-ca
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
done init
>>>> id=log
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
>>>> initialized log
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
>>>> initSubsystem id=jss
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
ready to
>>>> init id=jss
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem:
>>>> initializing JSS subsystem
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem:
enabled:
>>>> true
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: NSS
>>>> database: /var/lib/pki/pki-tomcat/alias/
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem:
>>>> initializing CryptoManager
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem:
>>>> initializing SSL
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem:
random:
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: -
>>>> algorithm: pkcs11prng
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem: -
>>>> provider: Mozilla-JSS
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: JssSubsystem:
>>>> initialization complete
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
restart at
>>>> autoShutdown? false
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
autoShutdown
>>>> crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
about to
>>>> look for cert for auto-shutdown support:auditSigningCert
cert-pki-ca
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine: found
>>>> cert:auditSigningCert cert-pki-ca
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
done init
>>>> id=jss
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
>>>> initialized jss
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
>>>> initSubsystem id=dbs
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMSEngine:
ready to
>>>> init id=dbs
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: DBSubsystem:
init()
>>>> mEnableSerialMgmt=true
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Creating
>>>> LdapBoundConnFactor(DBSubsystem)
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
LdapBoundConnFactory:
>>>> init
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> LdapBoundConnFactory:doCloning true
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: LdapAuthInfo:
init()
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: LdapAuthInfo:
init
>>>> begins
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: LdapAuthInfo:
init ends
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: init: before
>>>> makeConnection errorIfDown is true
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: makeConnection:
>>>> errorIfDown true
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: TCP
Keep-Alive: true
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> ldapconn/PKISocketFactory.makeSocket: begins
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> ldapconn/PKISocketFactory.makeSSLSocket: begins
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> SSLClientCertificateSelectionCB: Setting desired cert
nickname to:
>>>> subsystemCert cert-pki-ca
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> ldapconn/PKISocketFactory.makeSSLSocket: set client auth cert
>>>> nickname subsystemCert cert-pki-ca
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> SSLClientCertificatSelectionCB: Entering!
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: Candidate cert:
>>>> caSigningCert cert-pki-ca
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> SSLClientCertificateSelectionCB: returning: null
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> PKIClientSocketListener.handshakeCompleted: begins
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: SignedAuditLogger:
>>>> event CLIENT_ACCESS_SESSION_ESTABLISH
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: LogFile: event
type not
>>>> selected: CLIENT_ACCESS_SESSION_ESTABLISH
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> PKIClientSocketListener.handshakeCompleted:
>>>> CS_CLIENT_ACCESS_SESSION_ESTABLISH_SUCCESS
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
>>>> PKIClientSocketListener.handshakeCompleted: clientIP=192.168.0.45
>>>> serverIP=192.168.0.45 serverPort=31746
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: SSL handshake
happened
>>>> Could not connect to LDAP server host idmipa02.nix.mds.xyz
<
http://idmipa02.nix.mds.xyz> port 636
>>>> Error netscape.ldap.LDAPException: Authentication failed (48)
>>>> at
>>>>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
>>>>
>>>> at
>>>>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
>>>>
>>>> at
>>>>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
>>>>
>>>> at
>>>> com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:667)
>>>> at
>>>>
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1056)
>>>> at
>>>>
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:962)
>>>> at
>>>> com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:568)
>>>> at com.netscape.certsrv.apps.CMS.init(CMS.java:191)
>>>> at com.netscape.certsrv.apps.CMS.start(CMS.java:1458)
>>>> at
>>>>
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
>>>>
>>>> at
javax.servlet.GenericServlet.init(GenericServlet.java:158)
>>>> at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>
>>>> at
>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at
>>>>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
>>>> at
>>>>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
>>>> at
java.security.AccessController.doPrivileged(Native Method)
>>>> at
>>>> javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
>>>> at
>>>>
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
>>>> at
>>>>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
>>>>
>>>> at
>>>>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
>>>>
>>>> at
>>>>
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218)
>>>>
>>>> at
>>>>
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)
>>>>
>>>> at
>>>>
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)
>>>> at
>>>>
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377)
>>>>
>>>> at
>>>>
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669)
>>>>
>>>> at
>>>>
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
>>>> at
>>>>
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
>>>>
>>>> at
>>>>
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
>>>> at
>>>>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
>>>>
>>>> at
>>>>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
>>>>
>>>> at
java.security.AccessController.doPrivileged(Native Method)
>>>> at
>>>>
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
>>>> at
>>>>
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
>>>> at
>>>>
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
>>>>
>>>> at
>>>>
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
>>>>
>>>> at
>>>>
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Internal Database Error encountered: Could not connect to
LDAP server
>>>> host idmipa02.nix.mds.xyz <
http://idmipa02.nix.mds.xyz> port
636 Error netscape.ldap.LDAPException:
>>>> Authentication failed (48)
>>>> at
>>>> com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689)
>>>> at
>>>>
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1056)
>>>> at
>>>>
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:962)
>>>> at
>>>> com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:568)
>>>> at com.netscape.certsrv.apps.CMS.init(CMS.java:191)
>>>> at com.netscape.certsrv.apps.CMS.start(CMS.java:1458)
>>>> at
>>>>
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
>>>>
>>>> at
javax.servlet.GenericServlet.init(GenericServlet.java:158)
>>>> at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>
>>>> at
>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at
>>>>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
>>>> at
>>>>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
>>>> at
java.security.AccessController.doPrivileged(Native Method)
>>>> at
>>>> javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
>>>> at
>>>>
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
>>>> at
>>>>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
>>>>
>>>> at
>>>>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
>>>>
>>>> at
>>>>
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218)
>>>>
>>>> at
>>>>
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)
>>>>
>>>> at
>>>>
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)
>>>> at
>>>>
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377)
>>>>
>>>> at
>>>>
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669)
>>>>
>>>> at
>>>>
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
>>>> at
>>>>
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
>>>>
>>>> at
>>>>
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
>>>> at
>>>>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
>>>>
>>>> at
>>>>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
>>>>
>>>> at
java.security.AccessController.doPrivileged(Native Method)
>>>> at
>>>>
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
>>>> at
>>>>
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
>>>> at
>>>>
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
>>>>
>>>> at
>>>>
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
>>>>
>>>> at
>>>>
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]: CMS.start():
shutdown
>>>> server
>>>> [25/Sep/2022:00:05:28][localhost-startStop-1]:
CMSEngine.shutdown()
>>>>
>>>>
>>>
>>> Decided to start fresh and work off of idmipa01 (first host
and the
>>> master) instead.
>>>
>>> Eventually I got success (I'll write up a more detailed
procedure in the
>>> next few days from all the RH and FLo's and Fraser's blogs ):
>>>
>>> getcert list|grep -Ei "Request ID|status:|stuck:|expires"
>>> Request ID '20180122053031':
>>> status: MONITORING
>>> stuck: no
>>> expires: 2024-09-15 05:15:58 UTC
>>> Request ID '20180122053032':
>>> status: MONITORING
>>> stuck: no
>>> expires: 2024-09-15 05:09:34 UTC
>>> Request ID '20180122053033':
>>> status: MONITORING
>>> stuck: no
>>> expires: 2024-09-15 05:14:47 UTC
>>> Request ID '20180122053034':
>>> status: MONITORING
>>> stuck: no
>>> expires: 2042-09-11 09:07:22 UTC
>>> Request ID '20180122053035':
>>> status: MONITORING
>>> stuck: no
>>> expires: 2024-08-31 09:03:44 UTC
>>> Request ID '20180122053036':
>>> status: MONITORING
>>> stuck: no
>>> expires: 2024-08-31 09:03:43 UTC
>>> Request ID '20180122053037':
>>> status: MONITORING
>>> stuck: no
>>> expires: 2024-09-26 05:16:52 UTC
>>> Request ID '20180122053042':
>>> status: MONITORING
>>> stuck: no
>>> expires: 2024-09-26 05:16:38 UTC
>>> Request ID '20180122053135':
>>> status: MONITORING
>>> stuck: no
>>> expires: 2023-09-26 00:54:45 UTC
>>>
>>> My question is now how do I replciate to the secondary master
or would I
>>> have to regenerate all certs there?
>>>
>>> # ipa-replica-manage list -v
>>> idmipa01.nix.mds.xyz <
http://idmipa01.nix.mds.xyz>: master
>>> idmipa02.nix.mds.xyz <
http://idmipa02.nix.mds.xyz>: master
>>>
>>> # ipa-replica-manage list -v idmipa02.nix.mds.xyz
<
http://idmipa02.nix.mds.xyz>
>>> idmipa01.nix.mds.xyz <
http://idmipa01.nix.mds.xyz>: replica
>>> last update status: Error (18) Replication error acquiring
replica:
>>> Incremental update transient warning. Backing off, will retry
update
>>> later. (transient warning)
>>> last update ended: 1970-01-01 00:00:00+00:00
>>>
>>> # ipa-replica-manage list -v idmipa01.nix.mds.xyz
<
http://idmipa01.nix.mds.xyz>
>>> idmipa02.nix.mds.xyz <
http://idmipa02.nix.mds.xyz>: replica
>>> last update status: Error (0) Replica acquired successfully:
>>> Incremental update succeeded
>>> last update ended: 2022-09-26 05:40:34+00:00
>>>
>>
>> You need to get the replication issue resolved first. It may
come down
>> to re-initializing 02 from 01.
>>
>> The CA uses the same certificates, minus Server-Cert
cert-pki-ca, on its
>> clones so there is no re-generating them per-server.
>>
>> rob
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to
>> freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct:
>>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>> Do not reply to spam, report it:
>>
https://pagure.io/fedora-infrastructure/new_issue
>
>
> And that is exactly what I did to get it working and all synced up.
> Seems I'm ready for an upgrade. >
Spoke too soon it seems :(
[idmipa01] - (Primary) Master
# getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053031':
status: MONITORING
stuck: no
expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053032':
status: MONITORING
stuck: no
expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053033':
status: MONITORING
stuck: no
expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053034':
status: MONITORING
stuck: no
expires: 2042-09-11 09:07:22 UTC
Request ID '20180122053035':
status: MONITORING
stuck: no
expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053036':
status: MONITORING
stuck: no
expires: 2024-08-31 09:03:43 UTC
Request ID '20180122053037':
status: MONITORING
stuck: no
expires: 2024-09-26 05:16:52 UTC
Request ID '20180122053042':
status: MONITORING
stuck: no
expires: 2024-09-26 05:16:38 UTC
Request ID '20180122053135':
status: MONITORING
stuck: no
expires: 2023-09-26 00:54:45 UTC
# ipa config-show | grep 'IPA CA renewal master'
IPA CA renewal master: idmipa01.nix.mds.xyz
<
http://idmipa01.nix.mds.xyz>
[idmipa02] - (Secondary) Master
getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053638':
status: MONITORING
stuck: no
expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053639':
status: MONITORING
stuck: no
expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053640':
status: MONITORING
stuck: no
expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053641':
status: MONITORING
stuck: no
expires: 2036-11-21 07:32:02 UTC
Request ID '20180122053642':
status: MONITORING
stuck: no
expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053643':
status: CA_UNREACHABLE
stuck: no
expires: 2022-08-27 17:23:10 UTC
Request ID '20180122053644':
status: CA_UNREACHABLE
stuck: no
expires: 2022-09-29 17:22:58 UTC
Request ID '20180122053649':
status: CA_UNREACHABLE
stuck: no
expires: 2022-09-29 17:22:45 UTC
Request ID '20180122053742':
status: MONITORING
stuck: no
expires: 2023-09-26 00:54:54 UTC
# ipa config-show | grep 'IPA CA renewal master'
IPA CA renewal master: idmipa01.nix.mds.xyz
<
http://idmipa01.nix.mds.xyz>
Which commands can I safely run on idmipa02 to resolve the above?
ipa-cacert-manage renew # Definitely not as it will switch
the CA
renewal master, and apparently updates only the CA.
ipa-certupdate # Ran this, no luck.
This got me a bit further on idmipa02:
getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053638':
status: MONITORING
stuck: no
expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053639':
status: MONITORING
stuck: no
expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053640':
status: MONITORING
stuck: no
expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053641':
status: MONITORING
stuck: no
expires: 2036-11-21 07:32:02 UTC
Request ID '20180122053642':
status: MONITORING
stuck: no
expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053643':
status: CA_UNREACHABLE
stuck: no
expires: 2022-08-27 17:23:10 UTC
Request ID '20180122053644':
status: MONITORING
stuck: no
expires: 2024-09-27 23:42:10 UTC
Request ID '20180122053649':
status: MONITORING
stuck: no
expires: 2024-09-27 23:41:58 UTC
Request ID '20180122053742':
status: MONITORING
stuck: no
expires: 2023-09-26 00:54:54 UTC
This left me with only one issue on this host:
Request ID '20180122053643':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to
https://idmipa02.nix.mds.xyz:8443/ca/agent/ca/profileReview: Couldn't
connect to server.
Which I thought was due to the pki-tomcat being offline:
# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
Started it up:
# ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting smb Service
Restarting winbind Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
Which changed the error to:
Request ID '20180122053643':
status: CA_UNREACHABLE
ca-error: Error 60 connecting to
https://idmipa02.nix.mds.xyz:8443/ca/agent/ca/profileReview: Peer
certificate cannot be authenticated with given CA certificates.
On idmipa02 I receive:
# ipa ca-show ipa
ipa: ERROR: Failed to authenticate to CA REST API
Not so on idmipa01:
# ipa ca-show ipa
Name: ipa
Description: IPA CA
Authority ID: 338e27c3-3325-4b89-9a39-8ad7fd3f01b7
Subject DN: CN=Certificate Authority,O=NIX.MDS.XYZ
<
http://NIX.MDS.XYZ>
Issuer DN: CN=Certificate Authority,O=NIX.MDS.XYZ
<
http://NIX.MDS.XYZ>
Certificate: MIIDizCCAnOg...................6AzRwvlw=
Could you please let me know what is the right course of action here?
From your output it's difficult to know which cert is expired but I
guess it is the Server-Cert cert-pki-ca in /etc/pki/pki-tomcat/alias.
If you have ipa 4.6.6+, you can use the ipa-cert-fix command. It is
documented here:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/...
Before launching the command, backup your NSS DBs. You will need to
run the command on idmipa02 only (as I understand that idmipa01 is
already fixed and fully working), and note that the command will also
switch the CA master role to idmipa02. You may switch it back to
idmipa01 later if you want.
Tried it last night and it did fix the cert without switching the
renewal master to idmipa02 however:
# getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053638':
status: MONITORING
stuck: no
expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053639':
status: MONITORING
stuck: no
expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053640':
status: MONITORING
stuck: no
expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053641':
status: MONITORING
stuck: no
expires: 2042-09-11 09:07:22 UTC
Request ID '20180122053642':
status: MONITORING
stuck: no
expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053643':
status: MONITORING
stuck: no
expires: 2024-09-15 05:46:41 UTC
Request ID '20180122053644':
status: MONITORING
stuck: no
expires: 2024-09-27 23:42:10 UTC
Request ID '20180122053649':
status: MONITORING
stuck: no
expires: 2024-09-27 23:41:58 UTC
Request ID '20180122053742':
status: MONITORING
stuck: no
expires: 2023-09-26 00:54:54 UTC
# ipa config-show | grep 'IPA CA renewal master'
IPA CA renewal master: idmipa01.nix.mds.xyz
Decided to take a chance nontheless. Before going forward though I did
have snapshots of the VM.
Either way, I did end up writing up all the steps I took. Attached as
txt as well . Also made available online.