Hi Florence,
Done the same and tried installation for multiple times but same issue
Please find below response inline
Can you clean up the replica you're trying to install and start over, then send the most recent logs? Done
- on the failing replica: ipa-server-install --uninstall -U Done
- on the master: kinit admin; ipa server-del <replica> --force Done
- on the failing replica: perform the installation with your usual method (either in a 2-step process with ipa-client-install/ipa-replica-install or in a single step with ipa-replica-install). Done with below command
“ipa-replica-install -n ipa.subdomain.com --hostname=dirpav01.ipa.subdomain.com --server=aaa01.ipa.subdomain.com --realm=IPA.SUBDOMAIN.COM -P admin -w XXXXXXX --no-host-dns --setup-ca --setup-dns --mkhomedir --auto-reverse --no-forwarders”
-Also provide the timezone of the replica so that we can translate all the timestamps in UTC time.
4. Time Zone
[root@dirpav01 ~]# timedatectl
Local time: Fri 2022-09-02 20:11:53 CEST
Universal time: Fri 2022-09-02 18:11:53 UTC
RTC time: Fri 2022-09-02 18:11:52
Time zone: Europe/Madrid (CEST, +0200)
NTP enabled: no
NTP synchronized: yes
RTC in local TZ: no
DST active: yes
Last DST change: DST began at
Sun 2022-03-27 01:59:59 CET
Sun 2022-03-27 03:00:00 CEST
Next DST change: DST ends (the clock jumps one hour backwards) at
Sun 2022-10-30 02:59:59 CEST
Sun 2022-10-30 02:00:00 CET
[root@dirpav01 ~]#
=======================
Replica Installation:
=======================
[root@dirpav01 ~]# ipa-replica-install -n ipa.subdomain.com --hostname=dirpav01.ipa.subdomain.com --server=aaa01.ipa.subdomain.com --realm=IPA.SUBDOMAIN.COM -P admin -w Adm@onm0# --no-host-dns --setup-ca --setup-dns --mkhomedir --auto-reverse --no-forwarders
Configuring client side components
Client hostname: dirpav01.ipa.subdomain.com
Realm: IPA.SUBDOMAIN.COM
DNS Domain: ipa.subdomain.com
IPA Server: aaa01.ipa.subdomain.com
BaseDN: dc=ipa,dc=subdomain,dc=com
Skipping synchronizing time with NTP server.
Successfully retrieved CA cert
Subject: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
Issuer: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
Valid From: 2018-04-12 14:15:30
Valid Until: 2038-04-12 14:15:30
Enrolled in IPA realm IPA.SUBDOMAIN.COM
Created /etc/ipa/default.conf
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm IPA.SUBDOMAIN.COM
trying https://aaa01.ipa.subdomain.com/ipa/json
[try 1]: Forwarding 'schema' to json server 'https://aaa01.ipa.subdomain.com/ipa/json'
trying https://aaa01.ipa.subdomain.com/ipa/session/json
[try 1]: Forwarding 'ping' to json server 'https://aaa01.ipa.subdomain.com/ipa/session/json'
[try 1]: Forwarding 'ca_is_enabled' to json server 'https://aaa01.ipa.subdomain.com/ipa/session/json'
Systemwide CA database updated.
DNS query for dirpav01.ipa.subdomain.com. A failed: The DNS operation timed out after 30.0018370152 seconds
DNS resolution for hostname dirpav01.ipa.subdomain.com failed: The DNS operation timed out after 30.0018370152 seconds
Failed to update DNS records.
Missing A/AAAA record(s) for host dirpav01.ipa.subdomain.com: 10.26.60.179.
Missing reverse record(s) for address(es): 10.26.60.179.
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
[try 1]: Forwarding 'host_mod' to json server 'https://aaa01.ipa.subdomain.com/ipa/session/json'
Could not update DNS SSHFP records.
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring ipa.subdomain.com as NIS domain.
Client configuration complete.
The ipa-client-install command was successful
Warning: skipping DNS resolution of host dirpav01.ipa.subdomain.com
Warning: skipping DNS resolution of host aaa01.ipa.subdomain.com
Run connection check to master
Connection check OK
Configuring NTP daemon (ntpd)
[1/4]: stopping ntpd
[2/4]: writing configuration
[3/4]: configuring ntpd to start on boot
[4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 30 seconds
[1/42]: creating directory server instance
[2/42]: enabling ldapi
[3/42]: configure autobind for root
[4/42]: stopping directory server
[5/42]: updating configuration in dse.ldif
[6/42]: starting directory server
[7/42]: adding default schema
[8/42]: enabling memberof plugin
[9/42]: enabling winsync plugin
[10/42]: configure password logging
[11/42]: configuring replication version plugin
[12/42]: enabling IPA enrollment plugin
[13/42]: configuring uniqueness plugin
[14/42]: configuring uuid plugin
[15/42]: configuring modrdn plugin
[16/42]: configuring DNS plugin
[17/42]: enabling entryUSN plugin
[18/42]: configuring lockout plugin
[19/42]: configuring topology plugin
[20/42]: creating indices
[21/42]: enabling referential integrity plugin
[22/42]: configuring certmap.conf
[23/42]: configure new location for managed entries
[24/42]: configure dirsrv ccache
[25/42]: enabling SASL mapping fallback
[26/42]: restarting directory server
[27/42]: creating DS keytab
[28/42]: ignore time skew for initial replication
[29/42]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 31 seconds elapsed
Update succeeded
[30/42]: prevent time skew after initial replication
[31/42]: adding sasl mappings to the directory
[32/42]: updating schema
[33/42]: setting Auto Member configuration
[34/42]: enabling S4U2Proxy delegation
[35/42]: initializing group membership
[36/42]: adding master entry
[37/42]: initializing domain level
[38/42]: configuring Posix uid/gid generation
[39/42]: adding replication acis
[40/42]: activating sidgen plugin
[41/42]: activating extdom plugin
[42/42]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc)
[1/5]: configuring KDC
[2/5]: adding the password extension to the directory
[3/5]: creating anonymous principal
[4/5]: starting the KDC
[5/5]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
[1/2]: starting kadmin
[2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring directory server (dirsrv)
[1/3]: configuring TLS for DS instance
[2/3]: importing CA certificates from LDAP
[3/3]: restarting directory server
Done configuring directory server (dirsrv).
Configuring the web interface (httpd)
[1/22]: stopping httpd
[2/22]: setting mod_nss port to 443
[3/22]: setting mod_nss cipher suite
[4/22]: setting mod_nss protocol list to TLSv1.2
[5/22]: setting mod_nss password file
[6/22]: enabling mod_nss renegotiate
[7/22]: disabling mod_nss OCSP
[8/22]: adding URL rewriting rules
[9/22]: configuring httpd
[10/22]: setting up httpd keytab
[11/22]: configuring Gssproxy
[12/22]: setting up ssl
[13/22]: configure certmonger for renewals
[14/22]: importing CA certificates from LDAP
[15/22]: publish CA cert
[16/22]: clean up any existing httpd ccaches
[17/22]: configuring SELinux for httpd
[18/22]: create KDC proxy config
[19/22]: enable KDC proxy
[20/22]: starting httpd
[21/22]: configuring httpd to start on boot
[22/22]: enabling oddjobd
Done configuring the web interface (httpd).
Configuring ipa-otpd
[1/2]: starting ipa-otpd
[2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Configuring ipa-custodia
[1/4]: Generating ipa-custodia config file
[2/4]: Generating ipa-custodia keys
[3/4]: starting ipa-custodia
[4/4]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
[1/30]: creating certificate server db
[2/30]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 30 seconds elapsed
Update succeeded
[3/30]: creating ACIs for admin
[4/30]: creating installation admin user
[5/30]: configuring certificate server instance
[6/30]: secure AJP connector
[7/30]: reindex attributes
[8/30]: exporting Dogtag certificate store pin
[9/30]: stopping certificate server instance to update CS.cfg
[10/30]: backing up CS.cfg
[11/30]: disabling nonces
[12/30]: set up CRL publishing
[13/30]: enable PKIX certificate path discovery and validation
[14/30]: destroying installation admin user
[15/30]: starting certificate server instance
[16/30]: Finalize replication settings
[17/30]: configure certmonger for renewals
[18/30]: Importing RA key
[19/30]: setting audit signing renewal to 2 years
[20/30]: restarting certificate server
[21/30]: authorizing RA to modify profiles
[22/30]: authorizing RA to manage lightweight CAs
[23/30]: Ensure lightweight CAs container exists
[24/30]: configure certificate renewals
[25/30]: configure Server-Cert certificate renewal
[26/30]: Configure HTTP to proxy connections
[27/30]: restarting certificate server
[28/30]: updating IPA configuration
[29/30]: enabling CA instance
[30/30]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR CA did not start in 300.0s
ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
[root@dirpav01 ~]#
================================
/var/log/pki/pki-tomcat/ca/debug
================================
[02/Sep/2022:20:41:02][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca
[02/Sep/2022:20:41:02][localhost-startStop-1]: ldapconn/PKISocketFactory.makeSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca
[02/Sep/2022:20:41:02][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering!
[02/Sep/2022:20:41:02][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca
[02/Sep/2022:20:41:02][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca
[02/Sep/2022:20:41:02][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca
[02/Sep/2022:20:41:02][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca
[02/Sep/2022:20:41:02][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: begins
[02/Sep/2022:20:41:02][localhost-startStop-1]: SignedAuditLogger: event CLIENT_ACCESS_SESSION_ESTABLISH
[02/Sep/2022:20:41:02][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: CS_CLIENT_ACCESS_SESSION_ESTABLISH_SUCCESS
[02/Sep/2022:20:41:02][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: clientIP=10.26.60.179 serverIP=10.26.60.179 serverPort=31746
[02/Sep/2022:20:41:02][localhost-startStop-1]: SSL handshake happened
Could not connect to LDAP server host dirpav01.ipa.subdomain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:667)
at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1054)
at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:960)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:566)
at com.netscape.certsrv.apps.CMS.init(CMS.java:194)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1461)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Internal Database Error encountered: Could not connect to LDAP server host dirpav01.ipa.subdomain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689)
at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1054)
at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:960)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:566)
at com.netscape.certsrv.apps.CMS.init(CMS.java:194)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1461)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
[02/Sep/2022:20:41:02][localhost-startStop-1]: CMS.start(): shutdown server
[02/Sep/2022:20:41:02][localhost-startStop-1]: CMSEngine.shutdown()
[root@dirpav01 ~]#
================================
/var/log/ipareplica-install.log
================================
2022-09-02T18:42:31Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2022-09-02T18:42:31Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
2022-09-02T18:42:31Z DEBUG Waiting for CA to start...
2022-09-02T18:42:32Z DEBUG request POST http://dirpav01.ipa.subdomain.com:8080/ca/admin/ca/getStatus
2022-09-02T18:42:32Z DEBUG request body ''
2022-09-02T18:42:32Z DEBUG response status 500
2022-09-02T18:42:32Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 02 Sep 2022 18:42:32 GMT
Connection: close
2022-09-02T18:42:32Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2022-09-02T18:42:32Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
2022-09-02T18:42:32Z DEBUG Waiting for CA to start...
2022-09-02T18:42:33Z DEBUG request POST http://dirpav01.ipa.subdomain.com:8080/ca/admin/ca/getStatus
2022-09-02T18:42:33Z DEBUG request body ''
2022-09-02T18:42:34Z DEBUG response status 500
2022-09-02T18:42:34Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 02 Sep 2022 18:42:34 GMT
Connection: close
2022-09-02T18:42:34Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2022-09-02T18:42:34Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
2022-09-02T18:42:34Z DEBUG Waiting for CA to start...
2022-09-02T18:42:35Z DEBUG request POST http://dirpav01.ipa.subdomain.com:8080/ca/admin/ca/getStatus
2022-09-02T18:42:35Z DEBUG request body ''
2022-09-02T18:42:35Z DEBUG response status 500
2022-09-02T18:42:35Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 02 Sep 2022 18:42:35 GMT
Connection: close
2022-09-02T18:42:35Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2022-09-02T18:42:35Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
2022-09-02T18:42:35Z DEBUG Waiting for CA to start...
2022-09-02T18:42:36Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run
return cfgr.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 360, in run
return self.execute()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 386, in execute
for rval in self._executor():
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 464, in start
self.service.start(instance_name, capture_output=capture_output, wait=wait)
File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 192, in start
self.wait_until_running()
File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 186, in wait_until_running
raise RuntimeError('CA did not start in %ss' % timeout)
2022-09-02T18:42:36Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: CA did not start in 300.0s
2022-09-02T18:42:36Z ERROR CA did not start in 300.0s
2022-09-02T18:42:36Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
[root@dirpav01 ~]#
Sai
From: Florence Blanc-Renaud <flo@redhat.com>
Sent: Friday, September 2, 2022 5:37 PM
To: Polavarapu Manideep Sai <manideep.sai@onmobile.com>
Cc: Rob Crittenden <rcritten@redhat.com>; FreeIPA users list <freeipa-users@lists.fedorahosted.org>
Subject: Re: [Freeipa-users] Re: Free IPA Replica server retrieving two certificates from the IPA master server while installing IPA replica and installation fails
CAUTION. This email originated from outside the organization. Please exercise caution before clicking on links or attachments in case of suspicion or unknown senders.
Hi,
On Thu, Sep 1, 2022 at 7:47 PM Polavarapu Manideep Sai <manideep.sai@onmobile.com> wrote:
Hi Florence/Rob
Upon your advice, I have removed the certificate from the IPA master, Now IPA Replica retrieving one certificate from the IPA master as shown below
Facing another IPA Replica installation issue after deleting/removing the certificate from the IPA master server, please help us on this, please let us know anymore information required on this
PFB Replica installation Logs
==============================
/var/log/ipaclient-install.log :
==============================
2022-09-01T17:03:00Z DEBUG stderr=
2022-09-01T17:03:00Z DEBUG trying to retrieve CA cert via LDAP from aaa01.ipa.subdomain.com
2022-09-01T17:03:01Z DEBUG retrieving schema for SchemaCache url=ldap://aaa01.ipa.subdomain.com:389 conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f840831d3f8>
2022-09-01T17:03:02Z INFO Successfully retrieved CA cert
Subject: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
Issuer: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
Valid From: 2018-04-12 14:15:30
Valid Until: 2038-04-12 14:15:30
2022-09-01T17:03:02Z DEBUG Starting external process
2022-09-01T17:03:02Z DEBUG args=/usr/sbin/ipa-join -s aaa01.ipa.subdomain.com -b dc=ipa,dc=subdomain,dc=com -h dirpav01.ipa.subdomain.com -f
2022-09-01T17:03:07Z DEBUG Process finished, return code=0
2022-09-01T17:03:07Z DEBUG stdout=
2022-09-01T17:03:07Z DEBUG stderr=Keytab successfully retrieved and stored in: /etc/krb5.keytab
Certificate subject base is: O=IPA.SUBDOMAIN.COM
2022-09-01T17:03:07Z INFO Enrolled in IPA realm IPA.SUBDOMAIN.COM
2022-09-01T17:03:07Z DEBUG Starting external process
2022-09-01T17:03:07Z DEBUG args=/usr/bin/kdestroy
2022-09-01T17:03:07Z DEBUG Process finished, return code=0
2022-09-01T17:03:07Z DEBUG stdout=
2022-09-01T17:03:07Z DEBUG stderr=
======================================
Replica installation without debugging :
======================================
Done configuring ipa-custodia.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
[1/30]: creating certificate server db
[2/30]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 30 seconds elapsed
Update succeeded
[3/30]: creating ACIs for admin
[4/30]: creating installation admin user
[5/30]: configuring certificate server instance
[6/30]: secure AJP connector
[7/30]: reindex attributes
[8/30]: exporting Dogtag certificate store pin
[9/30]: stopping certificate server instance to update CS.cfg
[10/30]: backing up CS.cfg
[11/30]: disabling nonces
[12/30]: set up CRL publishing
[13/30]: enable PKIX certificate path discovery and validation
[14/30]: destroying installation admin user
[15/30]: starting certificate server instance
[16/30]: Finalize replication settings
[17/30]: configure certmonger for renewals
[18/30]: Importing RA key
[19/30]: setting audit signing renewal to 2 years
[20/30]: restarting certificate server
[21/30]: authorizing RA to modify profiles
[22/30]: authorizing RA to manage lightweight CAs
[23/30]: Ensure lightweight CAs container exists
[24/30]: configure certificate renewals
[25/30]: configure Server-Cert certificate renewal
[26/30]: Configure HTTP to proxy connections
[27/30]: restarting certificate server
[28/30]: updating IPA configuration
[29/30]: enabling CA instance
[30/30]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR CA did not start in 300.0s
ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
================================
/var/log/ipareplica-install.log
================================
2022-09-01T14:35:58Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2022-09-01T14:35:58Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
2022-09-01T14:35:58Z DEBUG Waiting for CA to start...
2022-09-01T14:35:59Z DEBUG request POST http://dirpav01.ipa.subdomain.com:8080/ca/admin/ca/getStatus
2022-09-01T14:35:59Z DEBUG request body ''
2022-09-01T14:35:59Z DEBUG response status 500
2022-09-01T14:35:59Z DEBUG response headers Server: Apache-Coyote/1.1^M
Content-Type: text/html;charset=utf-8^M
Content-Language: en^M
Content-Length: 2208^M
Date: Thu, 01 Sep 2022 14:35:59 GMT^M
Connection: close^M
2022-09-01T14:35:59Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2022-09-01T14:35:59Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
2022-09-01T14:35:59Z DEBUG Waiting for CA to start...
2022-09-01T14:36:00Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run
File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 186, in wait_until_running
raise RuntimeError('CA did not start in %ss' % timeout)
2022-09-01T14:36:00Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: CA did not start in 300.0s
2022-09-01T14:36:00Z ERROR CA did not start in 300.0s
2022-09-01T14:36:00Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
The logs are a bit confusing, the client install logs have timestamps around 2022-09-01T17:03:07Z but replica-install around 2022-09-01T14:36:00Z which is earlier? Same comment for the tomcat logs around 01/Sep/2022:16:45:21 (pki logs use the local timezone while client and repl logs use UTC times, but the times are completely unrelated here).
Can you clean up the replica you're trying to install and start over, then send the most recent logs?
- on the failing replica: ipa-server-install --uninstall -U
- on the master: kinit admin; ipa server-del <replica> --force
- on the failing replica: perform the installation with your usual method (either in a 2-step process with ipa-client-install/ipa-replica-install or in a single step with ipa-replica-install).
Also provide the timezone of the replica so that we can translate all the timestamps in UTC time.
flo
=================================
/var/log/pki/pki-tomcat/ca/debug :
=================================
[01/Sep/2022:16:45:21][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca
[01/Sep/2022:16:45:21][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca
[01/Sep/2022:16:45:21][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca
[01/Sep/2022:16:45:21][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca
[01/Sep/2022:16:45:21][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: begins
[01/Sep/2022:16:45:21][localhost-startStop-1]: SignedAuditLogger: event CLIENT_ACCESS_SESSION_ESTABLISH
[01/Sep/2022:16:45:21][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: CS_CLIENT_ACCESS_SESSION_ESTABLISH_SUCCESS
[01/Sep/2022:16:45:21][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: clientIP=10.26.60.179 serverIP=10.26.60.179 serverPort=31746
[01/Sep/2022:16:45:21][localhost-startStop-1]: SSL handshake happened
Could not connect to LDAP server host dirpav01.ipa.subdomain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:667)
at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1054)
at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:960)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:566)
at com.netscape.certsrv.apps.CMS.init(CMS.java:194)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1461)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Internal Database Error encountered: Could not connect to LDAP server host dirpav01.ipa.subdomain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689)
at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1054)
at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:960)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:566)
at com.netscape.certsrv.apps.CMS.init(CMS.java:194)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1461)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
[01/Sep/2022:16:45:21][localhost-startStop-1]: CMS.start(): shutdown server
[01/Sep/2022:16:45:21][localhost-startStop-1]: CMSEngine.shutdown()
Sai
From: Florence Blanc-Renaud via FreeIPA-users <freeipa-users@lists.fedorahosted.org>
Sent: Wednesday, August 31, 2022 12:28 PM
To: FreeIPA users list <freeipa-users@lists.fedorahosted.org>
Cc: Rob Crittenden <rcritten@redhat.com>; Polavarapu Manideep Sai <manideep.sai@onmobile.com>; Florence Blanc-Renaud <flo@redhat.com>
Subject: [Freeipa-users] Re: Free IPA Replica server retrieving two certificates from the IPA master server while installing IPA replica and installation fails
CAUTION. This email originated from outside the organization. Please exercise caution before clicking on links or attachments in case of suspicion or unknown senders.
Hi,
I'm replying to the same questions posted on my blog:
Hi floblanc,
Thank you for the reply,
I have a few queries, can you please clarify
1. should we run ipa-cert-update on IPA master server also and then after on all IPA replica server and their clients ?
Yes, ipa-certupdate has to be run on all the machines enrolled into IPA.
2. Do we need to consider only one common name i.e. “cn=directory manager” as we have two one is LADP and other one is for HTTP
dbm:/etc/dirsrv/slapd-IPA-ONMOBILE-COM/
dbm:/etc/httpd/aliasldapsearch -D “cn=directory manager” -W -b cn=certificates,cn=ipa,cn=etc,dc=ipa,dc=example,dc=com “(&(objectClass=ipaCertificate)(objectClass=pkiCA))”
Refer to ldapsearch man page to understand the options:
- the -D "cn=directory manager" option means that the LDAP operations will be authenticated with the user Directory Manager. When you installed the first IPA server with ipa-server-install, this user was created with the password provided with ipa-server-install -p|--ds-password DM_PASSWORD.
- the -W option means "prompt for password"
- the -b option specifies a search base. The CA certificates are stored below cn=certificates,cn=ipa,cn=etc,dc=ipa,dc=example,dc=com, the search needs to target this search base
- “(&(objectClass=ipaCertificate)(objectClass=pkiCA))” is the search filter allowing to find CA certificates
This single search allows to retrieve all the CA certificates, one ldap entry for each certificate.
Any other common name for HTTP:
ldapsearch -D “cn=?” -W -b cn=certificates,cn=ipa,cn=etc,dc=ipa,dc=example,dc=com “(&(objectClass=ipaCertificate)(objectClass=pkiCA))”
Or else this is the only query to search the ipaCertificate in whole ldap database?
if i want to search the all occurrence of this invalid certificate in the whole server/database, how can we achieve this
3. I have a infrastructure with one IPA master and 13 IPA Replicas, if i delete the certificate in IPA Master and run ipa-certupdate, and again run ipa-certupdate on 13 IPA Replica servers, and its clients, i hope there will not be any issue after changes and also pki-tomcatd.target service will be running
If the LDAP entry corresponding to the certificate is deleted on the IPA master, the replication will propagate this deletion to the other replicas. This means the entry will be removed from all the LDAP servers.
When ipa-certupdate is run, the list of CA certificates is refreshed (re-read from LDAP) and updated on the local NSS Databases.
HTH,
flo
Or do you suggest any other better way without any impact on services further as it is production setup
Note: As we deleted last time then pki-tomcat.target service was stopped and not started [we didn’t run ipa-certupdate on IPA Master]
How can we check all occurrence of this invalid certificate in IPA master server
On Tue, Aug 30, 2022 at 8:09 PM Polavarapu Manideep Sai via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
Hi Rob,
Can you please help me on this
Regards
ManideepSai
-----Original Message-----
From: Rob Crittenden <rcritten@redhat.com>
Sent: Tuesday, August 30, 2022 11:36 PM
To: FreeIPA users list <freeipa-users@lists.fedorahosted.org>
Cc: Polavarapu Manideep Sai <manideep.sai@onmobile.com>
Subject: Re: [Freeipa-users] Free IPA Replica server retrieving two certificates from the IPA master server while installing IPA replica and installation fails
CAUTION. This email originated from outside the organization. Please exercise caution before clicking on links or attachments in case of suspicion or unknown senders.
Polavarapu Manideep Sai via FreeIPA-users wrote:
> Hi Team,
>
>
>
> Need help from freeipa,
>
>
>
> Free IPA Replica server retrieving two certificates from the IPA master
> server while installing IPA replica and installation fails
>
>
>
> please check the below issue and let us know the fix and please let us
> know if any more details required
>
>
>
> Master server: aaa01
>
> Replica server1: dir01 (currently installing replica server )
>
> Replica server2: dirus02 (which was a replica server previously that has
> been removed from replication)
>
>
>
>
>
> As noticed while installing ipa replica server, replica server
> retrieving two certificates from the master server, and saving it in
> /etc/ipa/ca.crt in this process at the stage Configuring the web
> interface (httpd) we got the below error i.e.
>
>
>
> ipa-replica-install command failed, exception: CalledProcessError:
> Command '/usr/bin/certutil -d dbm:/etc/httpd/alias -A -n Server-Cert -t
> ,, -a -f /etc/httpd/alias/pwdfile.txt' returned non-zero exit status 255
>
>
>
> ===============================================
>
>
>
> While installing Replica /var/log/ipaclient-install.log
>
> ---------------------------------------------------
>
>
>
> 2022-08-15T13:52:08Z DEBUG stderr=
>
> 2022-08-15T13:52:08Z DEBUG trying to retrieve CA cert via LDAP from
> aaa01.ipa.subdomain.com
>
> 2022-08-15T13:52:09Z DEBUG retrieving schema for SchemaCache
> url=ldap://aaa01.ipa.subdomain.com:389
> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f17fe812440>
>
> 2022-08-15T13:52:11Z INFO Successfully retrieved CA cert
>
>
>
> Subject: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
>
> Issuer: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
>
> Valid From: 2018-04-12 14:15:30
>
> Valid Until: 2038-04-12 14:15:30
>
>
>
> Subject: CN=dirus02.ipa.subdomain.com,O=IPA.SUBDOMAIN.COM
>
> Issuer: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
>
> Valid From: 2019-01-21 11:54:13
>
> Valid Until: 2021-01-21 11:54:13
>
>
>
> 2022-08-15T13:52:11Z DEBUG Starting external process
>
> 2022-08-15T13:52:11Z DEBUG args=/usr/sbin/ipa-join -s
> aaa01.ipa.subdomain.com -b dc=ipa,dc=example,dc=com -h
> dirpav01-tfln-mdr1-omes.ipa.subdomain.com
>
> 2022-08-15T13:52:15Z DEBUG Process finished, return code=0
>
> 2022-08-15T13:52:15Z DEBUG stdout=
>
> 2022-08-15T13:52:15Z DEBUG stderr=Keytab successfully retrieved and
> stored in: /etc/krb5.keytab
>
> Certificate subject base is: O=IPA.SUBDOMAIN.COM
>
>
>
> 2022-08-15T13:52:15Z INFO Enrolled in IPA realm IPA.SUBDOMAIN.COM
>
> 2022-08-15T13:52:15Z DEBUG Starting external process
>
> 2022-08-15T13:52:15Z DEBUG args=/usr/bin/kdestroy
>
> 2022-08-15T13:52:15Z DEBUG Process finished, return code=0
>
> 2022-08-15T13:52:15Z DEBUG stdout=
>
>
>
> ==================================
>
>
>
>
>
>
>
> While installing replica /var/log/ipareplica-install.log
>
> --------------------------------------------------
>
>
>
> 2022-08-15T15:07:11Z DEBUG [14/22]: importing CA certificates from LDAP
>
> 2022-08-15T15:07:11Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
>
> 2022-08-15T15:07:11Z DEBUG Starting external process
>
> 2022-08-15T15:07:11Z DEBUG args=/usr/bin/certutil -d
> dbm:/etc/httpd/alias -A -n IPA.SUBDOMAIN.COM IPA CA -t CT,C,C -a -f
> /etc/httpd/alias/pwdfile.txt
>
> 2022-08-15T15:07:11Z DEBUG Process finished, return code=0
>
> 2022-08-15T15:07:11Z DEBUG stdout=
>
> 2022-08-15T15:07:11Z DEBUG stderr=
>
> 2022-08-15T15:07:11Z DEBUG Starting external process
>
> 2022-08-15T15:07:11Z DEBUG args=/usr/bin/certutil -d
> dbm:/etc/httpd/alias -A -n Server-Cert -t ,, -a -f
> /etc/httpd/alias/pwdfile.txt
>
> 2022-08-15T15:07:12Z DEBUG Process finished, return code=255
>
> 2022-08-15T15:07:12Z DEBUG stdout=
>
> 2022-08-15T15:07:12Z DEBUG stderr=certutil: could not add certificate to
> token or database: SEC_ERROR_ADDING_CERT: Error adding certificate to
> database.
>
>
>
> 2022-08-15T15:07:12Z DEBUG Traceback (most recent call last):
>
> File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> line 567, in start_creation
>
> run_step(full_msg, method)
>
> File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> line 557, in run_step
>
>
>
>
>
> Observation in Master server(aaa01) ldap database :
>
> =======================================
>
>
>
> [root@aaa01~]# ldapsearch -D 'cn=directory manager' -w XXXXXXXXX |
> grep "ipaCertSubject"
>
> ipaCertSubject: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
>
> ipaCertSubject: CN=dirus02.ipa.subdomain.com,O=IPA.SUBDOMAIN.COM
>
> [root@aaa01~]#
>
>
>
> ====================
>
> We could see this certificate
> "CN=dirus02.ipa.subdomain.com,O=IPA.SUBDOMAIN.COM" in IPA master server
> GUI as well we have revoked it too , but still it retrieves the same
> and installation got fails everytime
>
>
>
> =================
>
>
>
> In ideal case while installing replica it has to retrieve only one
> certificate i.e. CN=Certificate Authority,O=IPA.SUBDOMAIN.COM but this
> case it retrieves
>
>
>
>
>
> Please let us know if any more details required and let us know how can
> we fix this issue, without impact on whole setup
>
>
>
>
>
> ipaCertIssuerSerial
>
>
>
> ipaCertIssuerSerial: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM;1
> [which is a valid certificate]
>
> ipaCertIssuerSerial: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM;32 [
> invalid certificate retrieves from ipa master while installing ipa replica]
>
>
>
>
>
>
>
> [root@aaa01]# ipa cert-show
>
>
>
> Serial number: 32
>
> Issuing CA: ipa
>
> Certificate:
> MIIFGTCCBAGgAwIBAgIBIDANBgkqhkiG9w0BAQsFADA7MRkwFwYDVQQKDBBJUEEuT05NT0JJTEUuQ09NMR4wHAYDVQQ
>
> DDBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMTkwMTIxMTE1NDEzWhcNMjEwMTIxMTE1NDEzWjBMMRkwFwYDVQQKDBBJUEEuT
>
> 05NT0JJTEUuQ09NMS8wLQYDVQQDDCZkaXJ1czAyLW1pYS10bGZuLW9tdXMuaXBhLm9ubW9iaWxlLmNvbTCCASIwDQYJKoZIhvcNAQE
>
> BBQADggEPADCCAQoCggEBAKln0qNlB+38cXbyOurkVgK+GMYM9loUVFAvZGlydXMwMi1taWEtdGxmbi1vbXVzLmlwYS5vbm1vYmlsZS5
>
> jb21ASVBBLk9OTU9CSUxFLkNPTaBbBgYrBgEFAgKgUTBPoBIbEElQQS5PTk1PQklMRS5DT02hOTA3oAMCAQGhMDAuGwRIVFRQGyZkaXJ
>
> 1czAyLW1pYS10bGZuLW9tdXMuaXBhLm9ubW9iaWxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAcFbSY4tVpZHWVDGsahRNfCqv/x/xCT
>
> BEYHvCSdycHAV7Ogq6zEENviRDOEOYqe1x7BxyF7B/hhB3PX2uqYmFrgPffyfwCxGZb0DRnnOLnwldxe3QdwjIIuUptY9fOgvbjx+bd5iLIgNp
>
> aAZcN70PePdPA0xYpAo3CQkowCojAke2QGsPp6DrXS1wRrE4maH0LmEtu56hSbARoN4DgJ91PKgPkZ+BNyq9BmoPTRsxpAGBvms2SAbx
>
> q1iUmNcVCurqvF/Gu2Z8L5rlpPiVjSbup9Zq5LuhLtfeMsgrwfZOcwZQfSCCykMUH9eAipvsNoHvPxiJeHhDk8Zx+cADESTL4w==
>
>
>
> Subject: CN=dirus02.ipa.subdomain.com,O=IPA.SUBDOMAIN.COM
>
> Subject DNS name: dirus02.ipa.subdomain.com
>
> Subject UPN: HTTP/dirus02.ipa.subdomain.com@IPA.SUBDOMAIN.COM
>
> Subject Kerberos principal name:
> HTTP/dirus02.ipa.subdomain.com@IPA.SUBDOMAIN.COM
>
> Issuer: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
>
> Not Before: Mon Jan 21 11:54:13 2019 UTC
>
> Not After: Thu Jan 21 11:54:13 2021 UTC
>
> Serial number: 32
>
> Serial number (hex): 0x20
>
> Revoked: True
>
> Revocation reason: 2
>
> [root@aaa01~]#
The CA certificates are stored in LDAP under
cn=certificates,cn=ipa,cn=etc,dc=example,dc=test (substitute your own
basedn).
Find the incorrect entry and use ldapdelete to remove it. If you aren't
very familiar with LDAP command-line tools then something like Apache
Directory Studio may be a better choice.
rob
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
DISCLAIMER: The information in this message is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error. Further, this e-mail may contain viruses and all reasonable precaution to minimize the risk arising there from is taken by OnMobile. OnMobile is not liable for any damage sustained by you as a result of any virus in this e-mail. All applicable virus checks should be carried out by you before opening this e-mail or any attachment thereto.
Thank you - OnMobile Global Limited.