Hi Guys,
I have a 2 host basic IPA setup both IPA servers are running dns & ca. I'm running on Centos 7.6 using freeipa version 4.6.4 & dogtag version 10.5.9
I've made a subCA called vpnca and a certificate policy and all this is working fine with the exception of OCSP on the 2nd IPA box.
The original master works fine and issues OCSP responses for certifcates issued by the vpnca (subCA) however the replica IPA box fails to respond.
I've had a look through the logs and found in the /var/log/pki/pki-tomcat/ca/debug log an error on the 2nd box when doing an OCSP request against it for a certificate issued by the subCA. I should note here that OCSP requests for certificates issued by the main IPA CA work fine it's only for ones issued by the subCA on the replica that seem to be broken.
I have also spotted the 2nd IPA server complaining that is can't get caSigningCert [04/Sep/2019:13:24:01][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Running ExternalProcessKeyRetriever [04/Sep/2019:13:24:01][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: About to execute command: [/usr/libexec/ipa/ipa-pki-retrieve-key, caSigningCert cert-pki-ca dd4ea812-c044-41c0-93bf-ec376c732c93, man-fb-ipa-01.testhost.com] [04/Sep/2019:13:24:02][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Failed to retrieve key from any host. [04/Sep/2019:13:24:02][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: KeyRetriever did not return a result. [04/Sep/2019:13:24:02][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Retrying in 1946 seconds
I'm presuming this is the reason OCSP is failing as it can't sign the response for the subCA?
Does anyone know if this is a known issue or if there is something I need to modify to get the OCSP working on the replica host?
Any help would be greatly appreciated
Thanks Dave
See logs below.
2nd IPA Replica (Broken) /var/log/pki/pki-tomcat/ca/debug [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet:service() uri = /ca/ocsp [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet: caOCSP start to service. [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: IP: 10.128.164.2 [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet: no authMgrName [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet.authorize(DirAclAuthz) [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet: in auditSubjectID [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet: auditSubjectID auditContext {locale=en_GB, ipAddress=10.128.164.2} [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet auditSubjectID: subjectID: null [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet: in auditGroupID [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet: auditGroupID auditContext {locale=en_GB, ipAddress=10.128.164.2} [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet auditGroupID: groupID: null [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: In LdapBoundConnFactory::getConn() [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: masterConn is connected: true [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: getConn: conn is connected true [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: getConn: mNumConns now 2 [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: returnConn: mNumConns now 3 [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: AAclAuthz.checkPermission(certServer.ee.request.ocsp, submit) [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: checkAllowEntries(): expressions: ipaddress=".*" [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: evaluating expressions: ipaddress=".*" [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: evaluated expression: ipaddress=".*" to be true [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: DirAclAuthz: authorization passed [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: SignedAuditLogger: event AUTHZ [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: OCSPServlet: Servlet Path: /ocsp [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: OCSPServlet: RequestURI: /ca/ocsp [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: OCSPServlet: PathInfo: null [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: OCSPServlet: HTTP method: POST [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: OCSPServlet: processing POST request [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: OCSPServlet: decoding request [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: OCSPServlet: validating request [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: In LdapBoundConnFactory::getConn() [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: masterConn is connected: true [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: getConn: conn is connected true [04/Sep/2019:12:25:13][ajp-bio-127.0.0.1-8009-exec-1]: getConn: mNumConns now 2 [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: returnConn: mNumConns now 3 [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: In LdapBoundConnFactory::getConn() [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: masterConn is connected: true [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: getConn: conn is connected true [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: getConn: mNumConns now 2 [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: returnConn: mNumConns now 3 [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: CertificateAuthority: validating OCSP request [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: CertificateAuthority: processing request for cert 0x1b [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: In LdapBoundConnFactory::getConn() [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: masterConn is connected: true [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: getConn: conn is connected true [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: getConn: mNumConns now 2 [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: returnConn: mNumConns now 3 java.lang.NullPointerException at com.netscape.ca.CertificateAuthority.getResponderIDByName(CertificateAuthority.java:2340) at com.netscape.ca.CertificateAuthority.validate(CertificateAuthority.java:2473) at com.netscape.ca.CertificateAuthority.validate(CertificateAuthority.java:2428) at com.netscape.cms.servlet.ocsp.OCSPServlet.process(OCSPServlet.java:222) at com.netscape.cms.servlet.base.CMSServlet.service(CMSServlet.java:493) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297) at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet: in auditSubjectID [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet: auditSubjectID auditContext {locale=en_GB, ipAddress=10.128.164.2} [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet auditSubjectID: subjectID: null [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: SignedAuditLogger: event OCSP_GENERATION [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: OCSPServlet: response is null [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet.java: renderTemplate [04/Sep/2019:12:25:14][ajp-bio-127.0.0.1-8009-exec-1]: CMSServlet: curDate=Wed Sep 04 12:25:14 BST 2019 id=caOCSP time=213
If I look at 1st IPA server which is working I see
1st IPA Master (Working) /var/log/pki/pki-tomcat/ca/debug [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet:service() uri = /ca/ocsp [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet: caOCSP start to service. [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: IP: 10.128.167.2 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet: no authMgrName [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet.authorize(DirAclAuthz) [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet: in auditSubjectID [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet: auditSubjectID auditContext {locale=en_GB, ipAddress=10.128.167.2} [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet auditSubjectID: subjectID: null [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet: in auditGroupID [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet: auditGroupID auditContext {locale=en_GB, ipAddress=10.128.167.2} [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet auditGroupID: groupID: null [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: In LdapBoundConnFactory::getConn() [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: masterConn is connected: true [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: getConn: conn is connected true [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: getConn: mNumConns now 2 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: returnConn: mNumConns now 3 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: AAclAuthz.checkPermission(certServer.ee.request.ocsp, submit) [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: checkAllowEntries(): expressions: ipaddress=".*" [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: evaluating expressions: ipaddress=".*" [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: evaluated expression: ipaddress=".*" to be true [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: DirAclAuthz: authorization passed [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: SignedAuditLogger: event AUTHZ [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: Servlet Path: /ocsp [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: RequestURI: /ca/ocsp [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: PathInfo: null [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: HTTP method: POST [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: processing POST request [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: decoding request [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: validating request [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: In LdapBoundConnFactory::getConn() [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: masterConn is connected: true [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: getConn: conn is connected true [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: getConn: mNumConns now 4 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: returnConn: mNumConns now 5 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: In LdapBoundConnFactory::getConn() [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: masterConn is connected: true [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: getConn: conn is connected true [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: getConn: mNumConns now 4 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: returnConn: mNumConns now 5 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CertificateAuthority: validating OCSP request [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CertificateAuthority: processing request for cert 0x1b [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: In LdapBoundConnFactory::getConn() [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: masterConn is connected: true [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: getConn: conn is connected true [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: getConn: mNumConns now 4 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: returnConn: mNumConns now 5 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: adding signature [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: Getting algorithm context for SHA256withRSA RSASignatureWithSHA256Digest [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: Signing Certificate [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet: in auditSubjectID [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet: auditSubjectID auditContext {locale=en_GB, ipAddress=10.128.167.2} [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: CMSServlet auditSubjectID: subjectID: null [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: SignedAuditLogger: event OCSP_GENERATION [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: OCSP Request: [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: MGwwaqADAgEAMD4wPDA6MAkGBSsOAwIaBQAEFK377uGJz9Owh8lyIT07pU1YHAEs^M BBTDA9mf27XJPVL0EOy+SaFKAxCZhAIBG6IjMCEwHwYJKwYBBQUHMAECBBIEEJMj^M ZAn0Vjd91e0eZdmHXyo=^M
[04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: Serial Number: 27 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: OCSP Response Size: [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: 2364 [04/Sep/2019:13:22:06][ajp-bio-127.0.0.1-8009-exec-15]: OCSPServlet: OCSP Response Data: **SNIP**
On Wed, Sep 04, 2019 at 12:33:27PM -0000, David Etchen via FreeIPA-users wrote:
Hi Guys,
I have a 2 host basic IPA setup both IPA servers are running dns & ca. I'm running on Centos 7.6 using freeipa version 4.6.4 & dogtag version 10.5.9
I've made a subCA called vpnca and a certificate policy and all this is working fine with the exception of OCSP on the 2nd IPA box.
The original master works fine and issues OCSP responses for certifcates issued by the vpnca (subCA) however the replica IPA box fails to respond.
I've had a look through the logs and found in the /var/log/pki/pki-tomcat/ca/debug log an error on the 2nd box when doing an OCSP request against it for a certificate issued by the subCA. I should note here that OCSP requests for certificates issued by the main IPA CA work fine it's only for ones issued by the subCA on the replica that seem to be broken.
I have also spotted the 2nd IPA server complaining that is can't get caSigningCert [04/Sep/2019:13:24:01][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Running ExternalProcessKeyRetriever [04/Sep/2019:13:24:01][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: About to execute command: [/usr/libexec/ipa/ipa-pki-retrieve-key, caSigningCert cert-pki-ca dd4ea812-c044-41c0-93bf-ec376c732c93, man-fb-ipa-01.testhost.com] [04/Sep/2019:13:24:02][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Failed to retrieve key from any host. [04/Sep/2019:13:24:02][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: KeyRetriever did not return a result. [04/Sep/2019:13:24:02][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Retrying in 1946 seconds
I'm presuming this is the reason OCSP is failing as it can't sign the response for the subCA?
Does anyone know if this is a known issue or if there is something I need to modify to get the OCSP working on the replica host?
Any help would be greatly appreciated
Thanks Dave
Hi Dave,
Indeed OCSP is failing because the key is not presence (certificate issuance using the sub-CA will also fail on the replica). So we must investigate why key replication is failing.
When a sub-CA is created, replicas contact the Custodia service on the master and request the key. First, restart the ipa-custodia service on the master (maybe it is not working properly and a restart will resolve it). You may wish to restart the pki-tomcatd@pki-tomcat service on the *replica* too, because sub-CA key replication attempts use exponential backoff (I see from the log it was up to 1946 seconds). If key replication is still failing have a look at the journal and the httpd logs on the *master* for clues.
HTH, Fraser
Hi Fraser,
Thanks for replying.
I've restarted both sides like you suggested but still don't see a difference. I can see the back off time has started again like you said.
[04/Sep/2019:15:20:12][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Failed to retrieve key from any host. [04/Sep/2019:15:20:12][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: KeyRetriever did not return a result. [04/Sep/2019:15:20:12][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Retrying in 15 seconds [04/Sep/2019:15:20:27][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Running ExternalProcessKeyRetriever [04/Sep/2019:15:20:27][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: About to execute command: [/usr/libexec/ipa/ipa-pki-retrieve-key, caSigningCert cert-pki-ca dd4ea812-c044-41c0-93bf-ec376c732c93, man-fb-ipa-01.testhost.com] [04/Sep/2019:15:20:28][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Failed to retrieve key from any host. [04/Sep/2019:15:20:28][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: KeyRetriever did not return a result. [04/Sep/2019:15:20:28][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Retrying in 22 seconds
As I posted later running the command manually works and I get a reponse containing what looks like a JSON response with certificate and wrapped_key attributeswhich corespond to the subCA. I did have to do kinit first as I wasn't logged in with an IPA user.
I'm now puzzled as to why dogtag doesn't seem to get the response. Do you know how I could emulate running the command as dogtag to see if it's something to do with kerberos or something like that.
I had a quick look in the audit log just to check it wasn't selinux related but can't find anything.
Just did some further testing, if I run the command manually I can see my traffic in tcpdump connecting on port 443 to the master however when dogtag is supposidly running the scripts I don't see any connection attempts. So it looks like the script isn't actually running at least to the point where it tries to talk to the master anyway.
I dived off down the rabbit hole and thought maybe DNS isn't working for dogtag so added an entry to /etc/hosts for the master IPA server but this didn't make any difference.
For good measure I've disabled selinux. I'm now looking to see if I can crank up the logging output from dogtag to see if there is anything extra it's saying.
Any ideas welcome.
Thanks Dave
On Wed, Sep 04, 2019 at 03:08:30PM -0000, David Etchen via FreeIPA-users wrote:
Hi Fraser,
Thanks for replying.
I've restarted both sides like you suggested but still don't see a difference. I can see the back off time has started again like you said.
[04/Sep/2019:15:20:12][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Failed to retrieve key from any host. [04/Sep/2019:15:20:12][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: KeyRetriever did not return a result. [04/Sep/2019:15:20:12][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Retrying in 15 seconds [04/Sep/2019:15:20:27][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Running ExternalProcessKeyRetriever [04/Sep/2019:15:20:27][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: About to execute command: [/usr/libexec/ipa/ipa-pki-retrieve-key, caSigningCert cert-pki-ca dd4ea812-c044-41c0-93bf-ec376c732c93, man-fb-ipa-01.testhost.com] [04/Sep/2019:15:20:28][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Failed to retrieve key from any host. [04/Sep/2019:15:20:28][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: KeyRetriever did not return a result. [04/Sep/2019:15:20:28][KeyRetrieverRunner-dd4ea812-c044-41c0-93bf-ec376c732c93]: Retrying in 22 seconds
As I posted later running the command manually works and I get a reponse containing what looks like a JSON response with certificate and wrapped_key attributeswhich corespond to the subCA. I did have to do kinit first as I wasn't logged in with an IPA user.
I'm now puzzled as to why dogtag doesn't seem to get the response. Do you know how I could emulate running the command as dogtag to see if it's something to do with kerberos or something like that.
I had a quick look in the audit log just to check it wasn't selinux related but can't find anything.
Just did some further testing, if I run the command manually I can see my traffic in tcpdump connecting on port 443 to the master however when dogtag is supposidly running the scripts I don't see any connection attempts. So it looks like the script isn't actually running at least to the point where it tries to talk to the master anyway.
I dived off down the rabbit hole and thought maybe DNS isn't working for dogtag so added an entry to /etc/hosts for the master IPA server but this didn't make any difference.
For good measure I've disabled selinux. I'm now looking to see if I can crank up the logging output from dogtag to see if there is anything extra it's saying.
Any ideas welcome.
Thanks Dave
Try running:
sudo -u pkiuser /usr/libexec/ipa/ipa-pki-retrieve-key \ "caSigningCert cert-pki-ca dd4ea812-c044-41c0-93bf-ec376c732c93" \ man-fb-ipa-01.testhost.com
to run the command as pkiuser would run it.
What OS and version are you running on? There was a recent bug on Fedora 30 with similar symptoms to what you are describing (see https://pagure.io/freeipa/issue/7964 for details).
Cheers, Fraser
Ahh of course sudo I was trying su.
I'm on Centos 7.6 running freeipa 4.6.4 all from the standard yum packages.
It does look to be the exact same issue as you posted about Fedora 30.
This means that anyone running Centos 7.6 / RHEL 7.6 will be affected by this. (See below)
As a work around if I manually imported the cert into nssdb /etc/pki/pki-tomcat/alias would dogtag kick into life or is there more than this required? I only ask out of interest as I'm going to rebuild this current setup on RHEL 8 which is running IPA 4.7.1 which from what I can tell already includes the fix for this.
Thanks for your help on this. Dave
The output from running comes out as [root@man-fb-ipa-02 ~]# sudo -u pkiuser /usr/libexec/ipa/ipa-pki-retrieve-key "caSigningCert cert-pki-ca dd4ea812-c044-41c0-93bf-ec376c732c93" man-fb-ipa-01.testhost.com Traceback (most recent call last): File "/usr/libexec/ipa/ipa-pki-retrieve-key", line 39, in <module> main() File "/usr/libexec/ipa/ipa-pki-retrieve-key", line 30, in main keyfile=client_keyfile, keytab=client_keytab, File "/usr/lib/python2.7/site-packages/ipaserver/secrets/client.py", line 64, in __init__ self.kemcli = KEMClient(self._server_keys(server, realm), File "/usr/lib/python2.7/site-packages/ipaserver/secrets/client.py", line 27, in _server_keys sk = JWK(**json_decode(self.ikk.find_key(principal, KEY_USAGE_SIG))) File "/usr/lib/python2.7/site-packages/ipaserver/secrets/kem.py", line 225, in find_key return conn.get_key(usage, kid) File "/usr/lib/python2.7/site-packages/ipaserver/secrets/kem.py", line 71, in get_key conn = self.connect() File "/usr/lib/python2.7/site-packages/ipaserver/secrets/common.py", line 40, in connect conn.sasl_interactive_bind_s('', auth_tokens) File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 229, in sasl_interactive_bind_s return self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags) File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in _ldap_call result = func(*args,**kwargs) LOCAL_ERROR: {'info': 'SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available (default cache: KEYRING:persistent:17))', 'desc': 'Local error'}
I also get the ca-show failure. [root@man-fb-ipa-02 ~]# ipa ca-show vpn ipa: ERROR: Request failed with status 500: Non-2xx response from CA REST API: 500.
On Thu, Sep 05, 2019 at 10:12:10AM -0000, David Etchen via FreeIPA-users wrote:
Ahh of course sudo I was trying su.
I'm on Centos 7.6 running freeipa 4.6.4 all from the standard yum packages.
It does look to be the exact same issue as you posted about Fedora 30.
Thanks. I will need to investigate this. Maybe it was triggered by an update of some other package...
This means that anyone running Centos 7.6 / RHEL 7.6 will be affected by this. (See below)
As a work around if I manually imported the cert into nssdb /etc/pki/pki-tomcat/alias would dogtag kick into life or is there more than this required? I only ask out of interest as I'm going to rebuild this current setup on RHEL 8 which is running IPA 4.7.1 which from what I can tell already includes the fix for this.
You need not only the certificate but also the signing key. Use pk12util to export the cert and key from the one NSSDB, and import into the other **with the same nickname**.
Cheers, Fraser
Thanks for your help on this. Dave
The output from running comes out as [root@man-fb-ipa-02 ~]# sudo -u pkiuser /usr/libexec/ipa/ipa-pki-retrieve-key "caSigningCert cert-pki-ca dd4ea812-c044-41c0-93bf-ec376c732c93" man-fb-ipa-01.testhost.com Traceback (most recent call last): File "/usr/libexec/ipa/ipa-pki-retrieve-key", line 39, in <module> main() File "/usr/libexec/ipa/ipa-pki-retrieve-key", line 30, in main keyfile=client_keyfile, keytab=client_keytab, File "/usr/lib/python2.7/site-packages/ipaserver/secrets/client.py", line 64, in __init__ self.kemcli = KEMClient(self._server_keys(server, realm), File "/usr/lib/python2.7/site-packages/ipaserver/secrets/client.py", line 27, in _server_keys sk = JWK(**json_decode(self.ikk.find_key(principal, KEY_USAGE_SIG))) File "/usr/lib/python2.7/site-packages/ipaserver/secrets/kem.py", line 225, in find_key return conn.get_key(usage, kid) File "/usr/lib/python2.7/site-packages/ipaserver/secrets/kem.py", line 71, in get_key conn = self.connect() File "/usr/lib/python2.7/site-packages/ipaserver/secrets/common.py", line 40, in connect conn.sasl_interactive_bind_s('', auth_tokens) File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 229, in sasl_interactive_bind_s return self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags) File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in _ldap_call result = func(*args,**kwargs) LOCAL_ERROR: {'info': 'SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available (default cache: KEYRING:persistent:17))', 'desc': 'Local error'}
I also get the ca-show failure. [root@man-fb-ipa-02 ~]# ipa ca-show vpn ipa: ERROR: Request failed with status 500: Non-2xx response from CA REST API: 500.
On Fri, Sep 06, 2019 at 11:27:52AM +1000, Fraser Tweedale via FreeIPA-users wrote:
On Thu, Sep 05, 2019 at 10:12:10AM -0000, David Etchen via FreeIPA-users wrote:
Ahh of course sudo I was trying su.
I'm on Centos 7.6 running freeipa 4.6.4 all from the standard yum packages.
It does look to be the exact same issue as you posted about Fedora 30.
Thanks. I will need to investigate this. Maybe it was triggered by an update of some other package...
David could you please provide list of exact packages versions on the system (`rpm -qa`)?
Thanks, Fraser
Lucky I saw this early this morning as I'm about to destroy the machine. One other thing of note is that the ipa installation was done using ansible-freeipa.
Hope it helps
Dave
[root@man-fb-ipa-02 ~]# uname -a Linux man-fb-ipa-02.testhost.com 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@man-fb-ipa-02 ~]# cat /etc/centos-release CentOS Linux release 7.6.1810 (Core)
[root@man-fb-ipa-02 ~]# rpm -qa perl-Package-Constants-0.02-294.el7_6.noarch yum-3.4.3-161.el7.centos.noarch kbd-legacy-1.15.5-15.el7.noarch perl-IO-Compress-2.061-2.el7.noarch firewalld-0.5.3-5.el7.noarch bash-4.2.46-31.el7.x86_64 xorg-x11-font-utils-7.5-21.el7.x86_64 nss-softokn-freebl-3.36.0-5.el7_5.x86_64 net-tools-2.0-0.24.20131004git.el7.x86_64 python-sss-murmur-1.16.2-13.el7_6.8.x86_64 openssh-clients-7.4p1-16.el7.x86_64 copy-jdk-configs-3.3-10.el7_5.noarch audit-2.8.4-4.el7.x86_64 popt-1.13-16.el7.x86_64 mailcap-2.1.41-2.el7.noarch aic94xx-firmware-30-6.el7.noarch libattr-2.4.46-13.el7.x86_64 mod_nss-1.0.14-12.el7.x86_64 dracut-config-rescue-033-554.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 python-dateutil-1.5-7.el7.noarch keyutils-libs-1.5.8-3.el7.x86_64 python-ply-3.4-11.el7.noarch btrfs-progs-4.9.1-1.el7.x86_64 p11-kit-trust-0.23.5-3.el7.x86_64 cyrus-sasl-plain-2.1.26-23.el7.x86_64 rootfiles-8.1-11.el7.noarch libcgroup-0.41-20.el7.x86_64 iwl5000-firmware-8.83.5.1_1-69.el7.noarch policycoreutils-python-2.5-29.el7_6.1.x86_64 iwl7260-firmware-22.0.7.0-69.el7.noarch readline-6.2-10.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 iwl7265-firmware-22.0.7.0-69.el7.noarch which-2.20-7.el7.x86_64 libXrender-0.9.10-1.el7.x86_64 iwl6000-firmware-9.221.4.1-69.el7.noarch cpio-2.11-27.el7.x86_64 libXcomposite-0.4.4-4.1.el7.x86_64 grub2-common-2.02-0.76.el7.centos.1.noarch findutils-4.5.11-6.el7.x86_64 libXcursor-1.1.15-1.el7.x86_64 glibc-2.17-260.el7_6.6.x86_64 libxslt-1.1.28-5.el7.x86_64 libXxf86vm-1.1.4-1.el7.x86_64 libblkid-2.23.2-59.el7_6.1.x86_64 file-libs-5.11-35.el7.x86_64 libpath_utils-0.2.1-32.el7.x86_64 glib2-2.56.1-4.el7_6.x86_64 libaio-0.3.109-13.el7.x86_64 harfbuzz-1.7.5-2.el7.x86_64 nss-sysinit-3.36.0-7.1.el7_6.x86_64 cracklib-dicts-2.9.0-11.el7.x86_64 python-backports-1.0-8.el7.x86_64 dbus-libs-1.10.24-13.el7_6.x86_64 nss-softokn-3.36.0-5.el7_5.x86_64 python-jwcrypto-0.4.2-1.el7.noarch libssh2-1.4.3-12.el7_6.3.x86_64 libassuan-2.1.0-3.el7.x86_64 python-custodia-0.3.1-4.el7.noarch polkit-0.112-18.el7_6.1.x86_64 groff-base-1.22.2-8.el7.x86_64 python-netifaces-0.10.4-3.el7.x86_64 kernel-tools-libs-3.10.0-957.27.2.el7.x86_64 libunistring-0.9.3-9.el7.x86_64 lksctp-tools-1.0.17-2.el7.x86_64 util-linux-2.23.2-59.el7_6.1.x86_64 sysvinit-tools-2.88-14.dsf.el7.x86_64 libthai-0.1.14-9.el7.x86_64 device-mapper-event-libs-1.02.149-10.el7_6.8.x86_64 newt-0.52.15-4.el7.x86_64 sssd-common-pac-1.16.2-13.el7_6.8.x86_64 lvm2-libs-2.02.180-10.el7_6.8.x86_64 sssd-ldap-1.16.2-13.el7_6.8.x86_64 grub2-pc-2.02-0.76.el7.centos.1.x86_64 ethtool-4.8-9.el7.x86_64 libwayland-client-1.15.0-1.el7.x86_64 ipset-6.38-3.el7_6.x86_64 python-linux-procfs-0.4.9-4.el7.noarch pango-1.42.4-2.el7_6.x86_64 tuned-2.10.0-6.el7_6.4.noarch quota-4.01-17.el7.x86_64 systemd-sysv-219-62.el7_6.9.x86_64 libnetfilter_conntrack-1.0.6-1.el7_3.x86_64 gtk2-2.24.31-1.el7.x86_64 openssl-1.0.2k-16.el7_6.1.x86_64 gettext-0.19.8.1-2.el7.x86_64 xml-commons-apis-1.4.01-16.el7.noarch vim-minimal-7.4.160-6.el7_6.x86_64 xmlsec1-openssl-1.2.20-7.el7_4.x86_64 isorelax-0-0.15.release20050331.el7.noarch python-kitchen-1.1.1-5.el7.noarch pkgconfig-0.27.1-4.el7.x86_64 apache-commons-pool-1.6-9.el7.noarch libtevent-0.9.36-1.el7.x86_64 libdb-utils-5.3.21-24.el7.x86_64 hsqldb-1.8.1.3-14.el7.noarch libbasicobjects-0.1.1-32.el7.x86_64 mariadb-libs-5.5.60-1.el7_5.x86_64 resteasy-base-jaxrs-api-3.0.6-4.el7.noarch libsss_idmap-1.16.2-13.el7_6.8.x86_64 rpm-libs-4.11.3-35.el7.x86_64 objectweb-asm-3.3.1-9.el7.noarch libjpeg-turbo-1.2.90-6.el7.x86_64 python-pycurl-7.19.0-19.el7.x86_64 joda-time-2.2-3.tzdata2013c.el7.noarch fontpackages-filesystem-1.44-8.el7.noarch centos-logos-70.0.6-3.el7.centos.noarch avalon-logkit-2.1-14.el7.noarch xmlrpc-c-1.32.5-1905.svn2451.el7.x86_64 acl-2.2.51-14.el7.x86_64 jakarta-commons-httpclient-3.1-16.el7_0.noarch python-sssdconfig-1.16.2-13.el7_6.8.noarch pinentry-0.8.1-17.el7.x86_64 cal10n-0.7.7-4.el7.noarch libsss_nss_idmap-1.16.2-13.el7_6.8.x86_64 GeoIP-1.5.0-13.el7.x86_64 qdox-1.12.1-10.el7.noarch python-lxml-3.2.1-4.el7.x86_64 dmidecode-3.1-2.el7.x86_64 xpp3-1.1.3.8-11.el7.noarch certmonger-0.78.4-10.el7.x86_64 apache-commons-daemon-1.0.13-7.el7.x86_64 svrcore-4.1.3-2.el7.x86_64 hardlink-1.0-19.el7.x86_64 glassfish-jaxb-2.2.5-6.el7.noarch avahi-libs-0.6.31-19.el7.x86_64 libdaemon-0.14-7.el7.x86_64 resteasy-base-atom-provider-3.0.6-4.el7.noarch samba-common-libs-4.8.3-6.el7_6.x86_64 libpipeline-1.2.3-3.el7.x86_64 apache-commons-dbcp-1.4-17.el7.noarch libxshmfence-1.2-1.el7.x86_64 libmspack-0.5-0.6.alpha.el7.x86_64 jsr-311-1.1.1-6.el7.noarch libipa_hbac-1.16.2-13.el7_6.8.x86_64 pki-tools-10.5.9-13.el7_6.x86_64 libSM-1.2.2-2.el7.x86_64 pki-server-10.5.9-13.el7_6.noarch mesa-libgbm-18.0.5-4.el7_6.x86_64 389-ds-base-1.3.8.4-25.1.el7_6.x86_64 javapackages-tools-3.4.1-11.el7.noarch bind-9.9.4-74.el7_6.2.x86_64 xsom-0-10.20110809svn.el7.noarch iputils-20160308-10.el7.x86_64 bind-pkcs11-9.9.4-74.el7_6.2.x86_64 tomcat-jsp-2.2-api-7.0.76-9.el7_6.noarch libpcap-1.5.3-11.el7.x86_64 dejavu-fonts-common-2.33-6.el7.noarch tcpdump-4.9.2-3.el7.x86_64 open-sans-fonts-1.10-1.el7.noarch openssh-7.4p1-16.el7.x86_64 python-qrcode-core-5.0.1-1.el7.noarch perl-Pod-Perldoc-3.20-4.el7.noarch perl-Pod-Usage-1.63-3.el7.noarch wpa_supplicant-2.6-12.el7.x86_64 perl-Socket-2.010-4.el7.x86_64 alsa-firmware-1.0.28-2.el7.noarch perl-Exporter-5.68-3.el7.noarch dbus-python-1.1.1-9.el7.x86_64 perl-threads-shared-1.43-6.el7.x86_64 plymouth-core-libs-0.8.9-0.31.20140113.el7.centos.x86_64 perl-Scalar-List-Utils-1.27-248.el7.x86_64 pth-2.0.7-23.el7.x86_64 perl-Data-Dumper-2.145-3.el7.x86_64 gpgme-1.3.2-5.el7.x86_64 yum-plugin-fastestmirror-1.1.31-50.el7.noarch linux-firmware-20180911-69.git85c5d90.el7.noarch firewalld-filesystem-0.5.3-5.el7.noarch ncurses-base-5.9-14.20130511.el7_4.noarch kbd-1.15.5-15.el7.x86_64 chkconfig-1.7.4-1.el7.x86_64 setup-2.8.71-10.el7.noarch basesystem-10.0-7.el7.centos.noarch zlib-1.2.7-18.el7.x86_64 openssh-server-7.4p1-16.el7.x86_64 authconfig-6.2.8-30.el7.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64 postfix-2.10.1-7.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 irqbalance-1.0.7-11.el7.x86_64 gawk-4.0.2-4.el7_3.1.x86_64 rsyslog-8.24.0-34.el7.x86_64 libffi-3.0.13-18.el7.x86_64 biosdevname-0.7.3-1.el7.x86_64 libacl-2.2.51-14.el7.x86_64 parted-3.1-29.el7.x86_64 pcre-8.32-17.el7.x86_64 sg3_utils-1.37-17.el7.x86_64 sed-4.2.2-5.el7.x86_64 man-db-2.6.3-11.el7.x86_64 p11-kit-0.23.5-3.el7.x86_64 e2fsprogs-1.42.9-13.el7.x86_64 gmp-6.0.0-15.el7.x86_64 passwd-0.79-4.el7.x86_64 libtasn1-4.10-1.el7.x86_64 ca-certificates-2018.2.22-70.0.el7_5.noarch iwl1000-firmware-39.31.5.1-69.el7.noarch coreutils-8.22-23.el7.x86_64 iwl4965-firmware-228.61.2.24-69.el7.noarch iwl100-firmware-39.31.5.1-69.el7.noarch iwl6000g2b-firmware-17.168.5.2-69.el7.noarch bzip2-libs-1.0.6-13.el7.x86_64 iwl5150-firmware-8.24.2.2-69.el7.noarch libdb-5.3.21-24.el7.x86_64 iwl3160-firmware-22.0.7.0-69.el7.noarch elfutils-libelf-0.172-2.el7.x86_64 iwl3945-firmware-15.32.2.9-69.el7.noarch libgcrypt-1.5.3-14.el7.x86_64 iwl6050-firmware-41.28.5.1-69.el7.noarch libcap-ng-0.7.5-4.el7.x86_64 iwl6000g2a-firmware-17.168.5.3-69.el7.noarch gzip-1.5-10.el7.x86_64 iwl135-firmware-18.168.6.1-69.el7.noarch expat-2.1.0-10.el7_3.x86_64 libgcc-4.8.5-36.el7_6.2.x86_64 lua-5.1.4-15.el7.x86_64 grub2-pc-modules-2.02-0.76.el7.centos.1.noarch diffutils-3.3-4.el7.x86_64 glibc-common-2.17-260.el7_6.6.x86_64 cracklib-2.9.0-11.el7.x86_64 systemd-libs-219-62.el7_6.9.x86_64 libuuid-2.23.2-59.el7_6.1.x86_64 shared-mime-info-1.8-4.el7.x86_64 openssl-libs-1.0.2k-16.el7_6.1.x86_64 file-5.11-35.el7.x86_64 libmount-2.23.2-59.el7_6.1.x86_64 libmnl-1.0.3-7.el7.x86_64 shadow-utils-4.1.5.1-25.el7_6.1.x86_64 pciutils-libs-3.5.1-3.el7.x86_64 python-2.7.5-80.el7_6.x86_64 libcroco-0.6.12-4.el7.x86_64 nss-3.36.0-7.1.el7_6.x86_64 libpwquality-1.2.3-5.el7.x86_64 NetworkManager-libnm-1.12.0-10.el7_6.x86_64 libnl3-cli-3.2.28-4.el7.x86_64 nss-tools-3.36.0-7.1.el7_6.x86_64 python-perf-3.10.0-957.27.2.el7.x86_64 libcurl-7.29.0-51.el7_6.3.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 dbus-1.10.24-13.el7_6.x86_64 xz-5.2.2-1.el7.x86_64 NetworkManager-1.12.0-10.el7_6.x86_64 tar-1.26-35.el7.x86_64 cronie-1.4.11-20.el7_6.x86_64 libteam-1.27-6.el7_6.1.x86_64 libidn-1.28-4.el7.x86_64 libsmartcols-2.23.2-59.el7_6.1.x86_64 jansson-2.10-1.el7.x86_64 policycoreutils-2.5-29.el7_6.1.x86_64 libnfnetlink-1.0.1-4.el7.x86_64 device-mapper-1.02.149-10.el7_6.8.x86_64 slang-2.2.4-11.el7.x86_64 grub2-tools-minimal-2.02-0.76.el7.centos.1.x86_64 lz4-1.7.5-2.el7.x86_64 device-mapper-event-1.02.149-10.el7_6.8.x86_64 gdbm-1.10-8.el7.x86_64 selinux-policy-3.13.1-229.el7_6.15.noarch grub2-tools-extra-2.02-0.76.el7.centos.1.x86_64 hostname-3.13-3.el7.x86_64 ipset-libs-6.38-3.el7_6.x86_64 libselinux-python-2.5-14.1.el7.x86_64 bind-libs-lite-9.9.4-74.el7_6.2.x86_64 yum-metadata-parser-1.1.4-10.el7.x86_64 grub2-2.02-0.76.el7.centos.1.x86_64 pyliblzma-0.5.3-11.el7.x86_64 lvm2-2.02.180-10.el7_6.8.x86_64 python-schedutils-0.4-6.el7.x86_64 NetworkManager-team-1.12.0-10.el7_6.x86_64 python-iniparse-0.4-9.el7.noarch NetworkManager-tui-1.12.0-10.el7_6.x86_64 pyxattr-0.5.1-5.el7.x86_64 microcode_ctl-2.1-47.5.el7_6.x86_64 iptables-1.4.21-28.el7.x86_64 curl-7.29.0-51.el7_6.3.x86_64 gettext-libs-0.19.8.1-2.el7.x86_64 xfsprogs-4.5.0-19.el7_6.x86_64 less-458-9.el7.x86_64 device-mapper-persistent-data-0.7.3-3.el7.x86_64 python-chardet-2.2.1-1.el7_1.noarch libxml2-python-2.9.1-6.el7_2.3.x86_64 gobject-introspection-1.56.1-1.el7.x86_64 libtalloc-2.1.13-1.el7.x86_64 fuse-2.9.2-11.el7.x86_64 libcollection-0.7.0-32.el7.x86_64 grubby-8.28-25.el7.x86_64 libref_array-0.1.5-32.el7.x86_64 fipscheck-lib-1.4.1-6.el7.x86_64 libldb-1.3.4-1.el7.x86_64 libdhash-0.5.0-32.el7.x86_64 python-six-1.9.0-2.el7.noarch rpm-4.11.3-35.el7.x86_64 python2-pyasn1-0.1.9-7.el7.noarch libuser-0.60-9.el7.x86_64 ipa-common-4.6.4-10.el7.centos.6.noarch python-urlgrabber-3.10-9.el7.noarch python-dns-1.12.0-4.20150617git465785f.el7.noarch binutils-2.27-34.base.el7.x86_64 openldap-clients-2.4.44-21.el7_6.x86_64 alsa-lib-1.1.6-2.el7.x86_64 mesa-libglapi-18.0.5-4.el7_6.x86_64 python-enum34-1.0.4-1.el7.noarch libss-1.42.9-13.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64 mozjs17-17.0.0-20.el7.x86_64 psmisc-22.20-15.el7.x86_64 libicu-50.1.2-17.el7.x86_64 python-ipaddress-1.0.16-2.el7.noarch apr-1.4.8-3.el7_4.1.x86_64 apr-util-1.5.2-6.el7.x86_64 xmlrpc-c-client-1.32.5-1905.svn2451.el7.x86_64 libsemanage-2.5-14.el7.x86_64 rpcbind-0.2.0-47.el7.x86_64 libutempter-1.1.6-4.el7.x86_64 libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64 libfastjson-0.99.4-3.el7.x86_64 libsemanage-python-2.5-14.el7.x86_64 libndp-1.2-7.el7.x86_64 pyusb-1.0.0-0.11.b1.el7.noarch libseccomp-2.3.1-3.el7.x86_64 cups-libs-1.6.3-35.el7.x86_64 qrencode-libs-3.4.1-3.el7.x86_64 samba-client-libs-4.8.3-6.el7_6.x86_64 numactl-libs-2.0.9-7.el7.x86_64 keyutils-1.5.8-3.el7.x86_64 libestr-0.1.9-2.el7.x86_64 python-nss-0.16.0-3.el7.x86_64 sg3_utils-libs-1.37-17.el7.x86_64 libICE-1.0.9-9.el7.x86_64 json-c-0.11-4.el7_0.x86_64 gperftools-libs-2.6.1-1.el7.x86_64 kpartx-0.4.9-123.el7.x86_64 libsss_autofs-1.16.2-13.el7_6.8.x86_64 389-ds-base-libs-1.3.8.4-25.1.el7_6.x86_64 kmod-20-23.el7.x86_64 krb5-workstation-1.15.1-37.el7_6.x86_64 cryptsetup-libs-2.0.3-3.el7.x86_64 python-yubico-1.2.3-1.el7.noarch elfutils-default-yama-scope-0.172-2.el7.noarch polkit-pkla-compat-0.1-4.el7.x86_64 hwdata-0.252-9.1.el7.x86_64 initscripts-9.49.46-1.el7.x86_64 os-prober-1.58-9.el7.x86_64 crontabs-1.11-6.20121102git.el7.noarch dhcp-common-4.2.5-68.el7.centos.1.x86_64 dracut-network-033-554.el7.x86_64 pciutils-3.5.1-3.el7.x86_64 libdrm-2.4.91-3.el7.x86_64 fxload-2002_04_11-16.el7.x86_64 alsa-tools-firmware-1.1.0-1.el7.x86_64 dbus-glib-0.100-7.el7.x86_64 python-slip-dbus-0.4.0-4.el7.noarch python-pyudev-0.15-9.el7.noarch plymouth-scripts-0.8.9-0.31.20140113.el7.centos.x86_64 virt-what-1.18-4.el7.x86_64 gnupg2-2.0.22-5.el7_5.x86_64 rpm-python-4.11.3-35.el7.x86_64 perl-DB_File-1.830-6.el7.x86_64 perl-Compress-Raw-Bzip2-2.061-3.el7.x86_64 perl-Compress-Raw-Zlib-2.061-4.el7.x86_64 perl-IO-Zlib-1.10-294.el7_6.noarch libfontenc-1.1.3-3.el7.x86_64 jbigkit-libs-2.0-11.el7.x86_64 krb5-pkinit-1.15.1-37.el7_6.x86_64 quota-nls-4.01-17.el7.noarch bind-utils-9.9.4-74.el7_6.2.x86_64 audit-libs-python-2.8.4-4.el7.x86_64 tzdata-java-2019b-1.el7.noarch hicolor-icon-theme-0.12-7.el7.noarch xorg-x11-fonts-Type1-7.5-9.el7.noarch httpd-2.4.6-89.el7.centos.1.x86_64 mod_auth_gssapi-1.5.1-5.el7.x86_64 mod_lookup_identity-1.0.0-1.el7.x86_64 pixman-0.34.0-1.el7.x86_64 libX11-common-1.6.5-2.el7.noarch fribidi-1.0.2-1.el7.x86_64 python-pycparser-2.14-1.el7.noarch python-idna-2.4-1.el7.noarch python-IPy-0.75-6.el7.noarch autogen-libopts-5.18-5.el7.x86_64 words-3.0-22.el7.noarch setools-libs-3.3.8-4.el7.x86_64 augeas-libs-1.4.0-6.el7_6.1.x86_64 nuxwdog-1.0.3-8.el7.x86_64 libxcb-1.13-1.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXfixes-5.0.3-1.el7.x86_64 libXdamage-1.1.4-4.1.el7.x86_64 gdk-pixbuf2-2.36.12-3.el7.x86_64 libXtst-1.2.3-1.el7.x86_64 libXft-2.3.2-2.el7.x86_64 libXinerama-1.1.3-2.1.el7.x86_64 libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64 giflib-4.1.6-9.el7.x86_64 libini_config-1.3.1-32.el7.x86_64 graphite2-1.3.10-1.el7_3.x86_64 pcsc-lite-libs-1.8.8-8.el7.x86_64 ntp-4.2.6p5-28.el7.centos.x86_64 python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch python2-cryptography-1.7.2-2.el7.x86_64 python-urllib3-1.10.2-5.el7.noarch pki-base-10.5.9-13.el7_6.noarch custodia-0.3.1-4.el7.noarch http-parser-2.7.1-5.el7_4.x86_64 python2-ipalib-4.6.4-10.el7.centos.6.noarch python2-ipaserver-4.6.4-10.el7.centos.6.noarch java-1.8.0-openjdk-headless-1.8.0.222.b10-0.el7_6.x86_64 c-ares-1.10.0-3.el7.x86_64 libsss_sudo-1.16.2-13.el7_6.8.x86_64 sssd-krb5-common-1.16.2-13.el7_6.8.x86_64 sssd-ad-1.16.2-13.el7_6.8.x86_64 sssd-krb5-1.16.2-13.el7_6.8.x86_64 sssd-proxy-1.16.2-13.el7_6.8.x86_64 sssd-dbus-1.16.2-13.el7_6.8.x86_64 libglvnd-egl-1.0.1-0.8.git5baa1e5.el7.x86_64 cairo-1.15.12-3.el7.x86_64 hesiod-3.2.1-3.el7.x86_64 tcp_wrappers-7.6-77.el7.x86_64 nfs-utils-1.3.0-0.61.el7.x86_64 atk-2.28.1-1.el7.x86_64 java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64 apache-commons-lang-2.6-15.el7.noarch libstdc++-4.8.5-36.el7_6.2.x86_64 xml-commons-resolver-1.2-15.el7.noarch libgomp-4.8.5-36.el7_6.2.x86_64 xerces-j2-2.11.0-17.el7_0.noarch msv-xsdlib-2013.5.1-7.el7.noarch jss-4.4.4-5.el7_6.x86_64 apache-commons-collections-3.2.1-22.el7_2.noarch rngom-201103-0.8.20120119svn.el7.noarch apache-commons-io-2.4-12.el7.noarch codemodel-2.6-9.el7.noarch scannotation-1.0.3-0.7.r12.el7.noarch jing-20091111-14.el7.noarch args4j-2.0.16-13.el7.noarch joda-convert-1.3-5.el7.noarch httpcomponents-core-4.2.4-6.el7.noarch log4j-1.2.17-16.el7_4.noarch apache-commons-logging-1.1.2-7.el7.noarch httpcomponents-client-4.2.5-5.el7_0.noarch glassfish-dtd-parser-1.2-0.8.20120120svn.el7.noarch ecj-4.5.2-3.el7.x86_64 slf4j-1.7.4-4.el7_4.noarch easymock2-2.5.2-12.el7.noarch hamcrest-1.3-6.el7.noarch ws-jaxme-0.5.2-10.el7.noarch jdom-1.1.3-6.el7.noarch dom4j-1.6.1-20.el7.noarch glassfish-jaxb-api-2.2.7-4.el7.noarch glassfish-fastinfoset-1.2.12-9.el7.noarch resteasy-base-jaxb-provider-3.0.6-4.el7.noarch resteasy-base-jaxrs-3.0.6-4.el7.noarch resteasy-base-client-3.0.6-4.el7.noarch geronimo-jta-1.1.1-17.el7.noarch tomcat-lib-7.0.76-9.el7_6.noarch tomcatjss-7.2.1-8.el7_6.noarch jackson-1.9.4-7.el7.noarch pki-base-java-10.5.9-13.el7_6.noarch regexp-1.5-13.el7.noarch velocity-1.7-10.el7.noarch pki-ca-10.5.9-13.el7_6.noarch oddjob-mkhomedir-0.31.5-4.el7.x86_64 cyrus-sasl-md5-2.1.26-23.el7.x86_64 python-javapackages-3.4.1-11.el7.noarch slapi-nis-0.56.0-8.el7.x86_64 relaxngDatatype-1.0-11.el7.noarch libitm-4.8.5-36.el7_6.2.x86_64 bea-stax-api-1.2.0-9.el7.noarch opencryptoki-libs-3.10.0-2.el7.x86_64 stax2-api-3.1.1-10.el7.noarch bind-pkcs11-utils-9.9.4-74.el7_6.2.x86_64 jvnet-parent-4-2.el7.noarch opencryptoki-swtok-3.10.0-2.el7.x86_64 tomcat-el-2.2-api-7.0.76-9.el7_6.noarch bind-dyndb-ldap-11.1-4.el7.x86_64 sssd-client-1.16.2-13.el7_6.8.x86_64 ldns-1.6.16-10.el7.x86_64 dejavu-sans-fonts-2.33-6.el7.noarch ipa-server-dns-4.6.4-10.el7.centos.6.noarch fontawesome-fonts-4.1.0-2.el7.noarch mlocate-0.26-8.el7.x86_64 python-kdcproxy-0.3.2-1.el7.noarch python2-pyasn1-modules-0.1.9-7.el7.noarch perl-parent-0.225-244.el7.noarch perl-podlators-2.5.1-3.el7.noarch perl-Text-ParseWords-3.29-4.el7.noarch perl-Encode-2.51-7.el7.x86_64 perl-libs-5.16.3-294.el7_6.x86_64 perl-threads-1.87-4.el7.x86_64 perl-Storable-2.45-3.el7.x86_64 perl-Filter-1.49-3.el7.x86_64 perl-Time-Local-1.2300-2.el7.noarch perl-Carp-1.26-244.el7.noarch perl-File-Temp-0.23.01-3.el7.noarch perl-PathTools-3.40-5.el7.x86_64 perl-Pod-Simple-3.28-4.el7.noarch perl-5.16.3-294.el7_6.x86_64 perl-Mozilla-LDAP-1.5.3-12.el7.x86_64 pygpgme-0.3-9.el7.x86_64 perl-NetAddr-IP-4.069-3.el7.x86_64 kbd-misc-1.15.5-15.el7.noarch perl-Archive-Tar-1.92-2.el7.noarch kernel-3.10.0-957.el7.x86_64 ncurses-5.9-14.20130511.el7_4.x86_64 libtiff-4.0.3-27.el7_3.x86_64 open-vm-tools-10.2.5-3.el7.x86_64 filesystem-3.2-25.el7.x86_64 bind-libs-9.9.4-74.el7_6.2.x86_64 nspr-4.19.0-1.el7_5.x86_64 softhsm-2.1.0-2.el7.x86_64 lshw-B.02.18-12.el7.x86_64 info-5.1-5.el7.x86_64 ttmkfdir-3.0.9-42.el7.x86_64 libsepol-2.5-10.el7.x86_64 mod_wsgi-3.4-18.el7.x86_64 chrony-3.2-2.el7.x86_64 libcap-2.22-9.el7.x86_64 mod_session-2.4.6-89.el7.centos.1.x86_64 iprutils-2.4.16.1-1.el7.x86_64 grep-2.20-3.el7.x86_64 python-netaddr-0.7.5-9.el7.noarch sudo-1.8.23-3.el7.x86_64 libverto-0.2.5-4.el7.x86_64 python-cffi-1.6.0-5.el7.x86_64 libsysfs-2.1.0-16.el7.x86_64 centos-release-7-6.1810.2.el7.centos.x86_64 checkpolicy-2.5-8.el7.x86_64 ivtv-firmware-20080701-26.el7.noarch xz-libs-5.2.2-1.el7.x86_64 krb5-server-1.15.1-37.el7_6.x86_64 iwl2030-firmware-18.168.6.1-69.el7.noarch libxml2-2.9.1-6.el7_2.3.x86_64 python-augeas-0.5.0-2.el7.noarch iwl105-firmware-18.168.6.1-69.el7.noarch libgpg-error-1.12-3.el7.x86_64 libX11-1.6.5-2.el7.x86_64 iwl2000-firmware-18.168.6.1-69.el7.noarch audit-libs-2.8.4-4.el7.x86_64 libXi-1.7.9-1.el7.x86_64 gpg-pubkey-f4a80eb5-53a7ff4b sqlite-3.7.17-8.el7.x86_64 gtk-update-icon-cache-3.22.30-3.el7.x86_64 tzdata-2019b-1.el7.noarch libnl3-3.2.28-4.el7.x86_64 libXrandr-1.5.1-2.el7.x86_64 nss-util-3.36.0-1.1.el7_6.x86_64 mesa-libGL-18.0.5-4.el7_6.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libtool-ltdl-2.4.2-22.el7_3.x86_64 gssproxy-0.7.0-21.el7.x86_64 python-libs-2.7.5-80.el7_6.x86_64 xmlsec1-1.2.20-7.el7_4.x86_64 ntpdate-4.2.6p5-28.el7.centos.x86_64 nss-pem-1.0.3-5.el7_6.1.x86_64 pam-1.1.8-22.el7.x86_64 python-setuptools-0.9.8-7.el7.noarch openldap-2.4.44-21.el7_6.x86_64 python-requests-2.6.0-1.el7_1.noarch systemd-219-62.el7_6.9.x86_64 kmod-libs-20-23.el7.x86_64 ipa-server-common-4.6.4-10.el7.centos.6.noarch cronie-anacron-1.4.11-20.el7_6.x86_64 e2fsprogs-libs-1.42.9-13.el7.x86_64 python2-ipaclient-4.6.4-10.el7.centos.6.noarch teamd-1.27-6.el7_6.1.x86_64 libedit-3.0-12.20121213cvs.el7.x86_64 nuxwdog-client-java-1.0.3-8.el7.x86_64 device-mapper-libs-1.02.149-10.el7_6.8.x86_64 tcp_wrappers-libs-7.6-77.el7.x86_64 sssd-common-1.16.2-13.el7_6.8.x86_64 grub2-tools-2.02-0.76.el7.centos.1.x86_64 lzo-2.06-8.el7.x86_64 sssd-ipa-1.16.2-13.el7_6.8.x86_64 freetype-2.8-12.el7_6.1.x86_64 python-decorator-3.4.0-3.el7.noarch sssd-1.16.2-13.el7_6.8.x86_64 bind-license-9.9.4-74.el7_6.2.noarch python-slip-0.4.0-4.el7.noarch mesa-libEGL-18.0.5-4.el7_6.x86_64 selinux-policy-targeted-3.13.1-229.el7_6.15.noarch newt-python-0.52.15-4.el7.x86_64 autofs-5.0.7-99.el7.x86_64 kernel-tools-3.10.0-957.27.2.el7.x86_64 python-configobj-4.7.2-7.el7.noarch ipa-client-4.6.4-10.el7.centos.6.x86_64 kexec-tools-2.0.15-21.el7_6.4.x86_64 apache-commons-codec-1.8-7.el7.noarch iproute-4.11.0-14.el7_6.2.x86_64 xalan-j2-2.7.1-23.el7.noarch kernel-3.10.0-957.27.2.el7.x86_64 ldapjdk-4.19-5.el7.noarch yum-utils-1.1.31-50.el7.noarch python-gobject-base-3.22.0-1.el7_4.1.x86_64 javassist-3.16.1-10.el7.noarch libtdb-1.3.15-1.el7.x86_64 fipscheck-1.4.1-6.el7.x86_64 geronimo-jms-1.1.1-19.el7.noarch libsss_certmap-1.16.2-13.el7_6.8.x86_64 msv-msv-2013.5.1-7.el7.noarch python-ldap-2.4.15-2.el7.x86_64 txw2-20110809-8.el7.noarch cyrus-sasl-gssapi-2.1.26-23.el7.x86_64 logrotate-3.8.6-17.el7.x86_64 javamail-1.4.6-8.el7.noarch libtirpc-0.2.4-0.15.el7.x86_64 libselinux-utils-2.5-14.1.el7.x86_64 avalon-framework-4.3-10.el7.noarch python-gssapi-1.2.0-3.el7.x86_64 make-3.82-23.el7.x86_64 istack-commons-2.17-4.el7.noarch ipa-client-common-4.6.4-10.el7.centos.6.noarch snappy-1.1.0-3.el7.x86_64 antlr-tool-2.7.7-30.el7.noarch samba-common-4.8.3-6.el7_6.noarch libpng-1.5.13-7.el7_2.x86_64 junit-4.11-8.el7.noarch oddjob-0.31.5-4.el7.x86_64 ustr-1.0.4-16.el7.x86_64 jaxen-1.1.3-11.el7.noarch libverto-tevent-0.2.5-4.el7.x86_64 stax-ex-1.7.1-6.el7.noarch libwayland-server-1.15.0-1.el7.x86_64 fuse-libs-2.9.2-11.el7.x86_64 jboss-annotations-1.1-api-1.0.1-0.6.20120212git76e1a2.el7.noarch libwbclient-4.8.3-6.el7_6.x86_64 libdnet-1.12-13.1.el7.x86_64 apache-commons-cli-1.2-13.el7.noarch libkadm5-1.15.1-37.el7_6.x86_64 lsscsi-0.27-6.el7.x86_64 tomcat-7.0.76-9.el7_6.noarch systemd-python-219-62.el7_6.9.x86_64 procps-ng-3.3.10-23.el7.x86_64 resteasy-base-jackson-provider-3.0.6-4.el7.noarch python-libipa_hbac-1.16.2-13.el7_6.8.x86_64 dracut-033-554.el7.x86_64 bcel-5.2-18.el7.noarch libsmbclient-4.8.3-6.el7_6.x86_64 elfutils-libs-0.172-2.el7.x86_64 pki-kra-10.5.9-13.el7_6.noarch httpd-tools-2.4.6-89.el7.centos.1.x86_64 ipa-server-4.6.4-10.el7.centos.6.x86_64 tomcat-servlet-3.0-api-7.0.76-9.el7_6.noarch bind-pkcs11-libs-9.9.4-74.el7_6.2.x86_64 bea-stax-1.2.0-9.el7.noarch opencryptoki-3.10.0-2.el7.x86_64 jakarta-oro-2.0.8-16.el7.noarch opendnssec-1.4.7-4.el7.x86_64 fontconfig-2.13.0-4.3.el7.x86_64 dhcp-libs-4.2.5-68.el7.centos.1.x86_64 jasper-libs-1.900.1-33.el7.x86_64 dhclient-4.2.5-68.el7.centos.1.x86_64 perl-HTTP-Tiny-0.033-3.el7.noarch perl-Pod-Escapes-1.04-294.el7_6.noarch libpciaccess-0.14-1.el7.x86_64 perl-macros-5.16.3-294.el7_6.x86_64 ebtables-2.0.10-16.el7.x86_64 perl-Time-HiRes-1.9725-3.el7.x86_64 perl-constant-1.27-2.el7.noarch python-firewall-0.5.3-5.el7.noarch perl-File-Path-2.09-2.el7.noarch plymouth-0.8.9-0.31.20140113.el7.centos.x86_64 perl-Getopt-Long-2.40-3.el7.noarch rpm-build-libs-4.11.3-35.el7.x86_64
So just to add it seems that the 2nd IPA server hasn't managed to get the subCA cert & key as when I check the nssdb they aren't present on the 2nd IPA server. (See below)
Running the command as my own user /usr/libexec/ipa/ipa-pki-retrieve-key "caSigningCert cert-pki-ca dd4ea812-c044-41c0-93bf-ec376c732c93" man-fb-ipa-01.testhost.com returns with what looks like a JSON response with certificate and wrapped_key attributes which corespond to the subCA.
The question now is why does dogtag not get a response / thinks that it did not get a response?
Master IPA Server certutil -L -d sql:/etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca dd4ea812-c044-41c0-93bf-ec376c732c93 u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu auditSigningCert cert-pki-ca u,u,Pu Server-Cert cert-pki-ca u,u,u
Replica IPA Server certutil -L -d sql:/etc/pki/pki-tomcat/alias
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu auditSigningCert cert-pki-ca u,u,Pu Server-Cert cert-pki-ca u,u,u
freeipa-users@lists.fedorahosted.org