Already went through that page several times, All checks passed. All certs are good. none are expired. The cert in NSS is the same in LDAP. No errors communicating/logging in. It is the lack of errors that is the most troubling.

I know what is not the issue. No certs are expired expires: 2020-01-13 13:27:04 UTC expires: 2020-01-02 13:25:21 UTC expires: 2020-01-02 13:25:20 UTC expires: 2020-01-02 13:25:20 UTC expires: 2038-01-12 13:25:20 UTC expires: 2020-01-02 13:25:38 UTC expires: 2020-01-02 13:25:20 UTC expires: 2020-01-13 13:26:31 UTC expires: 2020-01-13 13:26:54 UTC trusts are good Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u and Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI SERVER IPA CA CT,C,C Server-Cert u,u,u I have moved system time back to a day prior to failure, restarted certmonger, then tried to start everything again. No go. Removed custom ldap Default group objectclasses Reverted CS.cfg to an archive Re-installed all the PKI rpms Checked file permissions Changed from SSL + Cert auth in CS.cfg to basic auth - spoiler, no difference. I do not want to rebuild the domain. Its looking more and more like I may have to. I may try the recover/replace CA steps I found in my searches.

This is on all 4 systems having the issue ipa --version VERSION: 4.6.4, API_VERSION: 2.229 When system was updated ipa-server-upgrade was ran, and it did complete successful 2018-12-19T23:34:26Z INFO The ipa-server-upgrade command was successful Running the command fails now, as the CA won't start, which is expected.

Attached is the logs On Tue, Jan 1, 2019 at 7:36 PM Fraser Tweedale <ftweedal(a)redhat.com&gt; wrote: -- Jason Wood

Jason, I've seen this caused by a replication conflict that prevents the CA certificate profiles from loading completely during start-up. Can you double-check that there are really no conflicts in ou=certificateprofiles,ou=ca,o=ipaca, just to confirm? Try this: 1. Check for replication conflicts in ou=certificateprofiles,ou=ca,o=ipaca: # ldapsearch -xLLL -D "cn=Directory Manager" -W -b ou=certificateprofiles,ou=ca,o=ipaca '(|(objectclass=*)(&(objectclass=ldapsubentry)(nsds5ReplConflict=*))' 2. If any objects are found, remove them and then try starting IPA: # ldapdelete -x -D "cn=Directory Manager" -W <dn of replication conflict object>

Jason, Yes, bad search filter there - apologies. This one is better: # ldapsearch -xLLL -D "cn=Directory Manager" -W -b ou=certificateprofiles,ou=ca,o=ipaca '(&(nsds5ReplConflict=*)(objectclass=ldapsubentry))' The base DN you want to specify is 'ou=certificateprofiles,ou=ca,o=ipaca'. This is the area from which the profile monitor thread obtains data.

Jason - Yes, all of these entries with +nsuniqueid=... embedded in the DN are conflict entries. Try removing each with: # ldapdelete -D "cn=directory manager" -W <DN of conflict entry> Once they are all removed, try restarting IPA.

On Wed, Feb 06, 2019 at 10:09:00AM -0500, Jason L Wood via FreeIPA-users wrote: There is a ticket for this issue: https://pagure.io/dogtagpki/issue/3078 I have a work-in-progress patch for the issue but it needs a bit more work and testing. Cheers, Fraser > > Very much appreciative with helping with this issue. > > > Jason Wood > > On 2/5/2019 3:10 PM, Marco Rhodes via FreeIPA-users wrote: > > Jason - > > > > Yes, all of these entries with +nsuniqueid=... embedded in the DN are conflict entries. > > > > Try removing each with: > > > > # ldapdelete -D "cn=directory manager" -W <DN of conflict entry> > > > > Once they are all removed, try restarting IPA. > > _______________________________________________ > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...

So I have an expired cert somewhere. Or something really weird. Setting system time to 10/01/2018 PKI-Tomcat starts. Restarted certmonger and dirsrv. Moved date to 11/01/2018, restarted certmonger, dirsrv and pki-tomcat. pki-tomcat started. Moved date to 12/01/2018, restarted services, tomcat fails to start. This is on a replica, not the master. Hmmmm? OK. Moved date to 10/17/2018, restarted services. Starting with certmonger. Did work. did certlist and there were some certs on that systems that were different start/end dates. I thought, if pki-tomcatd started then I should make this my master, which I did. tomcat may start, but it gives back 500 errors. Nothing I could do would get it to work. Moved the time back to current and right back to the same. The master, didn't work no matter what I did. I keep coming back, that I have ZERO errors. Nothing. No where. Since the OS update and the ipa-server-update ran without issue during the OS update, and the issue only happened after the system was rebooted, then some configuration some where was changed to the point that it effects this. What that is, I do not know. I have not found any added or removed settings in any system files comparing it to a system that upgraded fine. Maybe something in the LDAP db that I'm not seeing. I am very LDAP challenged, so there is a good possibility I missed something. At this point I've spent way too much time on this without resolution. Are there any docs/howto's on replacing a domain. I do not want to lose the config or anything. Just basically spin up new servers, copy the data, and connect the clients to it all on the same domain. I would rather fix whatever broke, but that isn't looking like it will happen.

Log isn't cut off, that is where it stops. It never reaches selftests. From the debug logs, it shows that it is connecting with the cert and there are no errors there. This leads me to believe the certs are fine, certmonger is doing what it is supposed to, I think the issue could be in the LDAP db. We did disable anonymous binds. I've changed that to allow rootdse, that did not effect anything. Also replaced the server.xml and CA.cfg with backups, same outcome. All the certs look good, valid dates, don't expire for another year certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca' Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=IPA.EXAMPLE.NET" Validity: Not Before: Fri Jan 12 13:25:20 2018 Not After : Tue Jan 12 13:25:20 2038 Subject: "CN=Certificate Authority,O=IPA.EXAMPLE.NET" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: c0:...:5d Exponent: 65537 (0x10001) Signed Extensions: Name: Certificate Authority Key Identifier Key ID: f1:...:2f Name: Certificate Basic Constraints Critical: True Data: Is a CA with no maximum path length. Name: Certificate Key Usage Critical: True Usages: Digital Signature Non-Repudiation Certificate Signing CRL Signing Name: Certificate Subject Key ID Data: f1:...:2f Name: Authority Information Access Method: PKIX Online Certificate Status Protocol Location: URI: "http://ipa-ca.IPA.EXAMPLE.NET/ca/ocsp" Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Signature: a8:...:4e Fingerprint (SHA-256): 6E:...:32 Fingerprint (SHA1): 98:...:0F Mozilla-CA-Policy: false (attribute missing) Certificate Trust Flags: SSL Flags: Valid CA Trusted CA User Trusted Client CA Email Flags: Valid CA Trusted CA User Object Signing Flags: Valid CA Trusted CA User certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'ocspSigningCert cert-pki-ca' Certificate: Data: Version: 3 (0x2) Serial Number: 2 (0x2) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=IPA.EXAMPLE.NET" Validity: Not Before: Fri Jan 12 13:25:20 2018 Not After : Thu Jan 02 13:25:20 2020 Subject: "CN=OCSP Subsystem,O=IPA.EXAMPLE.NET" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: 9f:...:5b Exponent: 65537 (0x10001) Signed Extensions: Name: Certificate Authority Key Identifier Key ID: f1:...:2f Name: Certificate Key Usage Critical: True Usages: Digital Signature Non-Repudiation Certificate Signing CRL Signing Name: Authority Information Access Method: PKIX Online Certificate Status Protocol Location: URI: "http://ipa-ca.IPA.EXAMPLE.NET/ca/ocsp" Name: Extended Key Usage OCSP Responder Certificate Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Signature: 2f:...:32 Fingerprint (SHA-256): 85:...:3C Fingerprint (SHA1): 73:...:9F Mozilla-CA-Policy: false (attribute missing) Certificate Trust Flags: SSL Flags: User Email Flags: User Object Signing Flags: User certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'subsystemCert cert-pki-ca' Certificate: Data: Version: 3 (0x2) Serial Number: 4 (0x4) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=IPA.EXAMPLE.NET" Validity: Not Before: Fri Jan 12 13:25:20 2018 Not After : Thu Jan 02 13:25:20 2020 Subject: "CN=CA Subsystem,O=IPA.EXAMPLE.NET" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: cc:...:89 Exponent: 65537 (0x10001) Signed Extensions: Name: Certificate Authority Key Identifier Key ID: f1:...:2f Name: Authority Information Access Method: PKIX Online Certificate Status Protocol Location: URI: "http://ipa-ca.IPA.EXAMPLE.NET/ca/ocsp" Name: Certificate Key Usage Critical: True Usages: Digital Signature Non-Repudiation Key Encipherment Data Encipherment Name: Extended Key Usage TLS Web Server Authentication Certificate TLS Web Client Authentication Certificate Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Signature: 50:...:49 Fingerprint (SHA-256): CB:...:E5 Fingerprint (SHA1): 7A:...:CB Mozilla-CA-Policy: false (attribute missing) Certificate Trust Flags: SSL Flags: User Email Flags: User Object Signing Flags: User certutil -L -d /etc/pki/pki-tomcat/alias/ Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u On Thu, Jan 10, 2019 at 6:00 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: -- Jason Wood

unfortunately no I have not.  I did get the socket error at one point trying to troubleshoot the issue. Was caused by something I did, so wasn't related. I have no usable errors anywhere. I do know it is not getting far enough in the sequence to do self tests.  The upgrade did work when the packages were installed, the failure started only after a reboot.  I'm hoping its a simple fix, if we can ever discover the cause. Jason W On 1/28/2019 10:19 PM, Patrice Gamache via FreeIPA-users wrote: > do you have any news on this issue... i havea similar trouble...after yum update from centos 7.4 to 7.6 > the pki-tomcatd services failed to start...it worked for a while then failed and now it won't start anymore... > looked at the certificate and all seems ok... > after a quick look at the different log file, here's what i found: > > In Ldap (bound) connection pool to host ipa port 636, Cannot connect to LDAP server. Error: netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1) > > also, when i run ipa-server-upgrade, it hang waiting for CA to come up ... > > The CA status is: check interrupted due to error: cannot connect to 'http://cgi-mgmt01d.mgmt.internal:8080/ca/admin/ca/getStatus': timed out > > > something broke with the update... and i can't put my finger on it... > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...