Yeah. I definitely lost on this one at this point. As far as I can tell, SOMEHOW I'm missing these certs in the directory? Does that sound right? How would one go about making sure is corrected? I'm guess I'd need to regenerate some type of certificate on the IPA host, but I'm afraid of breaking things worse. I have one more day before this one expires, so I'm trying to troubleshoot and fix it before then. This all started when I noticed that another IPA server/replica failed to restart. I think these two issues are related, but right now, my users are functional with just the 'ipa01' system (which has this ca-error' issue and the cert not found. I'm afraid to restart anything on that system because of that. I'm still reading and trying to understand and put the pieces together, however I'm worried about this issue. Anyway, if anyone has any thoughts or tips here, I'd really appreciate it as I feel lost at this exact moment. On Tue, Dec 4, 2018 at 2:33 PM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > IPA 4.5.4 (has been upgraded for years just to understand that there > is a history) > This system (ipa01) is the renewal master (in case that matters) > > I'm getting the following error on 'getcert'. My gut tells me this is > kinda a big deal. :) I really could use some help figuring this one > out as I'm not the most CA-versed. I have been learning quite a bit > reading some of the blogs, but there's definitely alot of ignorance of > the details on my part. > > The error: > ----------- > [root@orldc-prod-ipa01 log]# getcert list | grep -A12 -B1 error > status: MONITORING > ca-error: Server at > "http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit" > replied: Record not found > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=PASSUR.LOCAL > subject: CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL > expires: 2018-12-06 21:43:50 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "Server-Cert cert-pki-ca" > track: yes > ----------- > > > If I look at the cert referenced locally in the NSS DB: > ------ > [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias > -f /etc/httpd/alias/pwdfile.txt > > Certificate Nickname Trust Attributes > SSL,S/MIME,JAR/XPI > > Server-Cert cert-pki-ca u,u,u > auditSigningCert cert-pki-ca u,u,Pu > caSigningCert cert-pki-ca CTu,Cu,Cu > subsystemCert cert-pki-ca u,u,u > ocspSigningCert cert-pki-ca u,u,u > ------ > [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias > -f /etc/httpd/alias/pwdfile.txt -n 'Server-Cert cert-pki-ca' | grep > "Subject:\|Serial" > Serial Number: 268304422 (0xffe0026) > Subject: "CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL" > ----- > [root@orldc-prod-ipa01 log]# ipa cert-find --min-serial-number > 268304422 --max-serial-number 268304423 > ---------------------- > 0 certificates matched > ---------------------- > ----- > > I'm trying to figure out how to find this certificate. And IF somehow > it is wrong or missing, how do I fix such a scenario? > > Any help here is always appreciated! Unfortunately, I'm running out > of time based on the expiration date I see on 'getcert'. I'm not sure > of the ramifications, but this seems pretty critical on the surface. > > Thanks again for any help and direction! > > -- Chris

Output: ---- [root@orldc-prod-ipa01 alias]# ipa-csreplica-manage list -v `hostname`.passur.local Directory Manager password: orldc-prod-ipa02.passur.local last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (-1) Problem connecting to replica - LDAP error: Can't contact LDAP server (connection error) last update ended: 1970-01-01 00:00:00+00:00 ---- Granted, it's replication partner (orldc-prod-ipa02) is the one that I mentioned has the issue starting at this point. So, that likely has something to do with this output. Having said that, I'm not quite sure what I should do here. I have definitely been issuing certs from this system. One note is that the 'hostname' command on my systems return only the short hostname. I'm not sure if that would be an issue or not, but it is worth noting. (To add 'domainname' does show the proper domainname for the system and what-not. Any ideas on the best way to fix just this host? I don't mind the idea of removing the CA replicas on the others after fixing this and re-replicating or anything. I just want to get this functioning before something exprires and I end up in a really bad spot (which is sadly only a day away!). (Why, oh why, do we always 'find' these type of problems under a time crunch in this business?) :) On Tue, Dec 4, 2018 at 5:57 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > Christopher Young via FreeIPA-users wrote: > > Yeah. I definitely lost on this one at this point. As far as I can > > tell, SOMEHOW I'm missing these certs in the directory? Does that > > sound right? > > > > How would one go about making sure is corrected? I'm guess I'd need > > to regenerate some type of certificate on the IPA host, but I'm afraid > > of breaking things worse. I have one more day before this one > > expires, so I'm trying to troubleshoot and fix it before then. This > > all started when I noticed that another IPA server/replica failed to > > restart. I think these two issues are related, but right now, my > > users are functional with just the 'ipa01' system (which has this > > ca-error' issue and the cert not found. I'm afraid to restart > > anything on that system because of that. > > > > I'm still reading and trying to understand and put the pieces > > together, however I'm worried about this issue. > > It sounds like one of your CA's is not replicating. You can use > ipa-csreplica-manage list -v `hostname` on each CA master to get the status. > > Any given replica can only store so much data to replicate so depending > on how long they have been disconnected could impact whether this is > easily recoverable. > > Given that, on any master it should always use its own CA (if there is > one) when issuing certs so this is a bit strange. > > rob > > > > > Anyway, if anyone has any thoughts or tips here, I'd really appreciate > > it as I feel lost at this exact moment. > > On Tue, Dec 4, 2018 at 2:33 PM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > >> > >> IPA 4.5.4 (has been upgraded for years just to understand that there > >> is a history) > >> This system (ipa01) is the renewal master (in case that matters) > >> > >> I'm getting the following error on 'getcert'. My gut tells me this is > >> kinda a big deal. :) I really could use some help figuring this one > >> out as I'm not the most CA-versed. I have been learning quite a bit > >> reading some of the blogs, but there's definitely alot of ignorance of > >> the details on my part. > >> > >> The error: > >> ----------- > >> [root@orldc-prod-ipa01 log]# getcert list | grep -A12 -B1 error > >> status: MONITORING > >> ca-error: Server at > >> "http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit" > >> replied: Record not found > >> stuck: no > >> key pair storage: > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > >> cert-pki-ca',token='NSS Certificate DB',pin set > >> certificate: > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > >> cert-pki-ca',token='NSS Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=PASSUR.LOCAL > >> subject: CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL > >> expires: 2018-12-06 21:43:50 UTC > >> key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > >> "Server-Cert cert-pki-ca" > >> track: yes > >> ----------- > >> > >> > >> If I look at the cert referenced locally in the NSS DB: > >> ------ > >> [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias > >> -f /etc/httpd/alias/pwdfile.txt > >> > >> Certificate Nickname Trust Attributes > >> SSL,S/MIME,JAR/XPI > >> > >> Server-Cert cert-pki-ca u,u,u > >> auditSigningCert cert-pki-ca u,u,Pu > >> caSigningCert cert-pki-ca CTu,Cu,Cu > >> subsystemCert cert-pki-ca u,u,u > >> ocspSigningCert cert-pki-ca u,u,u > >> ------ > >> [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias > >> -f /etc/httpd/alias/pwdfile.txt -n 'Server-Cert cert-pki-ca' | grep > >> "Subject:\|Serial" > >> Serial Number: 268304422 (0xffe0026) > >> Subject: "CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL" > >> ----- > >> [root@orldc-prod-ipa01 log]# ipa cert-find --min-serial-number > >> 268304422 --max-serial-number 268304423 > >> ---------------------- > >> 0 certificates matched > >> ---------------------- > >> ----- > >> > >> I'm trying to figure out how to find this certificate. And IF somehow > >> it is wrong or missing, how do I fix such a scenario? > >> > >> Any help here is always appreciated! Unfortunately, I'm running out > >> of time based on the expiration date I see on 'getcert'. I'm not sure > >> of the ramifications, but this seems pretty critical on the surface. > >> > >> Thanks again for any help and direction! > >> > >> -- Chris > > _______________________________________________ > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > >

Hi Christopher, I agree with Rob that replication issue is the most likely cause. If there were replication issues, depending on your topology there may be serial/request ID range conflicts too. But the most critical issue is the about-to-expire certificate. A couple of quick points/questions: - The expiring certificate is the Server-Cert, other CA replicas will have different Server-Certs so they will continue to function. - Are there any other certs on this replica, or others, that are close to expiry? Now to the error: "http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit" replied: Record not found It is not clear exactly what record is missing. It is likely either the certificate record, or its corresponding request record. The Dogtag debug log (/var/log/pki/pki-tomcat/ca/debug) may reveal more. In any case, have a hunt for cn=268304422,ou=certificateRepository,ou=ca,o=ipaca If found, in the entry there should be an attribute: metaInfo: requestId:<N> for some value of <N>. Now also look for the entry: cn=<N>,ou=ca,ou=requests,o=ipaca If any of these entries can be found on other replicas but not the database on the replica where the cert is expiring, you can manually export/import them, and it might solve the issue. Otherwise, I recall a recent issue where the workaround was to make the Certmonger renewal helper do a "new issuance" rather than a "renewal"-based operation against the Dogtag CA. This could help in your situation too. I am not sure whether or where the steps were recorded so Rob, Florence - do you know? Anyhow it is possible I have gone down the garden path so it would really help to see the relevant portion of the Dogtag debug log. (Be aware Dogtag timestamps are in local time, when you are looking for the relevant output). Cheers, Fraser On Tue, Dec 04, 2018 at 09:47:11PM -0500, Christopher Young via FreeIPA-users wrote:

AND... it looks like I'll be changing my directory password after this! LOL Ugh. When you are in a hurry. On Wed, Dec 5, 2018 at 10:39 AM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > Thanks again for the response! So, this is interesting. an > ldapsearch actually does find a record, yet if I use something like > Apache Directory Studio to try and look at it, it doesn't show up. > ---- > [root@orldc-prod-ipa01 alias]# ldapsearch -h localhost -p 389 -D > 'cn=Directory Manager' -w "B\$ankers1" -b > "cn=268304420,ou=certificateRepository,ou=ca,o=ipaca" -LLL > dn: cn=268304420,ou=certificateRepository,ou=ca,o=ipaca > cn: 268304420 > issuedBy: ipara > autoRenew: ENABLED > certStatus: VALID > dateOfModify: 20161216163020Z > dateOfCreate: 20161216163020Z > signingAlgorithmId: 1.2.840.113549.1.1.11 > algorithmId: 1.2.840.113549.1.1.1 > version: 2 > userCertificate;binary:: MIIEIjxxxxxx > .... > .... > extension: 1.3.6.1.5.5.7.1.1 > extension: 2.5.29.14 > extension: 2.5.29.37 > extension: 2.5.29.35 > extension: 2.5.29.31 > extension: 2.5.29.15 > publicKeyData:: MIIBIjA... > .... > .... > issuerName: CN=Certificate Authority,O=PASSUR.LOCAL > subjectName: CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL > duration: 1163158400000 > notAfter: 20181217163020Z > notBefore: 20161216163020Z > metaInfo: requestId:9980041 > metaInfo: profileId:caIPAserviceCert > serialno: 09268304420 > objectClass: top > objectClass: certificateRecord > > > ---- > Strange. I'm wondering if there is some permissions problem in the > directory? I have no idea how I would fix that if it were, however > this is, in itself, revealing. > On Tue, Dec 4, 2018 at 10:57 PM Fraser Tweedale <ftweedal(a)redhat.com&gt; wrote: > > > > Hi Christopher, > > > > I agree with Rob that replication issue is the most likely cause. > > If there were replication issues, depending on your topology there > > may be serial/request ID range conflicts too. But the most critical > > issue is the about-to-expire certificate. > > > > A couple of quick points/questions: > > > > - The expiring certificate is the Server-Cert, other CA replicas > > will have different Server-Certs so they will continue to > > function. > > > > - Are there any other certs on this replica, or others, that are > > close to expiry? > > > > Now to the error: > > > > "http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit" > > replied: Record not found > > > > It is not clear exactly what record is missing. It is likely either > > the certificate record, or its corresponding request record. The > > Dogtag debug log (/var/log/pki/pki-tomcat/ca/debug) may reveal more. > > > > In any case, have a hunt for > > > > cn=268304422,ou=certificateRepository,ou=ca,o=ipaca > > > > If found, in the entry there should be an attribute: > > > > metaInfo: requestId:<N> > > > > for some value of <N>. Now also look for the entry: > > > > cn=<N>,ou=ca,ou=requests,o=ipaca > > > > If any of these entries can be found on other replicas but not the > > database on the replica where the cert is expiring, you can manually > > export/import them, and it might solve the issue. > > > > Otherwise, I recall a recent issue where the workaround was to make > > the Certmonger renewal helper do a "new issuance" rather than a > > "renewal"-based operation against the Dogtag CA. This could help in > > your situation too. I am not sure whether or where the steps were > > recorded so Rob, Florence - do you know? > > > > Anyhow it is possible I have gone down the garden path so it would > > really help to see the relevant portion of the Dogtag debug log. > > (Be aware Dogtag timestamps are in local time, when you are looking > > for the relevant output). > > > > Cheers, > > Fraser > > > > On Tue, Dec 04, 2018 at 09:47:11PM -0500, Christopher Young via FreeIPA-users wrote: > > > Another thing I notice that confuses me... (see attached) > > > > > > > > > Is it normal to have this many certificate with the same Subject for > > > an IPA server? I'm wondering if somewhere along it renewed and yet > > > didn't update locally or something. I'm really not sure what's going > > > on here, but what I'm confused about is IF I wanted to generate a new > > > server for the IPA server, how would I go about doing that in a manner > > > where the certificate would have all the right attributes? (and would > > > I want to do that?) > > > > > > Sorry for all the questions. I'm figuring the pieces out as I go. > > > On Tue, Dec 4, 2018 at 9:04 PM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > > > > > Output: > > > > ---- > > > > [root@orldc-prod-ipa01 alias]# ipa-csreplica-manage list -v > > > > `hostname`.passur.local > > > > Directory Manager password: > > > > > > > > orldc-prod-ipa02.passur.local > > > > last init status: None > > > > last init ended: 1970-01-01 00:00:00+00:00 > > > > last update status: Error (-1) Problem connecting to replica - LDAP > > > > error: Can't contact LDAP server (connection error) > > > > last update ended: 1970-01-01 00:00:00+00:00 > > > > ---- > > > > > > > > Granted, it's replication partner (orldc-prod-ipa02) is the one that I > > > > mentioned has the issue starting at this point. So, that likely has > > > > something to do with this output. Having said that, I'm not quite > > > > sure what I should do here. I have definitely been issuing certs from > > > > this system. One note is that the 'hostname' command on my systems > > > > return only the short hostname. I'm not sure if that would be an > > > > issue or not, but it is worth noting. (To add 'domainname' does show > > > > the proper domainname for the system and what-not. > > > > > > > > Any ideas on the best way to fix just this host? I don't mind the > > > > idea of removing the CA replicas on the others after fixing this and > > > > re-replicating or anything. I just want to get this functioning > > > > before something exprires and I end up in a really bad spot (which is > > > > sadly only a day away!). (Why, oh why, do we always 'find' these > > > > type of problems under a time crunch in this business?) :) > > > > > > > > > > > > > > > > > > > > On Tue, Dec 4, 2018 at 5:57 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > > > > > > > > > Christopher Young via FreeIPA-users wrote: > > > > > > Yeah. I definitely lost on this one at this point. As far as I can > > > > > > tell, SOMEHOW I'm missing these certs in the directory? Does that > > > > > > sound right? > > > > > > > > > > > > How would one go about making sure is corrected? I'm guess I'd need > > > > > > to regenerate some type of certificate on the IPA host, but I'm afraid > > > > > > of breaking things worse. I have one more day before this one > > > > > > expires, so I'm trying to troubleshoot and fix it before then. This > > > > > > all started when I noticed that another IPA server/replica failed to > > > > > > restart. I think these two issues are related, but right now, my > > > > > > users are functional with just the 'ipa01' system (which has this > > > > > > ca-error' issue and the cert not found. I'm afraid to restart > > > > > > anything on that system because of that. > > > > > > > > > > > > I'm still reading and trying to understand and put the pieces > > > > > > together, however I'm worried about this issue. > > > > > > > > > > It sounds like one of your CA's is not replicating. You can use > > > > > ipa-csreplica-manage list -v `hostname` on each CA master to get the status. > > > > > > > > > > Any given replica can only store so much data to replicate so depending > > > > > on how long they have been disconnected could impact whether this is > > > > > easily recoverable. > > > > > > > > > > Given that, on any master it should always use its own CA (if there is > > > > > one) when issuing certs so this is a bit strange. > > > > > > > > > > rob > > > > > > > > > > > > > > > > > Anyway, if anyone has any thoughts or tips here, I'd really appreciate > > > > > > it as I feel lost at this exact moment. > > > > > > On Tue, Dec 4, 2018 at 2:33 PM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > > >> > > > > > >> IPA 4.5.4 (has been upgraded for years just to understand that there > > > > > >> is a history) > > > > > >> This system (ipa01) is the renewal master (in case that matters) > > > > > >> > > > > > >> I'm getting the following error on 'getcert'. My gut tells me this is > > > > > >> kinda a big deal. :) I really could use some help figuring this one > > > > > >> out as I'm not the most CA-versed. I have been learning quite a bit > > > > > >> reading some of the blogs, but there's definitely alot of ignorance of > > > > > >> the details on my part. > > > > > >> > > > > > >> The error: > > > > > >> ----------- > > > > > >> [root@orldc-prod-ipa01 log]# getcert list | grep -A12 -B1 error > > > > > >> status: MONITORING > > > > > >> ca-error: Server at > > > > > >> "http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit" > > > > > >> replied: Record not found > > > > > >> stuck: no > > > > > >> key pair storage: > > > > > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > > > > > >> cert-pki-ca',token='NSS Certificate DB',pin set > > > > > >> certificate: > > > > > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > > > > > >> cert-pki-ca',token='NSS Certificate DB' > > > > > >> CA: dogtag-ipa-ca-renew-agent > > > > > >> issuer: CN=Certificate Authority,O=PASSUR.LOCAL > > > > > >> subject: CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL > > > > > >> expires: 2018-12-06 21:43:50 UTC > > > > > >> key usage: > > > > > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > > > > >> eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection > > > > > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > > > > >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > > > > > >> "Server-Cert cert-pki-ca" > > > > > >> track: yes > > > > > >> ----------- > > > > > >> > > > > > >> > > > > > >> If I look at the cert referenced locally in the NSS DB: > > > > > >> ------ > > > > > >> [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias > > > > > >> -f /etc/httpd/alias/pwdfile.txt > > > > > >> > > > > > >> Certificate Nickname Trust Attributes > > > > > >> SSL,S/MIME,JAR/XPI > > > > > >> > > > > > >> Server-Cert cert-pki-ca u,u,u > > > > > >> auditSigningCert cert-pki-ca u,u,Pu > > > > > >> caSigningCert cert-pki-ca CTu,Cu,Cu > > > > > >> subsystemCert cert-pki-ca u,u,u > > > > > >> ocspSigningCert cert-pki-ca u,u,u > > > > > >> ------ > > > > > >> [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias > > > > > >> -f /etc/httpd/alias/pwdfile.txt -n 'Server-Cert cert-pki-ca' | grep > > > > > >> "Subject:\|Serial" > > > > > >> Serial Number: 268304422 (0xffe0026) > > > > > >> Subject: "CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL" > > > > > >> ----- > > > > > >> [root@orldc-prod-ipa01 log]# ipa cert-find --min-serial-number > > > > > >> 268304422 --max-serial-number 268304423 > > > > > >> ---------------------- > > > > > >> 0 certificates matched > > > > > >> ---------------------- > > > > > >> ----- > > > > > >> > > > > > >> I'm trying to figure out how to find this certificate. And IF somehow > > > > > >> it is wrong or missing, how do I fix such a scenario? > > > > > >> > > > > > >> Any help here is always appreciated! Unfortunately, I'm running out > > > > > >> of time based on the expiration date I see on 'getcert'. I'm not sure > > > > > >> of the ramifications, but this seems pretty critical on the surface. > > > > > >> > > > > > >> Thanks again for any help and direction! > > > > > >> > > > > > >> -- Chris > > > > > > _______________________________________________ > > > > > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > > > > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > > > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > > > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > > > > > > > > > > > > > > > > > _______________________________________________ > > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > >

Ugh. I'm sorry for spamming the list (not in my nature). I see that I must have typo'ed the query. Let me get my head straight and I'll update this. Again, I really apologize. On Wed, Dec 5, 2018 at 10:48 AM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > Actually, I just noticed something with the 'serialno' attribute here. > It seems to not match the cn. That's very odd. I'm considering > just trying to manually change that and see what happens. Any > thoughts on that? > On Wed, Dec 5, 2018 at 10:41 AM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > AND... it looks like I'll be changing my directory password after > > this! LOL Ugh. > > > > When you are in a hurry. > > On Wed, Dec 5, 2018 at 10:39 AM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > > > Thanks again for the response! So, this is interesting. an > > > ldapsearch actually does find a record, yet if I use something like > > > Apache Directory Studio to try and look at it, it doesn't show up. > > > ---- > > > [root@orldc-prod-ipa01 alias]# ldapsearch -h localhost -p 389 -D > > > 'cn=Directory Manager' -w "B\$ankers1" -b > > > "cn=268304420,ou=certificateRepository,ou=ca,o=ipaca" -LLL > > > dn: cn=268304420,ou=certificateRepository,ou=ca,o=ipaca > > > cn: 268304420 > > > issuedBy: ipara > > > autoRenew: ENABLED > > > certStatus: VALID > > > dateOfModify: 20161216163020Z > > > dateOfCreate: 20161216163020Z > > > signingAlgorithmId: 1.2.840.113549.1.1.11 > > > algorithmId: 1.2.840.113549.1.1.1 > > > version: 2 > > > userCertificate;binary:: MIIEIjxxxxxx > > > .... > > > .... > > > extension: 1.3.6.1.5.5.7.1.1 > > > extension: 2.5.29.14 > > > extension: 2.5.29.37 > > > extension: 2.5.29.35 > > > extension: 2.5.29.31 > > > extension: 2.5.29.15 > > > publicKeyData:: MIIBIjA... > > > .... > > > .... > > > issuerName: CN=Certificate Authority,O=PASSUR.LOCAL > > > subjectName: CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL > > > duration: 1163158400000 > > > notAfter: 20181217163020Z > > > notBefore: 20161216163020Z > > > metaInfo: requestId:9980041 > > > metaInfo: profileId:caIPAserviceCert > > > serialno: 09268304420 > > > objectClass: top > > > objectClass: certificateRecord > > > > > > > > > ---- > > > Strange. I'm wondering if there is some permissions problem in the > > > directory? I have no idea how I would fix that if it were, however > > > this is, in itself, revealing. > > > On Tue, Dec 4, 2018 at 10:57 PM Fraser Tweedale <ftweedal(a)redhat.com&gt; wrote: > > > > > > > > Hi Christopher, > > > > > > > > I agree with Rob that replication issue is the most likely cause. > > > > If there were replication issues, depending on your topology there > > > > may be serial/request ID range conflicts too. But the most critical > > > > issue is the about-to-expire certificate. > > > > > > > > A couple of quick points/questions: > > > > > > > > - The expiring certificate is the Server-Cert, other CA replicas > > > > will have different Server-Certs so they will continue to > > > > function. > > > > > > > > - Are there any other certs on this replica, or others, that are > > > > close to expiry? > > > > > > > > Now to the error: > > > > > > > > "http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit" > > > > replied: Record not found > > > > > > > > It is not clear exactly what record is missing. It is likely either > > > > the certificate record, or its corresponding request record. The > > > > Dogtag debug log (/var/log/pki/pki-tomcat/ca/debug) may reveal more. > > > > > > > > In any case, have a hunt for > > > > > > > > cn=268304422,ou=certificateRepository,ou=ca,o=ipaca > > > > > > > > If found, in the entry there should be an attribute: > > > > > > > > metaInfo: requestId:<N> > > > > > > > > for some value of <N>. Now also look for the entry: > > > > > > > > cn=<N>,ou=ca,ou=requests,o=ipaca > > > > > > > > If any of these entries can be found on other replicas but not the > > > > database on the replica where the cert is expiring, you can manually > > > > export/import them, and it might solve the issue. > > > > > > > > Otherwise, I recall a recent issue where the workaround was to make > > > > the Certmonger renewal helper do a "new issuance" rather than a > > > > "renewal"-based operation against the Dogtag CA. This could help in > > > > your situation too. I am not sure whether or where the steps were > > > > recorded so Rob, Florence - do you know? > > > > > > > > Anyhow it is possible I have gone down the garden path so it would > > > > really help to see the relevant portion of the Dogtag debug log. > > > > (Be aware Dogtag timestamps are in local time, when you are looking > > > > for the relevant output). > > > > > > > > Cheers, > > > > Fraser > > > > > > > > On Tue, Dec 04, 2018 at 09:47:11PM -0500, Christopher Young via FreeIPA-users wrote: > > > > > Another thing I notice that confuses me... (see attached) > > > > > > > > > > > > > > > Is it normal to have this many certificate with the same Subject for > > > > > an IPA server? I'm wondering if somewhere along it renewed and yet > > > > > didn't update locally or something. I'm really not sure what's going > > > > > on here, but what I'm confused about is IF I wanted to generate a new > > > > > server for the IPA server, how would I go about doing that in a manner > > > > > where the certificate would have all the right attributes? (and would > > > > > I want to do that?) > > > > > > > > > > Sorry for all the questions. I'm figuring the pieces out as I go. > > > > > On Tue, Dec 4, 2018 at 9:04 PM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > > > > > > > > > Output: > > > > > > ---- > > > > > > [root@orldc-prod-ipa01 alias]# ipa-csreplica-manage list -v > > > > > > `hostname`.passur.local > > > > > > Directory Manager password: > > > > > > > > > > > > orldc-prod-ipa02.passur.local > > > > > > last init status: None > > > > > > last init ended: 1970-01-01 00:00:00+00:00 > > > > > > last update status: Error (-1) Problem connecting to replica - LDAP > > > > > > error: Can't contact LDAP server (connection error) > > > > > > last update ended: 1970-01-01 00:00:00+00:00 > > > > > > ---- > > > > > > > > > > > > Granted, it's replication partner (orldc-prod-ipa02) is the one that I > > > > > > mentioned has the issue starting at this point. So, that likely has > > > > > > something to do with this output. Having said that, I'm not quite > > > > > > sure what I should do here. I have definitely been issuing certs from > > > > > > this system. One note is that the 'hostname' command on my systems > > > > > > return only the short hostname. I'm not sure if that would be an > > > > > > issue or not, but it is worth noting. (To add 'domainname' does show > > > > > > the proper domainname for the system and what-not. > > > > > > > > > > > > Any ideas on the best way to fix just this host? I don't mind the > > > > > > idea of removing the CA replicas on the others after fixing this and > > > > > > re-replicating or anything. I just want to get this functioning > > > > > > before something exprires and I end up in a really bad spot (which is > > > > > > sadly only a day away!). (Why, oh why, do we always 'find' these > > > > > > type of problems under a time crunch in this business?) :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 4, 2018 at 5:57 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > > > > > > > > > > > > > Christopher Young via FreeIPA-users wrote: > > > > > > > > Yeah. I definitely lost on this one at this point. As far as I can > > > > > > > > tell, SOMEHOW I'm missing these certs in the directory? Does that > > > > > > > > sound right? > > > > > > > > > > > > > > > > How would one go about making sure is corrected? I'm guess I'd need > > > > > > > > to regenerate some type of certificate on the IPA host, but I'm afraid > > > > > > > > of breaking things worse. I have one more day before this one > > > > > > > > expires, so I'm trying to troubleshoot and fix it before then. This > > > > > > > > all started when I noticed that another IPA server/replica failed to > > > > > > > > restart. I think these two issues are related, but right now, my > > > > > > > > users are functional with just the 'ipa01' system (which has this > > > > > > > > ca-error' issue and the cert not found. I'm afraid to restart > > > > > > > > anything on that system because of that. > > > > > > > > > > > > > > > > I'm still reading and trying to understand and put the pieces > > > > > > > > together, however I'm worried about this issue. > > > > > > > > > > > > > > It sounds like one of your CA's is not replicating. You can use > > > > > > > ipa-csreplica-manage list -v `hostname` on each CA master to get the status. > > > > > > > > > > > > > > Any given replica can only store so much data to replicate so depending > > > > > > > on how long they have been disconnected could impact whether this is > > > > > > > easily recoverable. > > > > > > > > > > > > > > Given that, on any master it should always use its own CA (if there is > > > > > > > one) when issuing certs so this is a bit strange. > > > > > > > > > > > > > > rob > > > > > > > > > > > > > > > > > > > > > > > Anyway, if anyone has any thoughts or tips here, I'd really appreciate > > > > > > > > it as I feel lost at this exact moment. > > > > > > > > On Tue, Dec 4, 2018 at 2:33 PM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > > > > >> > > > > > > > >> IPA 4.5.4 (has been upgraded for years just to understand that there > > > > > > > >> is a history) > > > > > > > >> This system (ipa01) is the renewal master (in case that matters) > > > > > > > >> > > > > > > > >> I'm getting the following error on 'getcert'. My gut tells me this is > > > > > > > >> kinda a big deal. :) I really could use some help figuring this one > > > > > > > >> out as I'm not the most CA-versed. I have been learning quite a bit > > > > > > > >> reading some of the blogs, but there's definitely alot of ignorance of > > > > > > > >> the details on my part. > > > > > > > >> > > > > > > > >> The error: > > > > > > > >> ----------- > > > > > > > >> [root@orldc-prod-ipa01 log]# getcert list | grep -A12 -B1 error > > > > > > > >> status: MONITORING > > > > > > > >> ca-error: Server at > > > > > > > >> "http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit" > > > > > > > >> replied: Record not found > > > > > > > >> stuck: no > > > > > > > >> key pair storage: > > > > > > > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > > > > > > > >> cert-pki-ca',token='NSS Certificate DB',pin set > > > > > > > >> certificate: > > > > > > > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > > > > > > > >> cert-pki-ca',token='NSS Certificate DB' > > > > > > > >> CA: dogtag-ipa-ca-renew-agent > > > > > > > >> issuer: CN=Certificate Authority,O=PASSUR.LOCAL > > > > > > > >> subject: CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL > > > > > > > >> expires: 2018-12-06 21:43:50 UTC > > > > > > > >> key usage: > > > > > > > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > > > > > > >> eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection > > > > > > > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > > > > > > >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > > > > > > > >> "Server-Cert cert-pki-ca" > > > > > > > >> track: yes > > > > > > > >> ----------- > > > > > > > >> > > > > > > > >> > > > > > > > >> If I look at the cert referenced locally in the NSS DB: > > > > > > > >> ------ > > > > > > > >> [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias > > > > > > > >> -f /etc/httpd/alias/pwdfile.txt > > > > > > > >> > > > > > > > >> Certificate Nickname Trust Attributes > > > > > > > >> SSL,S/MIME,JAR/XPI > > > > > > > >> > > > > > > > >> Server-Cert cert-pki-ca u,u,u > > > > > > > >> auditSigningCert cert-pki-ca u,u,Pu > > > > > > > >> caSigningCert cert-pki-ca CTu,Cu,Cu > > > > > > > >> subsystemCert cert-pki-ca u,u,u > > > > > > > >> ocspSigningCert cert-pki-ca u,u,u > > > > > > > >> ------ > > > > > > > >> [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias > > > > > > > >> -f /etc/httpd/alias/pwdfile.txt -n 'Server-Cert cert-pki-ca' | grep > > > > > > > >> "Subject:\|Serial" > > > > > > > >> Serial Number: 268304422 (0xffe0026) > > > > > > > >> Subject: "CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL" > > > > > > > >> ----- > > > > > > > >> [root@orldc-prod-ipa01 log]# ipa cert-find --min-serial-number > > > > > > > >> 268304422 --max-serial-number 268304423 > > > > > > > >> ---------------------- > > > > > > > >> 0 certificates matched > > > > > > > >> ---------------------- > > > > > > > >> ----- > > > > > > > >> > > > > > > > >> I'm trying to figure out how to find this certificate. And IF somehow > > > > > > > >> it is wrong or missing, how do I fix such a scenario? > > > > > > > >> > > > > > > > >> Any help here is always appreciated! Unfortunately, I'm running out > > > > > > > >> of time based on the expiration date I see on 'getcert'. I'm not sure > > > > > > > >> of the ramifications, but this seems pretty critical on the surface. > > > > > > > >> > > > > > > > >> Thanks again for any help and direction! > > > > > > > >> > > > > > > > >> -- Chris > > > > > > > > _______________________________________________ > > > > > > > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > > > > > > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > > > > > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > > > > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > > > > > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > > > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > > >

On Wed, Dec 05, 2018 at 11:37:36AM -0500, Christopher Young wrote: The serial number attribute has run length encoding for some reason. I can't explain why off the top of my head (and I'm not going to bother investigating right now) but it's normal behaviour. I'll have a look at the debug log below and get back to you soon. Cheers, Fraser > In any case, I have the record there, and I get a 'resubmit' using > getcert on the ipa01 system for the certificate in question and it no > longer gets the 'Record not found' in the 'getcert list' output, > HOWEVER it still failed and now gives me an 'Internal Server Error' > result. I looked at the /var/log/pki/pki-tomcat/ca/debug file on > 'ipa01' and the output isn't much help to me at the moment. > > ---------- > [05/Dec/2018:11:27:33][Timer-0]: SessionTimer: run() > [05/Dec/2018:11:27:33][Timer-0]: LDAPSecurityDomainSessionTable: getSessionIds() > [05/Dec/2018:11:27:33][Timer-0]: LDAPSecurityDomainSessionTable: > searching ou=sessions,ou=Security Domain,o=ipaca > [05/Dec/2018:11:27:33][Timer-0]: In LdapBoundConnFactory::getConn() > [05/Dec/2018:11:27:33][Timer-0]: masterConn is connected: true > [05/Dec/2018:11:27:33][Timer-0]: getConn: conn is connected true > [05/Dec/2018:11:27:33][Timer-0]: getConn: mNumConns now 2 > [05/Dec/2018:11:27:33][Timer-0]: returnConn: mNumConns now 3 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CMSServlet:service() > uri = /ca/ee/ca/profileSubmit > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CMSServlet::service() > param name='profileId' value='caServerCert' > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CMSServlet::service() > param name='serial_num' value='268304422' > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CMSServlet::service() > param name='renewal' value='true' > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CMSServlet::service() > param name='xml' value='true' > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CMSServlet::service() > param name='requestor_name' value='IPA' > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CMSServlet: > caProfileSubmit start to service. > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: xmlOutput true > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: ProfileSubmitServlet: > isRenewal true > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: according to ccMode, > authorization for servlet: caProfileSubmit is LDAP based, not XML {1}, > use default authz mgr: {2}. > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: ProfileSubmitServlet: > profile: caServerCert > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CAProcessor: Input Parameters: > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CAProcessor: - isRenewal: false > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CAProcessor: - > remoteHost: 10.16.250.61 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CAProcessor: - > profileId: caServerCert > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CAProcessor: - > requestor_name: IPA > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CAProcessor: - > serial_num: 0xffe0026 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CAProcessor: - > remoteAddr: 10.16.250.61 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: RenewalProcessor: > processRenewal() > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: RenewalProcessor: > profile: caServerCert > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: RenewalProcessor: > serial number: 268304422 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: processRenewal: serial > number of cert to renew:268304422 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: In > LdapBoundConnFactory::getConn() > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: masterConn is connected: true > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: getConn: conn is connected true > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: getConn: mNumConns now 2 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: returnConn: mNumConns now 3 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: processRenewal: > origNotAfter =Thu Dec 06 16:43:50 EST 2018 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: processRenewal: orig > subj dn =CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: In > LdapBoundConnFactory::getConn() > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: masterConn is connected: true > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: getConn: conn is connected true > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: getConn: mNumConns now 2 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: Error: Record not found > Record not found > at com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:182) > at com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:137) > at com.netscape.cmscore.request.RequestQueue.readRequest(RequestQueue.java:83) > at com.netscape.cmscore.request.ARequestQueue.findRequest(ARequestQueue.java:342) > at com.netscape.cms.servlet.processors.CAProcessor.getOriginalRequest(CAProcessor.java:246) > at com.netscape.cms.servlet.cert.RenewalProcessor.processRenewal(RenewalProcessor.java:208) > at com.netscape.cms.servlet.profile.ProfileSubmitServlet.processRenewal(ProfileSubmitServlet.java:274) > at com.netscape.cms.servlet.profile.ProfileSubmitServlet.process(ProfileSubmitServlet.java:126) > at com.netscape.cms.servlet.base.CMSServlet.service(CMSServlet.java:512) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297) > at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) > at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) > at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) > at java.security.AccessController.doPrivileged(Native Method) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) > at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) > at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) > at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) > at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) > at java.security.AccessController.doPrivileged(Native Method) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) > at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) > at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087) > at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) > at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:748) > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: returnConn: mNumConns now 3 > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: processRenewal: > original request not found > Server Internal Error > at com.netscape.cms.servlet.cert.RenewalProcessor.processRenewal(RenewalProcessor.java:211) > at com.netscape.cms.servlet.profile.ProfileSubmitServlet.processRenewal(ProfileSubmitServlet.java:274) > at com.netscape.cms.servlet.profile.ProfileSubmitServlet.process(ProfileSubmitServlet.java:126) > at com.netscape.cms.servlet.base.CMSServlet.service(CMSServlet.java:512) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297) > at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) > at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) > at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) > at java.security.AccessController.doPrivileged(Native Method) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) > at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) > at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) > at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) > at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) > at java.security.AccessController.doPrivileged(Native Method) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) > at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) > at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087) > at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) > at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:748) > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: ProfileSubmitServlet: > error in processing request: Server Internal Error > [05/Dec/2018:11:29:09][http-bio-8080-exec-11]: CMSServlet: curDate=Wed > Dec 05 11:29:09 EST 2018 id=caProfileSubmit time=22 > [05/Dec/2018:11:32:33][Timer-0]: SessionTimer: run() > [05/Dec/2018:11:32:33][Timer-0]: LDAPSecurityDomainSessionTable: getSessionIds() > [05/Dec/2018:11:32:33][Timer-0]: LDAPSecurityDomainSessionTable: > searching ou=sessions,ou=Security Domain,o=ipaca > [05/Dec/2018:11:32:33][Timer-0]: In LdapBoundConnFactory::getConn() > [05/Dec/2018:11:32:33][Timer-0]: masterConn is connected: true > [05/Dec/2018:11:32:33][Timer-0]: getConn: conn is connected true > [05/Dec/2018:11:32:33][Timer-0]: getConn: mNumConns now 2 > [05/Dec/2018:11:32:33][Timer-0]: returnConn: mNumConns now 3 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: About to start > updateSerialNumbers > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Starting > updateSerialNumbers (entered lock) > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: CertificateRepository: > updateCounter mEnableRandomSerialNumbers=false mCounter=-1 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: In > LdapBoundConnFactory::getConn() > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: masterConn is connected: true > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: getConn: conn is connected true > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: getConn: mNumConns now 2 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Releasing ldap connection > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: returnConn: mNumConns now 3 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: DBSubsystem: > getEntryAttribute: dn=ou=certificateRepository, ou=ca, o=ipaca > attr=description:; > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: CertificateRepository: > updateCounter mEnableRandomSerialNumbers=false > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: CertificateRepository: > updateCounter CertificateRepositoryMode = > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: CertificateRepository: > updateCounter modeChange=false > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: CertificateRepository: > UpdateCounter mEnableRandomSerialNumbers=false mCounter=-1 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Starting cert checkRanges > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Repository: Serial > numbers left in range: 65498 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Repository: Last > serial number: 805240870 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Repository: Serial > numbers in next range: 268435456 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Repository: Serial > numbers available: 268500954 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Repository: Low water > mark: 33554432 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Checking for a range conflict > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: In > LdapBoundConnFactory::getConn() > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: masterConn is connected: true > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: getConn: conn is connected true > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: getConn: mNumConns now 2 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Releasing ldap connection > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: returnConn: mNumConns now 3 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Starting request checkRanges > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Repository: Serial > numbers left in range: 9942 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Repository: Last > serial number: 29990058 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Repository: Serial > numbers in next range: 10000000 > [05/Dec/2018:11:32:33][SerialNumberUpdateTask]: Repository: Serial > numbers available: 10009942 > ..... > ---------- > > I think I'm on to something here, however I'm still completely unsure > what to do going forward. I'll keep digging, but if you have any > thought or direction you can give me, I'd greatly appreciate it! > > -- Chris > On Wed, Dec 5, 2018 at 10:54 AM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > Ugh. I'm sorry for spamming the list (not in my nature). I see that > > I must have typo'ed the query. Let me get my head straight and I'll > > update this. Again, I really apologize. > > On Wed, Dec 5, 2018 at 10:48 AM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > > > Actually, I just noticed something with the 'serialno' attribute here. > > > It seems to not match the cn. That's very odd. I'm considering > > > just trying to manually change that and see what happens. Any > > > thoughts on that? > > > On Wed, Dec 5, 2018 at 10:41 AM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > > > > > AND... it looks like I'll be changing my directory password after > > > > this! LOL Ugh. > > > > > > > > When you are in a hurry. > > > > On Wed, Dec 5, 2018 at 10:39 AM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > > > > > > > Thanks again for the response! So, this is interesting. an > > > > > ldapsearch actually does find a record, yet if I use something like > > > > > Apache Directory Studio to try and look at it, it doesn't show up. > > > > > ---- > > > > > [root@orldc-prod-ipa01 alias]# ldapsearch -h localhost -p 389 -D > > > > > 'cn=Directory Manager' -w "B\$ankers1" -b > > > > > "cn=268304420,ou=certificateRepository,ou=ca,o=ipaca" -LLL > > > > > dn: cn=268304420,ou=certificateRepository,ou=ca,o=ipaca > > > > > cn: 268304420 > > > > > issuedBy: ipara > > > > > autoRenew: ENABLED > > > > > certStatus: VALID > > > > > dateOfModify: 20161216163020Z > > > > > dateOfCreate: 20161216163020Z > > > > > signingAlgorithmId: 1.2.840.113549.1.1.11 > > > > > algorithmId: 1.2.840.113549.1.1.1 > > > > > version: 2 > > > > > userCertificate;binary:: MIIEIjxxxxxx > > > > > .... > > > > > .... > > > > > extension: 1.3.6.1.5.5.7.1.1 > > > > > extension: 2.5.29.14 > > > > > extension: 2.5.29.37 > > > > > extension: 2.5.29.35 > > > > > extension: 2.5.29.31 > > > > > extension: 2.5.29.15 > > > > > publicKeyData:: MIIBIjA... > > > > > .... > > > > > .... > > > > > issuerName: CN=Certificate Authority,O=PASSUR.LOCAL > > > > > subjectName: CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL > > > > > duration: 1163158400000 > > > > > notAfter: 20181217163020Z > > > > > notBefore: 20161216163020Z > > > > > metaInfo: requestId:9980041 > > > > > metaInfo: profileId:caIPAserviceCert > > > > > serialno: 09268304420 > > > > > objectClass: top > > > > > objectClass: certificateRecord > > > > > > > > > > > > > > > ---- > > > > > Strange. I'm wondering if there is some permissions problem in the > > > > > directory? I have no idea how I would fix that if it were, however > > > > > this is, in itself, revealing. > > > > > On Tue, Dec 4, 2018 at 10:57 PM Fraser Tweedale <ftweedal(a)redhat.com&gt; wrote: > > > > > > > > > > > > Hi Christopher, > > > > > > > > > > > > I agree with Rob that replication issue is the most likely cause. > > > > > > If there were replication issues, depending on your topology there > > > > > > may be serial/request ID range conflicts too. But the most critical > > > > > > issue is the about-to-expire certificate. > > > > > > > > > > > > A couple of quick points/questions: > > > > > > > > > > > > - The expiring certificate is the Server-Cert, other CA replicas > > > > > > will have different Server-Certs so they will continue to > > > > > > function. > > > > > > > > > > > > - Are there any other certs on this replica, or others, that are > > > > > > close to expiry? > > > > > > > > > > > > Now to the error: > > > > > > > > > > > > "http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit" > > > > > > replied: Record not found > > > > > > > > > > > > It is not clear exactly what record is missing. It is likely either > > > > > > the certificate record, or its corresponding request record. The > > > > > > Dogtag debug log (/var/log/pki/pki-tomcat/ca/debug) may reveal more. > > > > > > > > > > > > In any case, have a hunt for > > > > > > > > > > > > cn=268304422,ou=certificateRepository,ou=ca,o=ipaca > > > > > > > > > > > > If found, in the entry there should be an attribute: > > > > > > > > > > > > metaInfo: requestId:<N> > > > > > > > > > > > > for some value of <N>. Now also look for the entry: > > > > > > > > > > > > cn=<N>,ou=ca,ou=requests,o=ipaca > > > > > > > > > > > > If any of these entries can be found on other replicas but not the > > > > > > database on the replica where the cert is expiring, you can manually > > > > > > export/import them, and it might solve the issue. > > > > > > > > > > > > Otherwise, I recall a recent issue where the workaround was to make > > > > > > the Certmonger renewal helper do a "new issuance" rather than a > > > > > > "renewal"-based operation against the Dogtag CA. This could help in > > > > > > your situation too. I am not sure whether or where the steps were > > > > > > recorded so Rob, Florence - do you know? > > > > > > > > > > > > Anyhow it is possible I have gone down the garden path so it would > > > > > > really help to see the relevant portion of the Dogtag debug log. > > > > > > (Be aware Dogtag timestamps are in local time, when you are looking > > > > > > for the relevant output). > > > > > > > > > > > > Cheers, > > > > > > Fraser > > > > > > > > > > > > On Tue, Dec 04, 2018 at 09:47:11PM -0500, Christopher Young via FreeIPA-users wrote: > > > > > > > Another thing I notice that confuses me... (see attached) > > > > > > > > > > > > > > > > > > > > > Is it normal to have this many certificate with the same Subject for > > > > > > > an IPA server? I'm wondering if somewhere along it renewed and yet > > > > > > > didn't update locally or something. I'm really not sure what's going > > > > > > > on here, but what I'm confused about is IF I wanted to generate a new > > > > > > > server for the IPA server, how would I go about doing that in a manner > > > > > > > where the certificate would have all the right attributes? (and would > > > > > > > I want to do that?) > > > > > > > > > > > > > > Sorry for all the questions. I'm figuring the pieces out as I go. > > > > > > > On Tue, Dec 4, 2018 at 9:04 PM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > > > > > > > > > > > > > Output: > > > > > > > > ---- > > > > > > > > [root@orldc-prod-ipa01 alias]# ipa-csreplica-manage list -v > > > > > > > > `hostname`.passur.local > > > > > > > > Directory Manager password: > > > > > > > > > > > > > > > > orldc-prod-ipa02.passur.local > > > > > > > > last init status: None > > > > > > > > last init ended: 1970-01-01 00:00:00+00:00 > > > > > > > > last update status: Error (-1) Problem connecting to replica - LDAP > > > > > > > > error: Can't contact LDAP server (connection error) > > > > > > > > last update ended: 1970-01-01 00:00:00+00:00 > > > > > > > > ---- > > > > > > > > > > > > > > > > Granted, it's replication partner (orldc-prod-ipa02) is the one that I > > > > > > > > mentioned has the issue starting at this point. So, that likely has > > > > > > > > something to do with this output. Having said that, I'm not quite > > > > > > > > sure what I should do here. I have definitely been issuing certs from > > > > > > > > this system. One note is that the 'hostname' command on my systems > > > > > > > > return only the short hostname. I'm not sure if that would be an > > > > > > > > issue or not, but it is worth noting. (To add 'domainname' does show > > > > > > > > the proper domainname for the system and what-not. > > > > > > > > > > > > > > > > Any ideas on the best way to fix just this host? I don't mind the > > > > > > > > idea of removing the CA replicas on the others after fixing this and > > > > > > > > re-replicating or anything. I just want to get this functioning > > > > > > > > before something exprires and I end up in a really bad spot (which is > > > > > > > > sadly only a day away!). (Why, oh why, do we always 'find' these > > > > > > > > type of problems under a time crunch in this business?) :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 4, 2018 at 5:57 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > > > > > > > > > > > > > > > > > Christopher Young via FreeIPA-users wrote: > > > > > > > > > > Yeah. I definitely lost on this one at this point. As far as I can > > > > > > > > > > tell, SOMEHOW I'm missing these certs in the directory? Does that > > > > > > > > > > sound right? > > > > > > > > > > > > > > > > > > > > How would one go about making sure is corrected? I'm guess I'd need > > > > > > > > > > to regenerate some type of certificate on the IPA host, but I'm afraid > > > > > > > > > > of breaking things worse. I have one more day before this one > > > > > > > > > > expires, so I'm trying to troubleshoot and fix it before then. This > > > > > > > > > > all started when I noticed that another IPA server/replica failed to > > > > > > > > > > restart. I think these two issues are related, but right now, my > > > > > > > > > > users are functional with just the 'ipa01' system (which has this > > > > > > > > > > ca-error' issue and the cert not found. I'm afraid to restart > > > > > > > > > > anything on that system because of that. > > > > > > > > > > > > > > > > > > > > I'm still reading and trying to understand and put the pieces > > > > > > > > > > together, however I'm worried about this issue. > > > > > > > > > > > > > > > > > > It sounds like one of your CA's is not replicating. You can use > > > > > > > > > ipa-csreplica-manage list -v `hostname` on each CA master to get the status. > > > > > > > > > > > > > > > > > > Any given replica can only store so much data to replicate so depending > > > > > > > > > on how long they have been disconnected could impact whether this is > > > > > > > > > easily recoverable. > > > > > > > > > > > > > > > > > > Given that, on any master it should always use its own CA (if there is > > > > > > > > > one) when issuing certs so this is a bit strange. > > > > > > > > > > > > > > > > > > rob > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Anyway, if anyone has any thoughts or tips here, I'd really appreciate > > > > > > > > > > it as I feel lost at this exact moment. > > > > > > > > > > On Tue, Dec 4, 2018 at 2:33 PM Christopher Young <mexigabacho(a)gmail.com&gt; wrote: > > > > > > > > > >> > > > > > > > > > >> IPA 4.5.4 (has been upgraded for years just to understand that there > > > > > > > > > >> is a history) > > > > > > > > > >> This system (ipa01) is the renewal master (in case that matters) > > > > > > > > > >> > > > > > > > > > >> I'm getting the following error on 'getcert'. My gut tells me this is > > > > > > > > > >> kinda a big deal. :) I really could use some help figuring this one > > > > > > > > > >> out as I'm not the most CA-versed. I have been learning quite a bit > > > > > > > > > >> reading some of the blogs, but there's definitely alot of ignorance of > > > > > > > > > >> the details on my part. > > > > > > > > > >> > > > > > > > > > >> The error: > > > > > > > > > >> ----------- > > > > > > > > > >> [root@orldc-prod-ipa01 log]# getcert list | grep -A12 -B1 error > > > > > > > > > >> status: MONITORING > > > > > > > > > >> ca-error: Server at > > > > > > > > > >> "http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit" > > > > > > > > > >> replied: Record not found > > > > > > > > > >> stuck: no > > > > > > > > > >> key pair storage: > > > > > > > > > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > > > > > > > > > >> cert-pki-ca',token='NSS Certificate DB',pin set > > > > > > > > > >> certificate: > > > > > > > > > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > > > > > > > > > >> cert-pki-ca',token='NSS Certificate DB' > > > > > > > > > >> CA: dogtag-ipa-ca-renew-agent > > > > > > > > > >> issuer: CN=Certificate Authority,O=PASSUR.LOCAL > > > > > > > > > >> subject: CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL > > > > > > > > > >> expires: 2018-12-06 21:43:50 UTC > > > > > > > > > >> key usage: > > > > > > > > > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > > > > > > > > >> eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection > > > > > > > > > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > > > > > > > > >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > > > > > > > > > >> "Server-Cert cert-pki-ca" > > > > > > > > > >> track: yes > > > > > > > > > >> ----------- > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> If I look at the cert referenced locally in the NSS DB: > > > > > > > > > >> ------ > > > > > > > > > >> [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias > > > > > > > > > >> -f /etc/httpd/alias/pwdfile.txt > > > > > > > > > >> > > > > > > > > > >> Certificate Nickname Trust Attributes > > > > > > > > > >> SSL,S/MIME,JAR/XPI > > > > > > > > > >> > > > > > > > > > >> Server-Cert cert-pki-ca u,u,u > > > > > > > > > >> auditSigningCert cert-pki-ca u,u,Pu > > > > > > > > > >> caSigningCert cert-pki-ca CTu,Cu,Cu > > > > > > > > > >> subsystemCert cert-pki-ca u,u,u > > > > > > > > > >> ocspSigningCert cert-pki-ca u,u,u > > > > > > > > > >> ------ > > > > > > > > > >> [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias > > > > > > > > > >> -f /etc/httpd/alias/pwdfile.txt -n 'Server-Cert cert-pki-ca' | grep > > > > > > > > > >> "Subject:\|Serial" > > > > > > > > > >> Serial Number: 268304422 (0xffe0026) > > > > > > > > > >> Subject: "CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL" > > > > > > > > > >> ----- > > > > > > > > > >> [root@orldc-prod-ipa01 log]# ipa cert-find --min-serial-number > > > > > > > > > >> 268304422 --max-serial-number 268304423 > > > > > > > > > >> ---------------------- > > > > > > > > > >> 0 certificates matched > > > > > > > > > >> ---------------------- > > > > > > > > > >> ----- > > > > > > > > > >> > > > > > > > > > >> I'm trying to figure out how to find this certificate. And IF somehow > > > > > > > > > >> it is wrong or missing, how do I fix such a scenario? > > > > > > > > > >> > > > > > > > > > >> Any help here is always appreciated! Unfortunately, I'm running out > > > > > > > > > >> of time based on the expiration date I see on 'getcert'. I'm not sure > > > > > > > > > >> of the ramifications, but this seems pretty critical on the surface. > > > > > > > > > >> > > > > > > > > > >> Thanks again for any help and direction! > > > > > > > > > >> > > > > > > > > > >> -- Chris > > > > > > > > > > _______________________________________________ > > > > > > > > > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > > > > > > > > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > > > > > > > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > > > > > > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > > > > > > > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > > > > > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > > > > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > > > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > > > > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > > > > >

Thank you so much! That appears to have worked! ----- [root@orldc-prod-ipa01 alias]# getcert list | grep 'pki-tomcat.*Server-Cert cert-pki-ca' -A10 -B3 Request ID '20181008203713': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PASSUR.LOCAL subject: CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL expires: 2020-11-24 18:09:49 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes ----- I'll this sit a bit, and then I've got to attack the issues I'm having on my 'ipa02' system (the first replica). Again, thanks alot for the help. On Wed, Dec 5, 2018 at 11:40 AM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > Christopher Young wrote: > > Another thing I notice that confuses me... (see attached) > > Yes. There are multiple services running on the same machine each with > their own private key. > > > Is it normal to have this many certificate with the same Subject for > > an IPA server? I'm wondering if somewhere along it renewed and yet > > didn't update locally or something. I'm really not sure what's going > > on here, but what I'm confused about is IF I wanted to generate a new > > server for the IPA server, how would I go about doing that in a manner > > where the certificate would have all the right attributes? (and would > > I want to do that?) > > > > Sorry for all the questions. I'm figuring the pieces out as I go. > > As Fraser mentioned elsewhere we ran into something that appears similar > and this is how we got around it. I can't guarantee this will fix things > for you but it seems like it might and should be safe to try. > > By default certmonger renews by serial number. So it tells the CA "hey, > I want the same type of cert I had for serial number X" and dogtag looks > it up and generates a new one. > > There is another option where a new CSR is sent instead, it just > requires a bit of manual work to do that. > > The steps are (for the case where your certs are still valid): > > 1. Stop certmonger > 2. grep dogtag-ipa-ca-renew-agent /var/lib/certmonger/cas/* > 3. There should be two. You want the one with "id=dogtag-ipa-ca-renew-agent" > 4. Modify that file and add -N to ca_external_helper. It needs to look like: > > ca_external_helper=/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit > -N > > -N means don't renew the existing serial number (certmonger will send > the CSR). For more details on the option see > certmonger-dogtag-ipa-renew-agent-submit(8). > > 5. Start certmonger > > certmonger may very well say "Hey! I've got a bunch of certs to renew, > let me get on that!" So watch the output of getcert list to see if it > already doing things. > > If this doesn't happen then manually resubmit the request doing > something like: > > getcert resubmit -d /etc/pki/pki-tomcat/alias -n 'Server-Cert > cert-pki-ca' > > This change should get you a renewed cert. > > AND CAPS SO YOU DON'T MISS THIS NEXT STEP :-) > > 6. Stop certmonger and remove the -N option so things should renew in > the future as they do in a typical environment. > > And that should do it. > > rob