Sina Owolabi via FreeIPA-users wrote: The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some pointers. I'd look at selftests.log first. My guess is that some of the CA certificates have failed to renew. getcert list | grep -i expires rob

Hi! getcert list | grep -i expires expires: 2019-04-13 12:08:20 UTC expires: 2019-04-13 12:08:06 UTC expires: 2019-04-13 12:07:50 UTC expires: 2035-06-01 08:33:01 UTC expires: 2019-04-13 12:07:41 UTC expires: 2019-04-13 12:06:55 UTC expires: 2019-05-05 12:06:41 UTC expires: 2019-05-05 12:06:56 UTC expires: 2020-01-17 19:56:03 UTC I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am creating one and running "ipactl restart". On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > Sina Owolabi via FreeIPA-users wrote: > > Hi! > > > > I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, > > api version 2.228), with one master, and two replicas, and I noticed > > that pki-tomcatd no longer works on the master, after attempting a > > reboot. > > pki-tomcatd works fine on the slaves. > > I noticed if I try to run IPA functions (dns record removal, hosts > > management, user passwords, etc), I receive responses like this: > > > > ipa: ERROR: Certificate operation cannot be completed: Unable to > > communicate with CMS (Internal Server Error) > > But on the replicas, functions work fine. > > Please can someone guide me on how to fix this? > > The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some > pointers. I'd look at selftests.log first. > > My guess is that some of the CA certificates have failed to renew. > > getcert list | grep -i expires > > rob

Hi! I tried to follow this solution for cert renewal for RHEL6: https://access.redhat.com/solutions/643753 (Sorry, desperation is setting in), but when I attempted Step 2, I got: # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" | grep -i after; done auditSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. ocspSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. subsystemCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. Server-Cert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. Could this be the root of my problems? And how can I convert them? On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > > Restarting ipa didnt create the logs. > Please, what else can i do? > > On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > > > > Hi! > > > > getcert list | grep -i expires > > expires: 2019-04-13 12:08:20 UTC > > expires: 2019-04-13 12:08:06 UTC > > expires: 2019-04-13 12:07:50 UTC > > expires: 2035-06-01 08:33:01 UTC > > expires: 2019-04-13 12:07:41 UTC > > expires: 2019-04-13 12:06:55 UTC > > expires: 2019-05-05 12:06:41 UTC > > expires: 2019-05-05 12:06:56 UTC > > expires: 2020-01-17 19:56:03 UTC > > > > I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am > > creating one and running "ipactl restart". > > > > On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > > > > > Sina Owolabi via FreeIPA-users wrote: > > > > Hi! > > > > > > > > I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, > > > > api version 2.228), with one master, and two replicas, and I noticed > > > > that pki-tomcatd no longer works on the master, after attempting a > > > > reboot. > > > > pki-tomcatd works fine on the slaves. > > > > I noticed if I try to run IPA functions (dns record removal, hosts > > > > management, user passwords, etc), I receive responses like this: > > > > > > > > ipa: ERROR: Certificate operation cannot be completed: Unable to > > > > communicate with CMS (Internal Server Error) > > > > But on the replicas, functions work fine. > > > > Please can someone guide me on how to fix this? > > > > > > The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some > > > pointers. I'd look at selftests.log first. > > > > > > My guess is that some of the CA certificates have failed to renew. > > > > > > getcert list | grep -i expires > > > > > > rob

Hi Florence and thanks for the help. ipactl status: [root@services ~]# ipactl status --ignore-service-failure; cat Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful systemctl status -l pki-tomcatd(a)pki-tomcat.service; cat ? pki-tomcatd(a)pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 1376 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd(a)pki-tomcat.service └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start systemctl status pki-tomcatd(a)pki-tomcat.service: Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process Mar 05 09:40:43 services.qrios.com server[1376]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Mar 05 09:40:43 services.qrios.com server[1376]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Mar 05 09:40:43 services.qrios.com server[1376]: at java.lang.Thread.run(Thread.java:748) On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud <flo(a)redhat.com&gt; wrote: > > On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote: > > Hi! > > > > I tried to follow this solution for cert renewal for RHEL6: > > https://access.redhat.com/solutions/643753 (Sorry, desperation is > > setting in), but when I attempted Step 2, I got: > > > Hi, > > 1. this note was written for RHEL 6 but you said in your first e-mail > that your server is running CentOS 7 with ipa 4.5.4. Please don't follow > those instructions as they are not adapted to your deployment. > The instructions for RHEL 7 are available at > https://access.redhat.com/solutions/3357261. > > 2. In a previous e-mail, the output of getcert list | grep -i expires > did not show any expired certificates, so I would not rush into wrong > conclusions. We need to understand first why pki did not start. > > What is the output of: > $ ipactl status > $ systemctl status pki-tomcatd(a)pki-tomcat.service > > flo > > > # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert > > cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do > > echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" > > | grep -i after; done > > auditSigningCert cert-pki-ca > > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > certificate/key database is in an old, unsupported format. > > ocspSigningCert cert-pki-ca > > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > certificate/key database is in an old, unsupported format. > > subsystemCert cert-pki-ca > > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > certificate/key database is in an old, unsupported format. > > Server-Cert cert-pki-ca > > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > certificate/key database is in an old, unsupported format. > > > > Could this be the root of my problems? > > And how can I convert them? > > > > On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > >> > >> Restarting ipa didnt create the logs. > >> Please, what else can i do? > >> > >> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > >>> > >>> Hi! > >>> > >>> getcert list | grep -i expires > >>> expires: 2019-04-13 12:08:20 UTC > >>> expires: 2019-04-13 12:08:06 UTC > >>> expires: 2019-04-13 12:07:50 UTC > >>> expires: 2035-06-01 08:33:01 UTC > >>> expires: 2019-04-13 12:07:41 UTC > >>> expires: 2019-04-13 12:06:55 UTC > >>> expires: 2019-05-05 12:06:41 UTC > >>> expires: 2019-05-05 12:06:56 UTC > >>> expires: 2020-01-17 19:56:03 UTC > >>> > >>> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am > >>> creating one and running "ipactl restart". > >>> > >>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > >>>> > >>>> Sina Owolabi via FreeIPA-users wrote: > >>>>> Hi! > >>>>> > >>>>> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, > >>>>> api version 2.228), with one master, and two replicas, and I noticed > >>>>> that pki-tomcatd no longer works on the master, after attempting a > >>>>> reboot. > >>>>> pki-tomcatd works fine on the slaves. > >>>>> I noticed if I try to run IPA functions (dns record removal, hosts > >>>>> management, user passwords, etc), I receive responses like this: > >>>>> > >>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to > >>>>> communicate with CMS (Internal Server Error) > >>>>> But on the replicas, functions work fine. > >>>>> Please can someone guide me on how to fix this? > >>>> > >>>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some > >>>> pointers. I'd look at selftests.log first. > >>>> > >>>> My guess is that some of the CA certificates have failed to renew. > >>>> > >>>> getcert list | grep -i expires > >>>> > >>>> rob > > _______________________________________________ > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > >

Log directories on the server: /var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/logs /var/log/pki/server/upgrade/10.1.2 /var/log/pki/server/upgrade/10.1.99 /var/log/pki/server/upgrade/10.2.1 /var/log/pki/server/upgrade/10.2.2 /var/log/pki/server/upgrade/10.2.3 /var/log/pki/server/upgrade/10.2.4 /var/log/pki/server/upgrade/10.2.5 /var/log/pki/server/upgrade/10.2.6 /var/log/pki/server/upgrade/10.3.0 /var/log/pki/server/upgrade/10.3.3 /var/log/pki/server/upgrade/10.4.0 /var/log/pki/server/upgrade/10.4.1 /var/log/pki/server/upgrade/10.5.1 /var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/logs are both empty. On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > Sina Owolabi wrote: > > Hi Florence > > > > and thanks for the help. > > ipactl status: > > [root@services ~]# ipactl status --ignore-service-failure; cat > > Directory Service: RUNNING > > krb5kdc Service: RUNNING > > kadmin Service: RUNNING > > named Service: RUNNING > > httpd Service: RUNNING > > ipa-custodia Service: RUNNING > > ntpd Service: RUNNING > > pki-tomcatd Service: STOPPED > > ipa-otpd Service: RUNNING > > ipa-dnskeysyncd Service: RUNNING > > ipa: INFO: The ipactl command was successful > > > > > > systemctl status -l pki-tomcatd(a)pki-tomcat.service; cat > > ? pki-tomcatd(a)pki-tomcat.service - PKI Tomcat Server pki-tomcat > > Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; > > vendor preset: disabled) > > Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago > > Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, > > status=0/SUCCESS) > > Main PID: 1376 (java) > > CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd(a)pki-tomcat.service > > └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java > > -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath > > /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar > > -Dcatalina.base=/var/lib/pki/pki-tomcat > > -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= > > -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp > > -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties > > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > > -Djava.security.manager > > -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy > > org.apache.catalina.startup.Bootstrap start > > > > systemctl status pki-tomcatd(a)pki-tomcat.service: > > > > Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception > > processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f > > background process > > Mar 05 09:40:43 services.qrios.com server[1376]: > > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > > Mar 05 09:40:43 services.qrios.com server[1376]: at > > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > > Mar 05 09:40:43 services.qrios.com server[1376]: at > > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > > Mar 05 09:40:43 services.qrios.com server[1376]: at > > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > > Mar 05 09:40:43 services.qrios.com server[1376]: at > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > > Mar 05 09:40:43 services.qrios.com server[1376]: at > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > Mar 05 09:40:43 services.qrios.com server[1376]: at > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > Mar 05 09:40:43 services.qrios.com server[1376]: at > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > > Mar 05 09:40:43 services.qrios.com server[1376]: at > > java.lang.Thread.run(Thread.java:748) > > The logs will contain much more useful information. dogtag keeps > changing the location of the logs and I forget exactly where it is in > your version but it's somewhere in /var/log/pki*/pki*/ca/... > > The log may be named debug or debug-<date> > > Also look at the selftest log in the same directory. > > There are a LOT of red herrings in the dogtag logs so proceed with caution. > > You do not need to touch or create anything for this logging to take > place. You should delete the directory you created. > > rob > > > > > > On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud <flo(a)redhat.com&gt; wrote: > >> > >> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote: > >>> Hi! > >>> > >>> I tried to follow this solution for cert renewal for RHEL6: > >>> https://access.redhat.com/solutions/643753 (Sorry, desperation is > >>> setting in), but when I attempted Step 2, I got: > >>> > >> Hi, > >> > >> 1. this note was written for RHEL 6 but you said in your first e-mail > >> that your server is running CentOS 7 with ipa 4.5.4. Please don't follow > >> those instructions as they are not adapted to your deployment. > >> The instructions for RHEL 7 are available at > >> https://access.redhat.com/solutions/3357261. > >> > >> 2. In a previous e-mail, the output of getcert list | grep -i expires > >> did not show any expired certificates, so I would not rush into wrong > >> conclusions. We need to understand first why pki did not start. > >> > >> What is the output of: > >> $ ipactl status > >> $ systemctl status pki-tomcatd(a)pki-tomcat.service > >> > >> flo > >> > >>> # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert > >>> cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do > >>> echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" > >>> | grep -i after; done > >>> auditSigningCert cert-pki-ca > >>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > >>> certificate/key database is in an old, unsupported format. > >>> ocspSigningCert cert-pki-ca > >>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > >>> certificate/key database is in an old, unsupported format. > >>> subsystemCert cert-pki-ca > >>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > >>> certificate/key database is in an old, unsupported format. > >>> Server-Cert cert-pki-ca > >>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > >>> certificate/key database is in an old, unsupported format. > >>> > >>> Could this be the root of my problems? > >>> And how can I convert them? > >>> > >>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > >>>> > >>>> Restarting ipa didnt create the logs. > >>>> Please, what else can i do? > >>>> > >>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > >>>>> > >>>>> Hi! > >>>>> > >>>>> getcert list | grep -i expires > >>>>> expires: 2019-04-13 12:08:20 UTC > >>>>> expires: 2019-04-13 12:08:06 UTC > >>>>> expires: 2019-04-13 12:07:50 UTC > >>>>> expires: 2035-06-01 08:33:01 UTC > >>>>> expires: 2019-04-13 12:07:41 UTC > >>>>> expires: 2019-04-13 12:06:55 UTC > >>>>> expires: 2019-05-05 12:06:41 UTC > >>>>> expires: 2019-05-05 12:06:56 UTC > >>>>> expires: 2020-01-17 19:56:03 UTC > >>>>> > >>>>> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am > >>>>> creating one and running "ipactl restart". > >>>>> > >>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > >>>>>> > >>>>>> Sina Owolabi via FreeIPA-users wrote: > >>>>>>> Hi! > >>>>>>> > >>>>>>> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, > >>>>>>> api version 2.228), with one master, and two replicas, and I noticed > >>>>>>> that pki-tomcatd no longer works on the master, after attempting a > >>>>>>> reboot. > >>>>>>> pki-tomcatd works fine on the slaves. > >>>>>>> I noticed if I try to run IPA functions (dns record removal, hosts > >>>>>>> management, user passwords, etc), I receive responses like this: > >>>>>>> > >>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to > >>>>>>> communicate with CMS (Internal Server Error) > >>>>>>> But on the replicas, functions work fine. > >>>>>>> Please can someone guide me on how to fix this? > >>>>>> > >>>>>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some > >>>>>> pointers. I'd look at selftests.log first. > >>>>>> > >>>>>> My guess is that some of the CA certificates have failed to renew. > >>>>>> > >>>>>> getcert list | grep -i expires > >>>>>> > >>>>>> rob > >>> _______________________________________________ > >>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > >>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > >>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > >>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > >>> > >> >

Hi Rob Today's catalina log file writes: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748) Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748) Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748) Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748) On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > Sina Owolabi wrote: > > Log directories on the server: > > > > /var/log/pki/pki-tomcat/ca/debug > > /var/log/pki/pki-tomcat/ca/logs > > /var/log/pki/server/upgrade/10.1.2 > > /var/log/pki/server/upgrade/10.1.99 > > /var/log/pki/server/upgrade/10.2.1 > > /var/log/pki/server/upgrade/10.2.2 > > /var/log/pki/server/upgrade/10.2.3 > > /var/log/pki/server/upgrade/10.2.4 > > /var/log/pki/server/upgrade/10.2.5 > > /var/log/pki/server/upgrade/10.2.6 > > /var/log/pki/server/upgrade/10.3.0 > > /var/log/pki/server/upgrade/10.3.3 > > /var/log/pki/server/upgrade/10.4.0 > > /var/log/pki/server/upgrade/10.4.1 > > /var/log/pki/server/upgrade/10.5.1 > > > > /var/log/pki/pki-tomcat/ca/debug > > You stated you had created this directory yourself. > > What are the contents of /var/log/pki/pki-tomcat/ca ? > > Could it be that the CA can't write its own logs? What does the latest > catalina log show in the parent directory? > > rob > > > /var/log/pki/pki-tomcat/ca/logs > > are both empty. > > > > On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > >> > >> Sina Owolabi wrote: > >>> Hi Florence > >>> > >>> and thanks for the help. > >>> ipactl status: > >>> [root@services ~]# ipactl status --ignore-service-failure; cat > >>> Directory Service: RUNNING > >>> krb5kdc Service: RUNNING > >>> kadmin Service: RUNNING > >>> named Service: RUNNING > >>> httpd Service: RUNNING > >>> ipa-custodia Service: RUNNING > >>> ntpd Service: RUNNING > >>> pki-tomcatd Service: STOPPED > >>> ipa-otpd Service: RUNNING > >>> ipa-dnskeysyncd Service: RUNNING > >>> ipa: INFO: The ipactl command was successful > >>> > >>> > >>> systemctl status -l pki-tomcatd(a)pki-tomcat.service; cat > >>> ? pki-tomcatd(a)pki-tomcat.service - PKI Tomcat Server pki-tomcat > >>> Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; > >>> vendor preset: disabled) > >>> Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago > >>> Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, > >>> status=0/SUCCESS) > >>> Main PID: 1376 (java) > >>> CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd(a)pki-tomcat.service > >>> └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java > >>> -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath > >>> /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar > >>> -Dcatalina.base=/var/lib/pki/pki-tomcat > >>> -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= > >>> -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp > >>> -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties > >>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > >>> -Djava.security.manager > >>> -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy > >>> org.apache.catalina.startup.Bootstrap start > >>> > >>> systemctl status pki-tomcatd(a)pki-tomcat.service: > >>> > >>> Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception > >>> processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f > >>> background process > >>> Mar 05 09:40:43 services.qrios.com server[1376]: > >>> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > >>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > >>> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > >>> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > >>> java.lang.Thread.run(Thread.java:748) > >> > >> The logs will contain much more useful information. dogtag keeps > >> changing the location of the logs and I forget exactly where it is in > >> your version but it's somewhere in /var/log/pki*/pki*/ca/... > >> > >> The log may be named debug or debug-<date> > >> > >> Also look at the selftest log in the same directory. > >> > >> There are a LOT of red herrings in the dogtag logs so proceed with caution. > >> > >> You do not need to touch or create anything for this logging to take > >> place. You should delete the directory you created. > >> > >> rob > >> > >> > >>> > >>> On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud <flo(a)redhat.com&gt; wrote: > >>>> > >>>> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote: > >>>>> Hi! > >>>>> > >>>>> I tried to follow this solution for cert renewal for RHEL6: > >>>>> https://access.redhat.com/solutions/643753 (Sorry, desperation is > >>>>> setting in), but when I attempted Step 2, I got: > >>>>> > >>>> Hi, > >>>> > >>>> 1. this note was written for RHEL 6 but you said in your first e-mail > >>>> that your server is running CentOS 7 with ipa 4.5.4. Please don't follow > >>>> those instructions as they are not adapted to your deployment. > >>>> The instructions for RHEL 7 are available at > >>>> https://access.redhat.com/solutions/3357261. > >>>> > >>>> 2. In a previous e-mail, the output of getcert list | grep -i expires > >>>> did not show any expired certificates, so I would not rush into wrong > >>>> conclusions. We need to understand first why pki did not start. > >>>> > >>>> What is the output of: > >>>> $ ipactl status > >>>> $ systemctl status pki-tomcatd(a)pki-tomcat.service > >>>> > >>>> flo > >>>> > >>>>> # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert > >>>>> cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do > >>>>> echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" > >>>>> | grep -i after; done > >>>>> auditSigningCert cert-pki-ca > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > >>>>> certificate/key database is in an old, unsupported format. > >>>>> ocspSigningCert cert-pki-ca > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > >>>>> certificate/key database is in an old, unsupported format. > >>>>> subsystemCert cert-pki-ca > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > >>>>> certificate/key database is in an old, unsupported format. > >>>>> Server-Cert cert-pki-ca > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > >>>>> certificate/key database is in an old, unsupported format. > >>>>> > >>>>> Could this be the root of my problems? > >>>>> And how can I convert them? > >>>>> > >>>>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > >>>>>> > >>>>>> Restarting ipa didnt create the logs. > >>>>>> Please, what else can i do? > >>>>>> > >>>>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > >>>>>>> > >>>>>>> Hi! > >>>>>>> > >>>>>>> getcert list | grep -i expires > >>>>>>> expires: 2019-04-13 12:08:20 UTC > >>>>>>> expires: 2019-04-13 12:08:06 UTC > >>>>>>> expires: 2019-04-13 12:07:50 UTC > >>>>>>> expires: 2035-06-01 08:33:01 UTC > >>>>>>> expires: 2019-04-13 12:07:41 UTC > >>>>>>> expires: 2019-04-13 12:06:55 UTC > >>>>>>> expires: 2019-05-05 12:06:41 UTC > >>>>>>> expires: 2019-05-05 12:06:56 UTC > >>>>>>> expires: 2020-01-17 19:56:03 UTC > >>>>>>> > >>>>>>> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am > >>>>>>> creating one and running "ipactl restart". > >>>>>>> > >>>>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > >>>>>>>> > >>>>>>>> Sina Owolabi via FreeIPA-users wrote: > >>>>>>>>> Hi! > >>>>>>>>> > >>>>>>>>> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, > >>>>>>>>> api version 2.228), with one master, and two replicas, and I noticed > >>>>>>>>> that pki-tomcatd no longer works on the master, after attempting a > >>>>>>>>> reboot. > >>>>>>>>> pki-tomcatd works fine on the slaves. > >>>>>>>>> I noticed if I try to run IPA functions (dns record removal, hosts > >>>>>>>>> management, user passwords, etc), I receive responses like this: > >>>>>>>>> > >>>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to > >>>>>>>>> communicate with CMS (Internal Server Error) > >>>>>>>>> But on the replicas, functions work fine. > >>>>>>>>> Please can someone guide me on how to fix this? > >>>>>>>> > >>>>>>>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some > >>>>>>>> pointers. I'd look at selftests.log first. > >>>>>>>> > >>>>>>>> My guess is that some of the CA certificates have failed to renew. > >>>>>>>> > >>>>>>>> getcert list | grep -i expires > >>>>>>>> > >>>>>>>> rob > >>>>> _______________________________________________ > >>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > >>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > >>>>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > >>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > >>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > >>>>> > >>>> > >> >

Sina Owolabi via FreeIPA-users wrote: Hard to know. If the process was already running at the time things may have appeared ok until it was restarted. debug is a log file, not a directory. My 4.4.4 install contains the following in /var/log/pki: drwxr-xr-x. 3 root root 21 Mar 30 2017 ./server drwxrwx---. 3 pkiuser pkiuser 12288 Mar 6 01:22 ./pki-tomcat drwxrwx---. 4 pkiuser pkiuser 4096 Feb 7 11:27 ./pki-tomcat/ca drwxrwx---. 2 pkiuser pkiuser 86 Dec 4 11:00 ./pki-tomcat/ca/archive drwxrwx---. 2 pkiuser pkiuser 84 Feb 7 11:27 ./pki-tomcat/ca/signedAudit Be sure to run restorecon -R on /var/log/pki to ensure the SELinux contexts are correct. rob

Would reinstalling IPA software help? On Wed, Mar 6, 2019 at 10:05 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > > Ah.. the server has SELinux disabled. > Curious why its not recreating its log directories after a restart. > > On Wed, Mar 6, 2019 at 4:33 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > > > Sina Owolabi via FreeIPA-users wrote: > > > Hi Rob > > > > > > Sorry I missed the full question: > > > What are the contents of /var/log/pki/pki-tomcat/ca ? > > > > > > Could it be that the CA can't write its own logs? What does the latest > > > catalina log show in the parent directory? > > > > > > /var/log/pki/pki-tomcat/ca was empty until I created > > > /var/log/pki/pki-tomcat/ca/logs and > > > /var/log/pki/pki-tomcat/ca/debug directories. > > > I dont think the ca would have trouble writing its logs, the structure > > > is all owned by pkiuser: > > > drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca > > > > > > Now that I think about it, I do remember some issues with runaway logs > > > filling up /var/log, and > > > I deleted some directories, and recreated them, but I dont think > > > pki-tomcat suffered then. > > > > Hard to know. If the process was already running at the time things may > > have appeared ok until it was restarted. > > > > debug is a log file, not a directory. > > > > My 4.4.4 install contains the following in /var/log/pki: > > > > drwxr-xr-x. 3 root root 21 Mar 30 2017 ./server > > drwxrwx---. 3 pkiuser pkiuser 12288 Mar 6 01:22 ./pki-tomcat > > drwxrwx---. 4 pkiuser pkiuser 4096 Feb 7 11:27 ./pki-tomcat/ca > > drwxrwx---. 2 pkiuser pkiuser 86 Dec 4 11:00 ./pki-tomcat/ca/archive > > drwxrwx---. 2 pkiuser pkiuser 84 Feb 7 11:27 ./pki-tomcat/ca/signedAudit > > > > Be sure to run restorecon -R on /var/log/pki to ensure the SELinux > > contexts are correct. > > > > rob > > > > > > > > On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > > >> > > >> Hi Rob > > >> > > >> Today's catalina log file writes: > > >> > > >> WARNING: Exception processing realm > > >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process > > >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > > >> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > > >> at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > > >> at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > > >> at java.lang.Thread.run(Thread.java:748) > > >> > > >> Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase > > >> backgroundProcess > > >> WARNING: Exception processing realm > > >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process > > >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > > >> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > > >> at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > > >> at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > > >> at java.lang.Thread.run(Thread.java:748) > > >> > > >> Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase > > >> backgroundProcess > > >> WARNING: Exception processing realm > > >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process > > >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > > >> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > > >> at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > > >> at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > > >> at java.lang.Thread.run(Thread.java:748) > > >> > > >> Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase > > >> backgroundProcess > > >> WARNING: Exception processing realm > > >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process > > >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > > >> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > > >> at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > > >> at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > > >> at java.lang.Thread.run(Thread.java:748) > > >> > > >> On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > >>> > > >>> Sina Owolabi wrote: > > >>>> Log directories on the server: > > >>>> > > >>>> /var/log/pki/pki-tomcat/ca/debug > > >>>> /var/log/pki/pki-tomcat/ca/logs > > >>>> /var/log/pki/server/upgrade/10.1.2 > > >>>> /var/log/pki/server/upgrade/10.1.99 > > >>>> /var/log/pki/server/upgrade/10.2.1 > > >>>> /var/log/pki/server/upgrade/10.2.2 > > >>>> /var/log/pki/server/upgrade/10.2.3 > > >>>> /var/log/pki/server/upgrade/10.2.4 > > >>>> /var/log/pki/server/upgrade/10.2.5 > > >>>> /var/log/pki/server/upgrade/10.2.6 > > >>>> /var/log/pki/server/upgrade/10.3.0 > > >>>> /var/log/pki/server/upgrade/10.3.3 > > >>>> /var/log/pki/server/upgrade/10.4.0 > > >>>> /var/log/pki/server/upgrade/10.4.1 > > >>>> /var/log/pki/server/upgrade/10.5.1 > > >>>> > > >>>> /var/log/pki/pki-tomcat/ca/debug > > >>> > > >>> You stated you had created this directory yourself. > > >>> > > >>> What are the contents of /var/log/pki/pki-tomcat/ca ? > > >>> > > >>> Could it be that the CA can't write its own logs? What does the latest > > >>> catalina log show in the parent directory? > > >>> > > >>> rob > > >>> > > >>>> /var/log/pki/pki-tomcat/ca/logs > > >>>> are both empty. > > >>>> > > >>>> On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > >>>>> > > >>>>> Sina Owolabi wrote: > > >>>>>> Hi Florence > > >>>>>> > > >>>>>> and thanks for the help. > > >>>>>> ipactl status: > > >>>>>> [root@services ~]# ipactl status --ignore-service-failure; cat > > >>>>>> Directory Service: RUNNING > > >>>>>> krb5kdc Service: RUNNING > > >>>>>> kadmin Service: RUNNING > > >>>>>> named Service: RUNNING > > >>>>>> httpd Service: RUNNING > > >>>>>> ipa-custodia Service: RUNNING > > >>>>>> ntpd Service: RUNNING > > >>>>>> pki-tomcatd Service: STOPPED > > >>>>>> ipa-otpd Service: RUNNING > > >>>>>> ipa-dnskeysyncd Service: RUNNING > > >>>>>> ipa: INFO: The ipactl command was successful > > >>>>>> > > >>>>>> > > >>>>>> systemctl status -l pki-tomcatd(a)pki-tomcat.service; cat > > >>>>>> ? pki-tomcatd(a)pki-tomcat.service - PKI Tomcat Server pki-tomcat > > >>>>>> Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; > > >>>>>> vendor preset: disabled) > > >>>>>> Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago > > >>>>>> Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, > > >>>>>> status=0/SUCCESS) > > >>>>>> Main PID: 1376 (java) > > >>>>>> CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd(a)pki-tomcat.service > > >>>>>> └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java > > >>>>>> -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath > > >>>>>> /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar > > >>>>>> -Dcatalina.base=/var/lib/pki/pki-tomcat > > >>>>>> -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= > > >>>>>> -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp > > >>>>>> -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties > > >>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > > >>>>>> -Djava.security.manager > > >>>>>> -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy > > >>>>>> org.apache.catalina.startup.Bootstrap start > > >>>>>> > > >>>>>> systemctl status pki-tomcatd(a)pki-tomcat.service: > > >>>>>> > > >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception > > >>>>>> processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f > > >>>>>> background process > > >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: > > >>>>>> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > > >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>>>>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > > >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>>>>> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > > >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>>>>> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > > >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > > >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > > >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>>>>> java.lang.Thread.run(Thread.java:748) > > >>>>> > > >>>>> The logs will contain much more useful information. dogtag keeps > > >>>>> changing the location of the logs and I forget exactly where it is in > > >>>>> your version but it's somewhere in /var/log/pki*/pki*/ca/... > > >>>>> > > >>>>> The log may be named debug or debug-<date> > > >>>>> > > >>>>> Also look at the selftest log in the same directory. > > >>>>> > > >>>>> There are a LOT of red herrings in the dogtag logs so proceed with caution. > > >>>>> > > >>>>> You do not need to touch or create anything for this logging to take > > >>>>> place. You should delete the directory you created. > > >>>>> > > >>>>> rob > > >>>>> > > >>>>> > > >>>>>> > > >>>>>> On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud <flo(a)redhat.com&gt; wrote: > > >>>>>>> > > >>>>>>> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote: > > >>>>>>>> Hi! > > >>>>>>>> > > >>>>>>>> I tried to follow this solution for cert renewal for RHEL6: > > >>>>>>>> https://access.redhat.com/solutions/643753 (Sorry, desperation is > > >>>>>>>> setting in), but when I attempted Step 2, I got: > > >>>>>>>> > > >>>>>>> Hi, > > >>>>>>> > > >>>>>>> 1. this note was written for RHEL 6 but you said in your first e-mail > > >>>>>>> that your server is running CentOS 7 with ipa 4.5.4. Please don't follow > > >>>>>>> those instructions as they are not adapted to your deployment. > > >>>>>>> The instructions for RHEL 7 are available at > > >>>>>>> https://access.redhat.com/solutions/3357261. > > >>>>>>> > > >>>>>>> 2. In a previous e-mail, the output of getcert list | grep -i expires > > >>>>>>> did not show any expired certificates, so I would not rush into wrong > > >>>>>>> conclusions. We need to understand first why pki did not start. > > >>>>>>> > > >>>>>>> What is the output of: > > >>>>>>> $ ipactl status > > >>>>>>> $ systemctl status pki-tomcatd(a)pki-tomcat.service > > >>>>>>> > > >>>>>>> flo > > >>>>>>> > > >>>>>>>> # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert > > >>>>>>>> cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do > > >>>>>>>> echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" > > >>>>>>>> | grep -i after; done > > >>>>>>>> auditSigningCert cert-pki-ca > > >>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > >>>>>>>> certificate/key database is in an old, unsupported format. > > >>>>>>>> ocspSigningCert cert-pki-ca > > >>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > >>>>>>>> certificate/key database is in an old, unsupported format. > > >>>>>>>> subsystemCert cert-pki-ca > > >>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > >>>>>>>> certificate/key database is in an old, unsupported format. > > >>>>>>>> Server-Cert cert-pki-ca > > >>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > >>>>>>>> certificate/key database is in an old, unsupported format. > > >>>>>>>> > > >>>>>>>> Could this be the root of my problems? > > >>>>>>>> And how can I convert them? > > >>>>>>>> > > >>>>>>>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > > >>>>>>>>> > > >>>>>>>>> Restarting ipa didnt create the logs. > > >>>>>>>>> Please, what else can i do? > > >>>>>>>>> > > >>>>>>>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi <notify.sina(a)gmail.com&gt; wrote: > > >>>>>>>>>> > > >>>>>>>>>> Hi! > > >>>>>>>>>> > > >>>>>>>>>> getcert list | grep -i expires > > >>>>>>>>>> expires: 2019-04-13 12:08:20 UTC > > >>>>>>>>>> expires: 2019-04-13 12:08:06 UTC > > >>>>>>>>>> expires: 2019-04-13 12:07:50 UTC > > >>>>>>>>>> expires: 2035-06-01 08:33:01 UTC > > >>>>>>>>>> expires: 2019-04-13 12:07:41 UTC > > >>>>>>>>>> expires: 2019-04-13 12:06:55 UTC > > >>>>>>>>>> expires: 2019-05-05 12:06:41 UTC > > >>>>>>>>>> expires: 2019-05-05 12:06:56 UTC > > >>>>>>>>>> expires: 2020-01-17 19:56:03 UTC > > >>>>>>>>>> > > >>>>>>>>>> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am > > >>>>>>>>>> creating one and running "ipactl restart". > > >>>>>>>>>> > > >>>>>>>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote: > > >>>>>>>>>>> > > >>>>>>>>>>> Sina Owolabi via FreeIPA-users wrote: > > >>>>>>>>>>>> Hi! > > >>>>>>>>>>>> > > >>>>>>>>>>>> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, > > >>>>>>>>>>>> api version 2.228), with one master, and two replicas, and I noticed > > >>>>>>>>>>>> that pki-tomcatd no longer works on the master, after attempting a > > >>>>>>>>>>>> reboot. > > >>>>>>>>>>>> pki-tomcatd works fine on the slaves. > > >>>>>>>>>>>> I noticed if I try to run IPA functions (dns record removal, hosts > > >>>>>>>>>>>> management, user passwords, etc), I receive responses like this: > > >>>>>>>>>>>> > > >>>>>>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to > > >>>>>>>>>>>> communicate with CMS (Internal Server Error) > > >>>>>>>>>>>> But on the replicas, functions work fine. > > >>>>>>>>>>>> Please can someone guide me on how to fix this? > > >>>>>>>>>>> > > >>>>>>>>>>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some > > >>>>>>>>>>> pointers. I'd look at selftests.log first. > > >>>>>>>>>>> > > >>>>>>>>>>> My guess is that some of the CA certificates have failed to renew. > > >>>>>>>>>>> > > >>>>>>>>>>> getcert list | grep -i expires > > >>>>>>>>>>> > > >>>>>>>>>>> rob > > >>>>>>>> _______________________________________________ > > >>>>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > >>>>>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > >>>>>>>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > >>>>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > >>>>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > >>>>>>>> > > >>>>>>> > > >>>>> > > >>> > > > _______________________________________________ > > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > > > >