Hi Rob
Sorry I missed the full question:
What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest
catalina log show in the parent directory?
/var/log/pki/pki-tomcat/ca was empty until I created
/var/log/pki/pki-tomcat/ca/logs and
/var/log/pki/pki-tomcat/ca/debug directories.
I dont think the ca would have trouble writing its logs, the structure
is all owned by pkiuser:
drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca
Now that I think about it, I do remember some issues with runaway logs
filling up /var/log, and
I deleted some directories, and recreated them, but I dont think
pki-tomcat suffered then.
On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi <notify.sina(a)gmail.com> wrote:
>
> Hi Rob
>
> Today's catalina log file writes:
>
> WARNING: Exception processing realm
> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> at java.lang.Thread.run(Thread.java:748)
>
> Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase
> backgroundProcess
> WARNING: Exception processing realm
> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> at java.lang.Thread.run(Thread.java:748)
>
> Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase
> backgroundProcess
> WARNING: Exception processing realm
> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> at java.lang.Thread.run(Thread.java:748)
>
> Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase
> backgroundProcess
> WARNING: Exception processing realm
> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> at java.lang.Thread.run(Thread.java:748)
>
> On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden <rcritten(a)redhat.com> wrote:
> >
> > Sina Owolabi wrote:
> > > Log directories on the server:
> > >
> > > /var/log/pki/pki-tomcat/ca/debug
> > > /var/log/pki/pki-tomcat/ca/logs
> > > /var/log/pki/server/upgrade/10.1.2
> > > /var/log/pki/server/upgrade/10.1.99
> > > /var/log/pki/server/upgrade/10.2.1
> > > /var/log/pki/server/upgrade/10.2.2
> > > /var/log/pki/server/upgrade/10.2.3
> > > /var/log/pki/server/upgrade/10.2.4
> > > /var/log/pki/server/upgrade/10.2.5
> > > /var/log/pki/server/upgrade/10.2.6
> > > /var/log/pki/server/upgrade/10.3.0
> > > /var/log/pki/server/upgrade/10.3.3
> > > /var/log/pki/server/upgrade/10.4.0
> > > /var/log/pki/server/upgrade/10.4.1
> > > /var/log/pki/server/upgrade/10.5.1
> > >
> > > /var/log/pki/pki-tomcat/ca/debug
> >
> > You stated you had created this directory yourself.
> >
> > What are the contents of /var/log/pki/pki-tomcat/ca ?
> >
> > Could it be that the CA can't write its own logs? What does the latest
> > catalina log show in the parent directory?
> >
> > rob
> >
> > > /var/log/pki/pki-tomcat/ca/logs
> > > are both empty.
> > >
> > > On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden <rcritten(a)redhat.com>
wrote:
> > >>
> > >> Sina Owolabi wrote:
> > >>> Hi Florence
> > >>>
> > >>> and thanks for the help.
> > >>> ipactl status:
> > >>> [root@services ~]# ipactl status --ignore-service-failure; cat
> > >>> Directory Service: RUNNING
> > >>> krb5kdc Service: RUNNING
> > >>> kadmin Service: RUNNING
> > >>> named Service: RUNNING
> > >>> httpd Service: RUNNING
> > >>> ipa-custodia Service: RUNNING
> > >>> ntpd Service: RUNNING
> > >>> pki-tomcatd Service: STOPPED
> > >>> ipa-otpd Service: RUNNING
> > >>> ipa-dnskeysyncd Service: RUNNING
> > >>> ipa: INFO: The ipactl command was successful
> > >>>
> > >>>
> > >>> systemctl status -l pki-tomcatd(a)pki-tomcat.service; cat
> > >>> ? pki-tomcatd(a)pki-tomcat.service - PKI Tomcat Server pki-tomcat
> > >>> Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service;
enabled;
> > >>> vendor preset: disabled)
> > >>> Active: active (running) since Tue 2019-03-05 09:14:15 WAT;
26min ago
> > >>> Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i
(code=exited,
> > >>> status=0/SUCCESS)
> > >>> Main PID: 1376 (java)
> > >>> CGroup:
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd(a)pki-tomcat.service
> > >>> └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
> > >>> -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath
> > >>>
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
> > >>> -Dcatalina.base=/var/lib/pki/pki-tomcat
> > >>> -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs=
> > >>> -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
> > >>>
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties
> > >>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> > >>> -Djava.security.manager
> > >>>
-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
> > >>> org.apache.catalina.startup.Bootstrap start
> > >>>
> > >>> systemctl status pki-tomcatd(a)pki-tomcat.service:
> > >>>
> > >>> Mar 05 09:40:43
services.qrios.com server[1376]: WARNING:
Exception
> > >>> processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f
> > >>> background process
> > >>> Mar 05 09:40:43
services.qrios.com server[1376]:
> > >>> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > >>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > >>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > >>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > >>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > >>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > >>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>> java.lang.Thread.run(Thread.java:748)
> > >>
> > >> The logs will contain much more useful information. dogtag keeps
> > >> changing the location of the logs and I forget exactly where it is in
> > >> your version but it's somewhere in /var/log/pki*/pki*/ca/...
> > >>
> > >> The log may be named debug or debug-<date>
> > >>
> > >> Also look at the selftest log in the same directory.
> > >>
> > >> There are a LOT of red herrings in the dogtag logs so proceed with
caution.
> > >>
> > >> You do not need to touch or create anything for this logging to take
> > >> place. You should delete the directory you created.
> > >>
> > >> rob
> > >>
> > >>
> > >>>
> > >>> On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud
<flo(a)redhat.com> wrote:
> > >>>>
> > >>>> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote:
> > >>>>> Hi!
> > >>>>>
> > >>>>> I tried to follow this solution for cert renewal for
RHEL6:
> > >>>>>
https://access.redhat.com/solutions/643753 (Sorry,
desperation is
> > >>>>> setting in), but when I attempted Step 2, I got:
> > >>>>>
> > >>>> Hi,
> > >>>>
> > >>>> 1. this note was written for RHEL 6 but you said in your first
e-mail
> > >>>> that your server is running CentOS 7 with ipa 4.5.4. Please
don't follow
> > >>>> those instructions as they are not adapted to your deployment.
> > >>>> The instructions for RHEL 7 are available at
> > >>>>
https://access.redhat.com/solutions/3357261.
> > >>>>
> > >>>> 2. In a previous e-mail, the output of getcert list | grep -i
expires
> > >>>> did not show any expired certificates, so I would not rush into
wrong
> > >>>> conclusions. We need to understand first why pki did not
start.
> > >>>>
> > >>>> What is the output of:
> > >>>> $ ipactl status
> > >>>> $ systemctl status pki-tomcatd(a)pki-tomcat.service
> > >>>>
> > >>>> flo
> > >>>>
> > >>>>> # for nickname in "auditSigningCert cert-pki-ca"
"ocspSigningCert
> > >>>>> cert-pki-ca" "subsystemCert cert-pki-ca"
"Server-Cert cert-pki-ca"; do
> > >>>>> echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n
"${nickname}"
> > >>>>> | grep -i after; done
> > >>>>> auditSigningCert cert-pki-ca
> > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The
> > >>>>> certificate/key database is in an old, unsupported format.
> > >>>>> ocspSigningCert cert-pki-ca
> > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The
> > >>>>> certificate/key database is in an old, unsupported format.
> > >>>>> subsystemCert cert-pki-ca
> > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The
> > >>>>> certificate/key database is in an old, unsupported format.
> > >>>>> Server-Cert cert-pki-ca
> > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The
> > >>>>> certificate/key database is in an old, unsupported format.
> > >>>>>
> > >>>>> Could this be the root of my problems?
> > >>>>> And how can I convert them?
> > >>>>>
> > >>>>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi
<notify.sina(a)gmail.com> wrote:
> > >>>>>>
> > >>>>>> Restarting ipa didnt create the logs.
> > >>>>>> Please, what else can i do?
> > >>>>>>
> > >>>>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi
<notify.sina(a)gmail.com> wrote:
> > >>>>>>>
> > >>>>>>> Hi!
> > >>>>>>>
> > >>>>>>> getcert list | grep -i expires
> > >>>>>>> expires: 2019-04-13 12:08:20 UTC
> > >>>>>>> expires: 2019-04-13 12:08:06 UTC
> > >>>>>>> expires: 2019-04-13 12:07:50 UTC
> > >>>>>>> expires: 2035-06-01 08:33:01 UTC
> > >>>>>>> expires: 2019-04-13 12:07:41 UTC
> > >>>>>>> expires: 2019-04-13 12:06:55 UTC
> > >>>>>>> expires: 2019-05-05 12:06:41 UTC
> > >>>>>>> expires: 2019-05-05 12:06:56 UTC
> > >>>>>>> expires: 2020-01-17 19:56:03 UTC
> > >>>>>>>
> > >>>>>>> I didnt find a /var/log/pki/pki-tomcat/ca/debug
directory, but I am
> > >>>>>>> creating one and running "ipactl
restart".
> > >>>>>>>
> > >>>>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden
<rcritten(a)redhat.com> wrote:
> > >>>>>>>>
> > >>>>>>>> Sina Owolabi via FreeIPA-users wrote:
> > >>>>>>>>> Hi!
> > >>>>>>>>>
> > >>>>>>>>> I am running a small IPA domain (CentOS 7
servers, ipa version 4.5.4,
> > >>>>>>>>> api version 2.228), with one master, and
two replicas, and I noticed
> > >>>>>>>>> that pki-tomcatd no longer works on the
master, after attempting a
> > >>>>>>>>> reboot.
> > >>>>>>>>> pki-tomcatd works fine on the slaves.
> > >>>>>>>>> I noticed if I try to run IPA functions
(dns record removal, hosts
> > >>>>>>>>> management, user passwords, etc), I receive
responses like this:
> > >>>>>>>>>
> > >>>>>>>>> ipa: ERROR: Certificate operation cannot be
completed: Unable to
> > >>>>>>>>> communicate with CMS (Internal Server
Error)
> > >>>>>>>>> But on the replicas, functions work fine.
> > >>>>>>>>> Please can someone guide me on how to fix
this?
> > >>>>>>>>
> > >>>>>>>> The CA log is in
/var/log/pki/pki-tomcat/ca/debug. That may have some
> > >>>>>>>> pointers. I'd look at selftests.log first.
> > >>>>>>>>
> > >>>>>>>> My guess is that some of the CA certificates
have failed to renew.
> > >>>>>>>>
> > >>>>>>>> getcert list | grep -i expires
> > >>>>>>>>
> > >>>>>>>> rob
> > >>>>> _______________________________________________
> > >>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
> > >>>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
> > >>>>> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
> > >>>>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > >>>>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
> > >>>>>
> > >>>>
> > >>
> >