Ah.. the server has SELinux disabled.
Curious why its not recreating its log directories after a restart.
On Wed, Mar 6, 2019 at 4:33 PM Rob Crittenden <rcritten(a)redhat.com> wrote:
>
> Sina Owolabi via FreeIPA-users wrote:
> > Hi Rob
> >
> > Sorry I missed the full question:
> > What are the contents of /var/log/pki/pki-tomcat/ca ?
> >
> > Could it be that the CA can't write its own logs? What does the latest
> > catalina log show in the parent directory?
> >
> > /var/log/pki/pki-tomcat/ca was empty until I created
> > /var/log/pki/pki-tomcat/ca/logs and
> > /var/log/pki/pki-tomcat/ca/debug directories.
> > I dont think the ca would have trouble writing its logs, the structure
> > is all owned by pkiuser:
> > drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca
> >
> > Now that I think about it, I do remember some issues with runaway logs
> > filling up /var/log, and
> > I deleted some directories, and recreated them, but I dont think
> > pki-tomcat suffered then.
>
> Hard to know. If the process was already running at the time things may
> have appeared ok until it was restarted.
>
> debug is a log file, not a directory.
>
> My 4.4.4 install contains the following in /var/log/pki:
>
> drwxr-xr-x. 3 root root 21 Mar 30 2017 ./server
> drwxrwx---. 3 pkiuser pkiuser 12288 Mar 6 01:22 ./pki-tomcat
> drwxrwx---. 4 pkiuser pkiuser 4096 Feb 7 11:27 ./pki-tomcat/ca
> drwxrwx---. 2 pkiuser pkiuser 86 Dec 4 11:00 ./pki-tomcat/ca/archive
> drwxrwx---. 2 pkiuser pkiuser 84 Feb 7 11:27 ./pki-tomcat/ca/signedAudit
>
> Be sure to run restorecon -R on /var/log/pki to ensure the SELinux
> contexts are correct.
>
> rob
>
> >
> > On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi <notify.sina(a)gmail.com>
wrote:
> >>
> >> Hi Rob
> >>
> >> Today's catalina log file writes:
> >>
> >> WARNING: Exception processing realm
> >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> >> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> >> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> >> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >> Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase
> >> backgroundProcess
> >> WARNING: Exception processing realm
> >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> >> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> >> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> >> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >> Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase
> >> backgroundProcess
> >> WARNING: Exception processing realm
> >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> >> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> >> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> >> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >> Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase
> >> backgroundProcess
> >> WARNING: Exception processing realm
> >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> >> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> >> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> >> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >> On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden <rcritten(a)redhat.com>
wrote:
> >>>
> >>> Sina Owolabi wrote:
> >>>> Log directories on the server:
> >>>>
> >>>> /var/log/pki/pki-tomcat/ca/debug
> >>>> /var/log/pki/pki-tomcat/ca/logs
> >>>> /var/log/pki/server/upgrade/10.1.2
> >>>> /var/log/pki/server/upgrade/10.1.99
> >>>> /var/log/pki/server/upgrade/10.2.1
> >>>> /var/log/pki/server/upgrade/10.2.2
> >>>> /var/log/pki/server/upgrade/10.2.3
> >>>> /var/log/pki/server/upgrade/10.2.4
> >>>> /var/log/pki/server/upgrade/10.2.5
> >>>> /var/log/pki/server/upgrade/10.2.6
> >>>> /var/log/pki/server/upgrade/10.3.0
> >>>> /var/log/pki/server/upgrade/10.3.3
> >>>> /var/log/pki/server/upgrade/10.4.0
> >>>> /var/log/pki/server/upgrade/10.4.1
> >>>> /var/log/pki/server/upgrade/10.5.1
> >>>>
> >>>> /var/log/pki/pki-tomcat/ca/debug
> >>>
> >>> You stated you had created this directory yourself.
> >>>
> >>> What are the contents of /var/log/pki/pki-tomcat/ca ?
> >>>
> >>> Could it be that the CA can't write its own logs? What does the
latest
> >>> catalina log show in the parent directory?
> >>>
> >>> rob
> >>>
> >>>> /var/log/pki/pki-tomcat/ca/logs
> >>>> are both empty.
> >>>>
> >>>> On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden
<rcritten(a)redhat.com> wrote:
> >>>>>
> >>>>> Sina Owolabi wrote:
> >>>>>> Hi Florence
> >>>>>>
> >>>>>> and thanks for the help.
> >>>>>> ipactl status:
> >>>>>> [root@services ~]# ipactl status --ignore-service-failure;
cat
> >>>>>> Directory Service: RUNNING
> >>>>>> krb5kdc Service: RUNNING
> >>>>>> kadmin Service: RUNNING
> >>>>>> named Service: RUNNING
> >>>>>> httpd Service: RUNNING
> >>>>>> ipa-custodia Service: RUNNING
> >>>>>> ntpd Service: RUNNING
> >>>>>> pki-tomcatd Service: STOPPED
> >>>>>> ipa-otpd Service: RUNNING
> >>>>>> ipa-dnskeysyncd Service: RUNNING
> >>>>>> ipa: INFO: The ipactl command was successful
> >>>>>>
> >>>>>>
> >>>>>> systemctl status -l pki-tomcatd(a)pki-tomcat.service; cat
> >>>>>> ? pki-tomcatd(a)pki-tomcat.service - PKI Tomcat Server
pki-tomcat
> >>>>>> Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service;
enabled;
> >>>>>> vendor preset: disabled)
> >>>>>> Active: active (running) since Tue 2019-03-05 09:14:15
WAT; 26min ago
> >>>>>> Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i
(code=exited,
> >>>>>> status=0/SUCCESS)
> >>>>>> Main PID: 1376 (java)
> >>>>>> CGroup:
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd(a)pki-tomcat.service
> >>>>>> └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
> >>>>>> -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath
> >>>>>>
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
> >>>>>> -Dcatalina.base=/var/lib/pki/pki-tomcat
> >>>>>> -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs=
> >>>>>> -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
> >>>>>>
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties
> >>>>>>
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> >>>>>> -Djava.security.manager
> >>>>>>
-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
> >>>>>> org.apache.catalina.startup.Bootstrap start
> >>>>>>
> >>>>>> systemctl status pki-tomcatd(a)pki-tomcat.service:
> >>>>>>
> >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: WARNING:
Exception
> >>>>>> processing realm
com.netscape.cms.tomcat.ProxyRealm@2bfea12f
> >>>>>> background process
> >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]:
> >>>>>> javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable
> >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> >>>>>>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> >>>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> >>>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> >>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> >>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> >>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> >>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> >>>>>> java.lang.Thread.run(Thread.java:748)
> >>>>>
> >>>>> The logs will contain much more useful information. dogtag
keeps
> >>>>> changing the location of the logs and I forget exactly where it
is in
> >>>>> your version but it's somewhere in
/var/log/pki*/pki*/ca/...
> >>>>>
> >>>>> The log may be named debug or debug-<date>
> >>>>>
> >>>>> Also look at the selftest log in the same directory.
> >>>>>
> >>>>> There are a LOT of red herrings in the dogtag logs so proceed
with caution.
> >>>>>
> >>>>> You do not need to touch or create anything for this logging to
take
> >>>>> place. You should delete the directory you created.
> >>>>>
> >>>>> rob
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud
<flo(a)redhat.com> wrote:
> >>>>>>>
> >>>>>>> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users
wrote:
> >>>>>>>> Hi!
> >>>>>>>>
> >>>>>>>> I tried to follow this solution for cert renewal for
RHEL6:
> >>>>>>>>
https://access.redhat.com/solutions/643753 (Sorry,
desperation is
> >>>>>>>> setting in), but when I attempted Step 2, I got:
> >>>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> 1. this note was written for RHEL 6 but you said in your
first e-mail
> >>>>>>> that your server is running CentOS 7 with ipa 4.5.4.
Please don't follow
> >>>>>>> those instructions as they are not adapted to your
deployment.
> >>>>>>> The instructions for RHEL 7 are available at
> >>>>>>>
https://access.redhat.com/solutions/3357261.
> >>>>>>>
> >>>>>>> 2. In a previous e-mail, the output of getcert list |
grep -i expires
> >>>>>>> did not show any expired certificates, so I would not
rush into wrong
> >>>>>>> conclusions. We need to understand first why pki did not
start.
> >>>>>>>
> >>>>>>> What is the output of:
> >>>>>>> $ ipactl status
> >>>>>>> $ systemctl status pki-tomcatd(a)pki-tomcat.service
> >>>>>>>
> >>>>>>> flo
> >>>>>>>
> >>>>>>>> # for nickname in "auditSigningCert
cert-pki-ca" "ocspSigningCert
> >>>>>>>> cert-pki-ca" "subsystemCert
cert-pki-ca" "Server-Cert cert-pki-ca"; do
> >>>>>>>> echo $nickname; certutil -L -d /var/lib/pki-ca/alias
-n "${nickname}"
> >>>>>>>> | grep -i after; done
> >>>>>>>> auditSigningCert cert-pki-ca
> >>>>>>>> certutil: function failed:
SEC_ERROR_LEGACY_DATABASE: The
> >>>>>>>> certificate/key database is in an old, unsupported
format.
> >>>>>>>> ocspSigningCert cert-pki-ca
> >>>>>>>> certutil: function failed:
SEC_ERROR_LEGACY_DATABASE: The
> >>>>>>>> certificate/key database is in an old, unsupported
format.
> >>>>>>>> subsystemCert cert-pki-ca
> >>>>>>>> certutil: function failed:
SEC_ERROR_LEGACY_DATABASE: The
> >>>>>>>> certificate/key database is in an old, unsupported
format.
> >>>>>>>> Server-Cert cert-pki-ca
> >>>>>>>> certutil: function failed:
SEC_ERROR_LEGACY_DATABASE: The
> >>>>>>>> certificate/key database is in an old, unsupported
format.
> >>>>>>>>
> >>>>>>>> Could this be the root of my problems?
> >>>>>>>> And how can I convert them?
> >>>>>>>>
> >>>>>>>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi
<notify.sina(a)gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Restarting ipa didnt create the logs.
> >>>>>>>>> Please, what else can i do?
> >>>>>>>>>
> >>>>>>>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi
<notify.sina(a)gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi!
> >>>>>>>>>>
> >>>>>>>>>> getcert list | grep -i expires
> >>>>>>>>>> expires: 2019-04-13 12:08:20 UTC
> >>>>>>>>>> expires: 2019-04-13 12:08:06 UTC
> >>>>>>>>>> expires: 2019-04-13 12:07:50 UTC
> >>>>>>>>>> expires: 2035-06-01 08:33:01 UTC
> >>>>>>>>>> expires: 2019-04-13 12:07:41 UTC
> >>>>>>>>>> expires: 2019-04-13 12:06:55 UTC
> >>>>>>>>>> expires: 2019-05-05 12:06:41 UTC
> >>>>>>>>>> expires: 2019-05-05 12:06:56 UTC
> >>>>>>>>>> expires: 2020-01-17 19:56:03 UTC
> >>>>>>>>>>
> >>>>>>>>>> I didnt find a
/var/log/pki/pki-tomcat/ca/debug directory, but I am
> >>>>>>>>>> creating one and running "ipactl
restart".
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob
Crittenden <rcritten(a)redhat.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Sina Owolabi via FreeIPA-users wrote:
> >>>>>>>>>>>> Hi!
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am running a small IPA domain
(CentOS 7 servers, ipa version 4.5.4,
> >>>>>>>>>>>> api version 2.228), with one master,
and two replicas, and I noticed
> >>>>>>>>>>>> that pki-tomcatd no longer works on
the master, after attempting a
> >>>>>>>>>>>> reboot.
> >>>>>>>>>>>> pki-tomcatd works fine on the
slaves.
> >>>>>>>>>>>> I noticed if I try to run IPA
functions (dns record removal, hosts
> >>>>>>>>>>>> management, user passwords, etc), I
receive responses like this:
> >>>>>>>>>>>>
> >>>>>>>>>>>> ipa: ERROR: Certificate operation
cannot be completed: Unable to
> >>>>>>>>>>>> communicate with CMS (Internal
Server Error)
> >>>>>>>>>>>> But on the replicas, functions work
fine.
> >>>>>>>>>>>> Please can someone guide me on how
to fix this?
> >>>>>>>>>>>
> >>>>>>>>>>> The CA log is in
/var/log/pki/pki-tomcat/ca/debug. That may have some
> >>>>>>>>>>> pointers. I'd look at selftests.log
first.
> >>>>>>>>>>>
> >>>>>>>>>>> My guess is that some of the CA
certificates have failed to renew.
> >>>>>>>>>>>
> >>>>>>>>>>> getcert list | grep -i expires
> >>>>>>>>>>>
> >>>>>>>>>>> rob
> >>>>>>>> _______________________________________________
> >>>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
> >>>>>>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>>>> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
> >>>>>>>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>>>>>>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
> > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
> >
>