Would reinstalling IPA software help?
On Wed, Mar 6, 2019 at 10:05 PM Sina Owolabi <notify.sina(a)gmail.com> wrote:
>
> Ah.. the server has SELinux disabled.
> Curious why its not recreating its log directories after a restart.
>
> On Wed, Mar 6, 2019 at 4:33 PM Rob Crittenden <rcritten(a)redhat.com> wrote:
> >
> > Sina Owolabi via FreeIPA-users wrote:
> > > Hi Rob
> > >
> > > Sorry I missed the full question:
> > > What are the contents of /var/log/pki/pki-tomcat/ca ?
> > >
> > > Could it be that the CA can't write its own logs? What does the latest
> > > catalina log show in the parent directory?
> > >
> > > /var/log/pki/pki-tomcat/ca was empty until I created
> > > /var/log/pki/pki-tomcat/ca/logs and
> > > /var/log/pki/pki-tomcat/ca/debug directories.
> > > I dont think the ca would have trouble writing its logs, the structure
> > > is all owned by pkiuser:
> > > drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca
> > >
> > > Now that I think about it, I do remember some issues with runaway logs
> > > filling up /var/log, and
> > > I deleted some directories, and recreated them, but I dont think
> > > pki-tomcat suffered then.
> >
> > Hard to know. If the process was already running at the time things may
> > have appeared ok until it was restarted.
> >
> > debug is a log file, not a directory.
> >
> > My 4.4.4 install contains the following in /var/log/pki:
> >
> > drwxr-xr-x. 3 root root 21 Mar 30 2017 ./server
> > drwxrwx---. 3 pkiuser pkiuser 12288 Mar 6 01:22 ./pki-tomcat
> > drwxrwx---. 4 pkiuser pkiuser 4096 Feb 7 11:27 ./pki-tomcat/ca
> > drwxrwx---. 2 pkiuser pkiuser 86 Dec 4 11:00 ./pki-tomcat/ca/archive
> > drwxrwx---. 2 pkiuser pkiuser 84 Feb 7 11:27 ./pki-tomcat/ca/signedAudit
> >
> > Be sure to run restorecon -R on /var/log/pki to ensure the SELinux
> > contexts are correct.
> >
> > rob
> >
> > >
> > > On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi <notify.sina(a)gmail.com>
wrote:
> > >>
> > >> Hi Rob
> > >>
> > >> Today's catalina log file writes:
> > >>
> > >> WARNING: Exception processing realm
> > >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> > >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > >> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > >> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > >> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > >> at java.lang.Thread.run(Thread.java:748)
> > >>
> > >> Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase
> > >> backgroundProcess
> > >> WARNING: Exception processing realm
> > >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> > >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > >> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > >> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > >> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > >> at java.lang.Thread.run(Thread.java:748)
> > >>
> > >> Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase
> > >> backgroundProcess
> > >> WARNING: Exception processing realm
> > >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> > >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > >> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > >> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > >> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > >> at java.lang.Thread.run(Thread.java:748)
> > >>
> > >> Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase
> > >> backgroundProcess
> > >> WARNING: Exception processing realm
> > >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> > >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > >> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > >> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > >> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > >> at java.lang.Thread.run(Thread.java:748)
> > >>
> > >> On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden
<rcritten(a)redhat.com> wrote:
> > >>>
> > >>> Sina Owolabi wrote:
> > >>>> Log directories on the server:
> > >>>>
> > >>>> /var/log/pki/pki-tomcat/ca/debug
> > >>>> /var/log/pki/pki-tomcat/ca/logs
> > >>>> /var/log/pki/server/upgrade/10.1.2
> > >>>> /var/log/pki/server/upgrade/10.1.99
> > >>>> /var/log/pki/server/upgrade/10.2.1
> > >>>> /var/log/pki/server/upgrade/10.2.2
> > >>>> /var/log/pki/server/upgrade/10.2.3
> > >>>> /var/log/pki/server/upgrade/10.2.4
> > >>>> /var/log/pki/server/upgrade/10.2.5
> > >>>> /var/log/pki/server/upgrade/10.2.6
> > >>>> /var/log/pki/server/upgrade/10.3.0
> > >>>> /var/log/pki/server/upgrade/10.3.3
> > >>>> /var/log/pki/server/upgrade/10.4.0
> > >>>> /var/log/pki/server/upgrade/10.4.1
> > >>>> /var/log/pki/server/upgrade/10.5.1
> > >>>>
> > >>>> /var/log/pki/pki-tomcat/ca/debug
> > >>>
> > >>> You stated you had created this directory yourself.
> > >>>
> > >>> What are the contents of /var/log/pki/pki-tomcat/ca ?
> > >>>
> > >>> Could it be that the CA can't write its own logs? What does the
latest
> > >>> catalina log show in the parent directory?
> > >>>
> > >>> rob
> > >>>
> > >>>> /var/log/pki/pki-tomcat/ca/logs
> > >>>> are both empty.
> > >>>>
> > >>>> On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden
<rcritten(a)redhat.com> wrote:
> > >>>>>
> > >>>>> Sina Owolabi wrote:
> > >>>>>> Hi Florence
> > >>>>>>
> > >>>>>> and thanks for the help.
> > >>>>>> ipactl status:
> > >>>>>> [root@services ~]# ipactl status
--ignore-service-failure; cat
> > >>>>>> Directory Service: RUNNING
> > >>>>>> krb5kdc Service: RUNNING
> > >>>>>> kadmin Service: RUNNING
> > >>>>>> named Service: RUNNING
> > >>>>>> httpd Service: RUNNING
> > >>>>>> ipa-custodia Service: RUNNING
> > >>>>>> ntpd Service: RUNNING
> > >>>>>> pki-tomcatd Service: STOPPED
> > >>>>>> ipa-otpd Service: RUNNING
> > >>>>>> ipa-dnskeysyncd Service: RUNNING
> > >>>>>> ipa: INFO: The ipactl command was successful
> > >>>>>>
> > >>>>>>
> > >>>>>> systemctl status -l pki-tomcatd(a)pki-tomcat.service;
cat
> > >>>>>> ? pki-tomcatd(a)pki-tomcat.service - PKI Tomcat Server
pki-tomcat
> > >>>>>> Loaded: loaded
(/lib/systemd/system/pki-tomcatd@.service; enabled;
> > >>>>>> vendor preset: disabled)
> > >>>>>> Active: active (running) since Tue 2019-03-05
09:14:15 WAT; 26min ago
> > >>>>>> Process: 1233 ExecStartPre=/usr/bin/pkidaemon start
%i (code=exited,
> > >>>>>> status=0/SUCCESS)
> > >>>>>> Main PID: 1376 (java)
> > >>>>>> CGroup:
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd(a)pki-tomcat.service
> > >>>>>> └─1376
/usr/lib/jvm/jre-1.8.0-openjdk/bin/java
> > >>>>>> -DRESTEASY_LIB=/usr/share/java/resteasy-base
-classpath
> > >>>>>>
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
> > >>>>>> -Dcatalina.base=/var/lib/pki/pki-tomcat
> > >>>>>> -Dcatalina.home=/usr/share/tomcat
-Djava.endorsed.dirs=
> > >>>>>> -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
> > >>>>>>
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties
> > >>>>>>
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> > >>>>>> -Djava.security.manager
> > >>>>>>
-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
> > >>>>>> org.apache.catalina.startup.Bootstrap start
> > >>>>>>
> > >>>>>> systemctl status pki-tomcatd(a)pki-tomcat.service:
> > >>>>>>
> > >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]:
WARNING: Exception
> > >>>>>> processing realm
com.netscape.cms.tomcat.ProxyRealm@2bfea12f
> > >>>>>> background process
> > >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]:
> > >>>>>> javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable
> > >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>>>>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > >>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
> > >>>>>> java.lang.Thread.run(Thread.java:748)
> > >>>>>
> > >>>>> The logs will contain much more useful information. dogtag
keeps
> > >>>>> changing the location of the logs and I forget exactly
where it is in
> > >>>>> your version but it's somewhere in
/var/log/pki*/pki*/ca/...
> > >>>>>
> > >>>>> The log may be named debug or debug-<date>
> > >>>>>
> > >>>>> Also look at the selftest log in the same directory.
> > >>>>>
> > >>>>> There are a LOT of red herrings in the dogtag logs so
proceed with caution.
> > >>>>>
> > >>>>> You do not need to touch or create anything for this
logging to take
> > >>>>> place. You should delete the directory you created.
> > >>>>>
> > >>>>> rob
> > >>>>>
> > >>>>>
> > >>>>>>
> > >>>>>> On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud
<flo(a)redhat.com> wrote:
> > >>>>>>>
> > >>>>>>> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users
wrote:
> > >>>>>>>> Hi!
> > >>>>>>>>
> > >>>>>>>> I tried to follow this solution for cert
renewal for RHEL6:
> > >>>>>>>>
https://access.redhat.com/solutions/643753
(Sorry, desperation is
> > >>>>>>>> setting in), but when I attempted Step 2, I
got:
> > >>>>>>>>
> > >>>>>>> Hi,
> > >>>>>>>
> > >>>>>>> 1. this note was written for RHEL 6 but you said in
your first e-mail
> > >>>>>>> that your server is running CentOS 7 with ipa
4.5.4. Please don't follow
> > >>>>>>> those instructions as they are not adapted to your
deployment.
> > >>>>>>> The instructions for RHEL 7 are available at
> > >>>>>>>
https://access.redhat.com/solutions/3357261.
> > >>>>>>>
> > >>>>>>> 2. In a previous e-mail, the output of getcert list
| grep -i expires
> > >>>>>>> did not show any expired certificates, so I would
not rush into wrong
> > >>>>>>> conclusions. We need to understand first why pki
did not start.
> > >>>>>>>
> > >>>>>>> What is the output of:
> > >>>>>>> $ ipactl status
> > >>>>>>> $ systemctl status pki-tomcatd(a)pki-tomcat.service
> > >>>>>>>
> > >>>>>>> flo
> > >>>>>>>
> > >>>>>>>> # for nickname in "auditSigningCert
cert-pki-ca" "ocspSigningCert
> > >>>>>>>> cert-pki-ca" "subsystemCert
cert-pki-ca" "Server-Cert cert-pki-ca"; do
> > >>>>>>>> echo $nickname; certutil -L -d
/var/lib/pki-ca/alias -n "${nickname}"
> > >>>>>>>> | grep -i after; done
> > >>>>>>>> auditSigningCert cert-pki-ca
> > >>>>>>>> certutil: function failed:
SEC_ERROR_LEGACY_DATABASE: The
> > >>>>>>>> certificate/key database is in an old,
unsupported format.
> > >>>>>>>> ocspSigningCert cert-pki-ca
> > >>>>>>>> certutil: function failed:
SEC_ERROR_LEGACY_DATABASE: The
> > >>>>>>>> certificate/key database is in an old,
unsupported format.
> > >>>>>>>> subsystemCert cert-pki-ca
> > >>>>>>>> certutil: function failed:
SEC_ERROR_LEGACY_DATABASE: The
> > >>>>>>>> certificate/key database is in an old,
unsupported format.
> > >>>>>>>> Server-Cert cert-pki-ca
> > >>>>>>>> certutil: function failed:
SEC_ERROR_LEGACY_DATABASE: The
> > >>>>>>>> certificate/key database is in an old,
unsupported format.
> > >>>>>>>>
> > >>>>>>>> Could this be the root of my problems?
> > >>>>>>>> And how can I convert them?
> > >>>>>>>>
> > >>>>>>>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi
<notify.sina(a)gmail.com> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Restarting ipa didnt create the logs.
> > >>>>>>>>> Please, what else can i do?
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi
<notify.sina(a)gmail.com> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Hi!
> > >>>>>>>>>>
> > >>>>>>>>>> getcert list | grep -i expires
> > >>>>>>>>>> expires: 2019-04-13 12:08:20
UTC
> > >>>>>>>>>> expires: 2019-04-13 12:08:06
UTC
> > >>>>>>>>>> expires: 2019-04-13 12:07:50
UTC
> > >>>>>>>>>> expires: 2035-06-01 08:33:01
UTC
> > >>>>>>>>>> expires: 2019-04-13 12:07:41
UTC
> > >>>>>>>>>> expires: 2019-04-13 12:06:55
UTC
> > >>>>>>>>>> expires: 2019-05-05 12:06:41
UTC
> > >>>>>>>>>> expires: 2019-05-05 12:06:56
UTC
> > >>>>>>>>>> expires: 2020-01-17 19:56:03
UTC
> > >>>>>>>>>>
> > >>>>>>>>>> I didnt find a
/var/log/pki/pki-tomcat/ca/debug directory, but I am
> > >>>>>>>>>> creating one and running "ipactl
restart".
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob
Crittenden <rcritten(a)redhat.com> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Sina Owolabi via FreeIPA-users
wrote:
> > >>>>>>>>>>>> Hi!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I am running a small IPA domain
(CentOS 7 servers, ipa version 4.5.4,
> > >>>>>>>>>>>> api version 2.228), with one
master, and two replicas, and I noticed
> > >>>>>>>>>>>> that pki-tomcatd no longer
works on the master, after attempting a
> > >>>>>>>>>>>> reboot.
> > >>>>>>>>>>>> pki-tomcatd works fine on the
slaves.
> > >>>>>>>>>>>> I noticed if I try to run IPA
functions (dns record removal, hosts
> > >>>>>>>>>>>> management, user passwords,
etc), I receive responses like this:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ipa: ERROR: Certificate
operation cannot be completed: Unable to
> > >>>>>>>>>>>> communicate with CMS (Internal
Server Error)
> > >>>>>>>>>>>> But on the replicas, functions
work fine.
> > >>>>>>>>>>>> Please can someone guide me on
how to fix this?
> > >>>>>>>>>>>
> > >>>>>>>>>>> The CA log is in
/var/log/pki/pki-tomcat/ca/debug. That may have some
> > >>>>>>>>>>> pointers. I'd look at
selftests.log first.
> > >>>>>>>>>>>
> > >>>>>>>>>>> My guess is that some of the CA
certificates have failed to renew.
> > >>>>>>>>>>>
> > >>>>>>>>>>> getcert list | grep -i expires
> > >>>>>>>>>>>
> > >>>>>>>>>>> rob
> > >>>>>>>>
_______________________________________________
> > >>>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
> > >>>>>>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
> > >>>>>>>> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
> > >>>>>>>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > >>>>>>>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>
> > > _______________________________________________
> > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> > > Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
> > > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
> > >
> >