Hi Rob
Sorry I missed the full question:
What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest
catalina log show in the parent directory?
/var/log/pki/pki-tomcat/ca was empty until I created
/var/log/pki/pki-tomcat/ca/logs and
/var/log/pki/pki-tomcat/ca/debug directories.
I dont think the ca would have trouble writing its logs, the structure
is all owned by pkiuser:
drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca
Now that I think about it, I do remember some issues with runaway logs
filling up /var/log, and
I deleted some directories, and recreated them, but I dont think
pki-tomcat suffered then.
Hard to know. If the process was already running at the time things may
have appeared ok until it was restarted.
debug is a log file, not a directory.
My 4.4.4 install contains the following in /var/log/pki:
drwxr-xr-x. 3 root root 21 Mar 30 2017 ./server
drwxrwx---. 3 pkiuser pkiuser 12288 Mar 6 01:22 ./pki-tomcat
drwxrwx---. 4 pkiuser pkiuser 4096 Feb 7 11:27 ./pki-tomcat/ca
drwxrwx---. 2 pkiuser pkiuser 86 Dec 4 11:00 ./pki-tomcat/ca/archive
drwxrwx---. 2 pkiuser pkiuser 84 Feb 7 11:27 ./pki-tomcat/ca/signedAudit
Be sure to run restorecon -R on /var/log/pki to ensure the SELinux
contexts are correct.
rob
On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi <notify.sina(a)gmail.com> wrote:
>
> Hi Rob
>
> Today's catalina log file writes:
>
> WARNING: Exception processing realm
> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> at java.lang.Thread.run(Thread.java:748)
>
> Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase
> backgroundProcess
> WARNING: Exception processing realm
> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> at java.lang.Thread.run(Thread.java:748)
>
> Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase
> backgroundProcess
> WARNING: Exception processing realm
> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> at java.lang.Thread.run(Thread.java:748)
>
> Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase
> backgroundProcess
> WARNING: Exception processing realm
> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> at java.lang.Thread.run(Thread.java:748)
>
> On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden <rcritten(a)redhat.com> wrote:
>>
>> Sina Owolabi wrote:
>>> Log directories on the server:
>>>
>>> /var/log/pki/pki-tomcat/ca/debug
>>> /var/log/pki/pki-tomcat/ca/logs
>>> /var/log/pki/server/upgrade/10.1.2
>>> /var/log/pki/server/upgrade/10.1.99
>>> /var/log/pki/server/upgrade/10.2.1
>>> /var/log/pki/server/upgrade/10.2.2
>>> /var/log/pki/server/upgrade/10.2.3
>>> /var/log/pki/server/upgrade/10.2.4
>>> /var/log/pki/server/upgrade/10.2.5
>>> /var/log/pki/server/upgrade/10.2.6
>>> /var/log/pki/server/upgrade/10.3.0
>>> /var/log/pki/server/upgrade/10.3.3
>>> /var/log/pki/server/upgrade/10.4.0
>>> /var/log/pki/server/upgrade/10.4.1
>>> /var/log/pki/server/upgrade/10.5.1
>>>
>>> /var/log/pki/pki-tomcat/ca/debug
>>
>> You stated you had created this directory yourself.
>>
>> What are the contents of /var/log/pki/pki-tomcat/ca ?
>>
>> Could it be that the CA can't write its own logs? What does the latest
>> catalina log show in the parent directory?
>>
>> rob
>>
>>> /var/log/pki/pki-tomcat/ca/logs
>>> are both empty.
>>>
>>> On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden <rcritten(a)redhat.com>
wrote:
>>>>
>>>> Sina Owolabi wrote:
>>>>> Hi Florence
>>>>>
>>>>> and thanks for the help.
>>>>> ipactl status:
>>>>> [root@services ~]# ipactl status --ignore-service-failure; cat
>>>>> Directory Service: RUNNING
>>>>> krb5kdc Service: RUNNING
>>>>> kadmin Service: RUNNING
>>>>> named Service: RUNNING
>>>>> httpd Service: RUNNING
>>>>> ipa-custodia Service: RUNNING
>>>>> ntpd Service: RUNNING
>>>>> pki-tomcatd Service: STOPPED
>>>>> ipa-otpd Service: RUNNING
>>>>> ipa-dnskeysyncd Service: RUNNING
>>>>> ipa: INFO: The ipactl command was successful
>>>>>
>>>>>
>>>>> systemctl status -l pki-tomcatd(a)pki-tomcat.service; cat
>>>>> ? pki-tomcatd(a)pki-tomcat.service - PKI Tomcat Server pki-tomcat
>>>>> Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service;
enabled;
>>>>> vendor preset: disabled)
>>>>> Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min
ago
>>>>> Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i
(code=exited,
>>>>> status=0/SUCCESS)
>>>>> Main PID: 1376 (java)
>>>>> CGroup:
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd(a)pki-tomcat.service
>>>>> └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
>>>>> -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath
>>>>>
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
>>>>> -Dcatalina.base=/var/lib/pki/pki-tomcat
>>>>> -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs=
>>>>> -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
>>>>>
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Djava.security.manager
>>>>> -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>
>>>>> systemctl status pki-tomcatd(a)pki-tomcat.service:
>>>>>
>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: WARNING: Exception
>>>>> processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f
>>>>> background process
>>>>> Mar 05 09:40:43
services.qrios.com server[1376]:
>>>>> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
>>>>>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
>>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
>>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>> Mar 05 09:40:43
services.qrios.com server[1376]: at
>>>>> java.lang.Thread.run(Thread.java:748)
>>>>
>>>> The logs will contain much more useful information. dogtag keeps
>>>> changing the location of the logs and I forget exactly where it is in
>>>> your version but it's somewhere in /var/log/pki*/pki*/ca/...
>>>>
>>>> The log may be named debug or debug-<date>
>>>>
>>>> Also look at the selftest log in the same directory.
>>>>
>>>> There are a LOT of red herrings in the dogtag logs so proceed with
caution.
>>>>
>>>> You do not need to touch or create anything for this logging to take
>>>> place. You should delete the directory you created.
>>>>
>>>> rob
>>>>
>>>>
>>>>>
>>>>> On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud
<flo(a)redhat.com> wrote:
>>>>>>
>>>>>> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote:
>>>>>>> Hi!
>>>>>>>
>>>>>>> I tried to follow this solution for cert renewal for RHEL6:
>>>>>>>
https://access.redhat.com/solutions/643753 (Sorry,
desperation is
>>>>>>> setting in), but when I attempted Step 2, I got:
>>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> 1. this note was written for RHEL 6 but you said in your first
e-mail
>>>>>> that your server is running CentOS 7 with ipa 4.5.4. Please
don't follow
>>>>>> those instructions as they are not adapted to your deployment.
>>>>>> The instructions for RHEL 7 are available at
>>>>>>
https://access.redhat.com/solutions/3357261.
>>>>>>
>>>>>> 2. In a previous e-mail, the output of getcert list | grep -i
expires
>>>>>> did not show any expired certificates, so I would not rush into
wrong
>>>>>> conclusions. We need to understand first why pki did not start.
>>>>>>
>>>>>> What is the output of:
>>>>>> $ ipactl status
>>>>>> $ systemctl status pki-tomcatd(a)pki-tomcat.service
>>>>>>
>>>>>> flo
>>>>>>
>>>>>>> # for nickname in "auditSigningCert cert-pki-ca"
"ocspSigningCert
>>>>>>> cert-pki-ca" "subsystemCert cert-pki-ca"
"Server-Cert cert-pki-ca"; do
>>>>>>> echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n
"${nickname}"
>>>>>>> | grep -i after; done
>>>>>>> auditSigningCert cert-pki-ca
>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The
>>>>>>> certificate/key database is in an old, unsupported format.
>>>>>>> ocspSigningCert cert-pki-ca
>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The
>>>>>>> certificate/key database is in an old, unsupported format.
>>>>>>> subsystemCert cert-pki-ca
>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The
>>>>>>> certificate/key database is in an old, unsupported format.
>>>>>>> Server-Cert cert-pki-ca
>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The
>>>>>>> certificate/key database is in an old, unsupported format.
>>>>>>>
>>>>>>> Could this be the root of my problems?
>>>>>>> And how can I convert them?
>>>>>>>
>>>>>>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi
<notify.sina(a)gmail.com> wrote:
>>>>>>>>
>>>>>>>> Restarting ipa didnt create the logs.
>>>>>>>> Please, what else can i do?
>>>>>>>>
>>>>>>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi
<notify.sina(a)gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> getcert list | grep -i expires
>>>>>>>>> expires: 2019-04-13 12:08:20 UTC
>>>>>>>>> expires: 2019-04-13 12:08:06 UTC
>>>>>>>>> expires: 2019-04-13 12:07:50 UTC
>>>>>>>>> expires: 2035-06-01 08:33:01 UTC
>>>>>>>>> expires: 2019-04-13 12:07:41 UTC
>>>>>>>>> expires: 2019-04-13 12:06:55 UTC
>>>>>>>>> expires: 2019-05-05 12:06:41 UTC
>>>>>>>>> expires: 2019-05-05 12:06:56 UTC
>>>>>>>>> expires: 2020-01-17 19:56:03 UTC
>>>>>>>>>
>>>>>>>>> I didnt find a /var/log/pki/pki-tomcat/ca/debug
directory, but I am
>>>>>>>>> creating one and running "ipactl restart".
>>>>>>>>>
>>>>>>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden
<rcritten(a)redhat.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Sina Owolabi via FreeIPA-users wrote:
>>>>>>>>>>> Hi!
>>>>>>>>>>>
>>>>>>>>>>> I am running a small IPA domain (CentOS 7
servers, ipa version 4.5.4,
>>>>>>>>>>> api version 2.228), with one master, and two
replicas, and I noticed
>>>>>>>>>>> that pki-tomcatd no longer works on the
master, after attempting a
>>>>>>>>>>> reboot.
>>>>>>>>>>> pki-tomcatd works fine on the slaves.
>>>>>>>>>>> I noticed if I try to run IPA functions (dns
record removal, hosts
>>>>>>>>>>> management, user passwords, etc), I receive
responses like this:
>>>>>>>>>>>
>>>>>>>>>>> ipa: ERROR: Certificate operation cannot be
completed: Unable to
>>>>>>>>>>> communicate with CMS (Internal Server Error)
>>>>>>>>>>> But on the replicas, functions work fine.
>>>>>>>>>>> Please can someone guide me on how to fix
this?
>>>>>>>>>>
>>>>>>>>>> The CA log is in
/var/log/pki/pki-tomcat/ca/debug. That may have some
>>>>>>>>>> pointers. I'd look at selftests.log first.
>>>>>>>>>>
>>>>>>>>>> My guess is that some of the CA certificates have
failed to renew.
>>>>>>>>>>
>>>>>>>>>> getcert list | grep -i expires
>>>>>>>>>>
>>>>>>>>>> rob
>>>>>>> _______________________________________________
>>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>>>>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
>>>>>>> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
>>>>>>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>>
>>>>>>
>>>>
>>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...