Failed Upgrade?

Sunday, 30 July 2017

I had an unexpected restart of an IPA server that had apparently had
updates run but had not been restarted.  ipactl says pki-tomcatd would
not start.

Strangely, the actual service appears to be running:

[root@seattlenfs slapd-BPT-ROCKS]# systemctl status
pki-tomcatd(a)pki-tomcat.service
● pki-tomcatd(a)pki-tomcat.service - PKI Tomcat Server pki-tomcat
   Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled;
vendor preset: disabled)
   Active: active (running) since Fri 2017-07-28 11:03:34 PDT; 36min ago
  Process: 14289 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited,
status=0/SUCCESS)
 Main PID: 14406 (java)
   CGroup:
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd(a)pki-tomcat.service
           └─14406 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
-DRESTEASY_LIB=/usr/share/java/resteasy-base
-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/...

Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: Jul 28, 2017
11:39:50 AM org.apache.catalina.core.ContainerBase backgroundProcess
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: WARNING: Exception
processing realm com.netscape.cms.tomcat.ProxyRealm@67cf2df background
process
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]:
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
java.lang.Thread.run(Thread.java:748)

However, the /var/log/ipaupgrade.log is full of trouble.  It ends with:

2017-07-28T17:01:19Z DEBUG The CA status is: check interrupted due to
error: Retrieving CA status failed with status 500
2017-07-28T17:01:19Z DEBUG Waiting for CA to start...
2017-07-28T17:01:20Z DEBUG request POST
http://seattlenfs.bpt.rocks:8080/ca/admin/ca/getStatus
2017-07-28T17:01:20Z DEBUG request body ''
2017-07-28T17:01:20Z DEBUG response status 500
2017-07-28T17:01:20Z DEBUG response headers {'content-length': '2208',
'content-language': 'en', 'server': 'Apache-Coyote/1.1',
'connection':
'close', 'date': 'Fri, 28 Jul 2017 17:01:20 GMT',
'content-type':
'text/html;charset=utf-8'}
2017-07-28T17:01:20Z DEBUG response body
'<html><head><title>Apache
Tomcat/7.0.69 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem
unavailable</h1><HR
size="1" noshade="noshade"><p><b>type</b>
Exception
report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server
encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.69 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
Tomcat/7.0.69</h3></body></html>'
2017-07-28T17:01:20Z DEBUG The CA status is: check interrupted due to
error: Retrieving CA status failed with status 500
2017-07-28T17:01:20Z DEBUG Waiting for CA to start...
2017-07-28T17:01:21Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2017-07-28T17:01:21Z DEBUG   File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in
execute
    return_value = self.run()
  File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 48, in run
    raise admintool.ScriptError(str(e))

2017-07-28T17:01:21Z DEBUG The ipa-server-upgrade command failed,
exception: ScriptError: CA did not start in 300.0s
2017-07-28T17:01:21Z ERROR CA did not start in 300.0s
2017-07-28T17:01:21Z ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information

Should I just blindly run ipa-server-upgrade again?

Googling had me look at certificate expirations, they seem to be good.

[root@seattlenfs slapd-BPT-ROCKS]# getcert list | grep expires
	expires: 2019-05-29 05:54:06 UTC
	expires: 2019-05-29 05:53:57 UTC
	expires: 2019-05-29 05:53:16 UTC
	expires: 2035-07-16 12:51:42 UTC
	expires: 2019-05-29 05:53:37 UTC
	expires: 2018-08-15 05:20:24 UTC
	expires: 2018-08-26 05:01:42 UTC
	expires: 2018-08-26 05:01:43 UTC

[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep ipa-
ipa-admintools.noarch                    4.4.0-14.el7.centos.7
@test-centos7-updates
ipa-client.x86_64                        4.4.0-14.el7.centos.7
@test-centos7-updates
ipa-client-common.noarch                 4.4.0-14.el7.centos.7
@test-centos7-updates
ipa-common.noarch                        4.4.0-14.el7.centos.7
@test-centos7-updates
ipa-python-compat.noarch                 4.4.0-14.el7.centos.7
@test-centos7-updates
ipa-server.x86_64                        4.4.0-14.el7.centos.7
@test-centos7-updates
ipa-server-common.noarch                 4.4.0-14.el7.centos.7
@test-centos7-updates
ipa-server-dns.noarch                    4.4.0-14.el7.centos.7
@test-centos7-updates

[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep pki-
pki-base.noarch                          10.3.3-19.el7_3
@updates
pki-base-java.noarch                     10.3.3-19.el7_3
@updates
pki-ca.noarch                            10.3.3-19.el7_3
@updates
pki-kra.noarch                           10.3.3-19.el7_3
@updates
pki-server.noarch                        10.3.3-19.el7_3
@updates
pki-tools.x86_64                         10.3.3-19.el7_3
@updates

[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep tomcat
tomcat.noarch                            7.0.69-12.el7_3
@updates
tomcat-el-2.2-api.noarch                 7.0.69-12.el7_3
@updates
tomcat-jsp-2.2-api.noarch                7.0.69-12.el7_3
@updates
tomcat-lib.noarch                        7.0.69-12.el7_3
@updates
tomcat-servlet-3.0-api.noarch            7.0.69-12.el7_3
@updates
tomcatjss.noarch                         7.1.2-3.el7
@base

[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep java
java-1.7.0-openjdk.x86_64                1:1.7.0.141-2.6.10.1.el7_3
@test-centos7-updates
java-1.7.0-openjdk-devel.x86_64          1:1.7.0.141-2.6.10.1.el7_3
@test-centos7-updates
java-1.7.0-openjdk-headless.x86_64       1:1.7.0.141-2.6.10.1.el7_3
@test-centos7-updates
java-1.8.0-openjdk.x86_64                1:1.8.0.141-1.b16.el7_3
@updates
java-1.8.0-openjdk-headless.x86_64       1:1.8.0.141-1.b16.el7_3
@updates
javamail.noarch                          1.4.6-8.el7
@base
javapackages-tools.noarch                3.4.1-11.el7
@base
javassist.noarch                         3.16.1-10.el7
@base
nuxwdog-client-java.x86_64               1.0.3-5.el7
@base
pki-base-java.noarch                     10.3.3-19.el7_3
@updates
python-javapackages.noarch               3.4.1-11.el7
@base
tzdata-java.noarch                       2017a-1.el7
@test-centos7-updates

Any other useful information I can provide?

-- 
Ian Harding
IT Director
Brown Paper Tickets
1-800-838-3006 ext 7186
http://www.brownpapertickets.com

2024

2023

2022

2021

2020

2019

2018

2017