12:58 p.m.
If the subject isn't vague enough, perhaps I can explain in some better detail.
I have IPA setup with a couple of replicas and it's been running fine for a few
months; perriodic runs of ipa-healthcheck didn't show any issues.
During an update of the system packages, something went wrong and the CA is now
unavailable from the webui and ipa tools.
When accessing from the webui (Authentication->Certificates) I get th following error:
"Certificate operation cannot be completed: Unable to communicate with CMS
(403)"
From some cli tools (ipa cert-show) I get this error:
"ipa: ERROR: Certificate operation cannot be completed: Request failed with status
403: Non-2xx response from CA REST API: 403. (403)"
It appears that dogtag is working, as I can use it's toolset to query "pki
ca-cert-show 0x1 --pretty" for example returns the certificate as expected so I
suspect the issue lies somewhere in the API.
I compared the cert which I believe to be used for pki with the following:
"grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 >
/tmp/pwdfile.txt"
"certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert
cert-pki-ca'"
So from what I've read it seems pki-tomcat should be able to access it's private
key and certificate.
I also compared the cert with the one in LDAP and the two are the same.
"ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca userCertificate"
In digging through some old backups, I found that entries in
/etc/pki/pki-tomcat/server.xml had changed, so I reset the "requiredSecret" back
to an earlier value, which seemed to allow the webui to work again, at least partially as
I can now get a listing from the CA in the webui though I cannot see details of any
particular certificate.
Assuming I'm on the right path, is there a documented process to regenerate the
server.xml file or am I chasing the wrong problem?
1:09 p.m.
D Trom via FreeIPA-users wrote:
If the subject isn't vague enough, perhaps I can explain in some
better detail.
I have IPA setup with a couple of replicas and it's been running fine for a few
months; perriodic runs of ipa-healthcheck didn't show any issues.
During an update of the system packages, something went wrong and the CA is now
unavailable from the webui and ipa tools.
When accessing from the webui (Authentication->Certificates) I get th following
error:
"Certificate operation cannot be completed: Unable to communicate with CMS
(403)"
From some cli tools (ipa cert-show) I get this error:
"ipa: ERROR: Certificate operation cannot be completed: Request failed with status
403: Non-2xx response from CA REST API: 403. (403)"
It appears that dogtag is working, as I can use it's toolset to query "pki
ca-cert-show 0x1 --pretty" for example returns the certificate as expected so I
suspect the issue lies somewhere in the API.
I compared the cert which I believe to be used for pki with the following:
"grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 >
/tmp/pwdfile.txt"
"certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert
cert-pki-ca'"
So from what I've read it seems pki-tomcat should be able to access it's private
key and certificate.
I also compared the cert with the one in LDAP and the two are the same.
"ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca userCertificate"
In digging through some old backups, I found that entries in
/etc/pki/pki-tomcat/server.xml had changed, so I reset the "requiredSecret" back
to an earlier value, which seemed to allow the webui to work again, at least partially as
I can now get a listing from the CA in the webui though I cannot see details of any
particular certificate.
Assuming I'm on the right path, is there a documented process to regenerate the
server.xml file or am I chasing the wrong problem?
First, it was a most excellent idea to use the pki command to verify
that the CA was alive and well!
The value of requiredSecret needs to match the value in
/etc/httpd/conf.d/ipa-pki-proxy.conf in the ProxyPassMatch statements
(the last argument)
The argument name in server.xml is determined by the version of tomcat.
pre-9.0.31.0 uses 'requiredSecret' and afterward uses 'secret'.
There is no documented process on regenerating server.xml that I know
of. But since you already did the detective work to figure out it is
requiredSecret related look at your version of tomcat vs what was added
in the IPA upgrader (run in rpm %post).
Note that any manual changes you make may be overwritten on the next
package update.
rob
1:54 p.m.
Thank you for the hint, it's gotten me farther. I can now see cert details in the
webui; however, cli tools still fail with
"ipa: ERROR: Certificate operation cannot be completed: Request failed with status
403: Non-2xx response from CA REST API: 403. (403)"
Specifically, "ipa cert show 4" (where 4 is a valid certificate serial number)
Here's the output of "ipa-healthcheck". Of note, valid.tld is sanitized, it
really is valid and not literally "valid.tld". The replica server4.valid.tld is
a failed server which has been removed and does not show in the output of
"ipa-replica-manage list"
"ipa topologysuffix-verify [domain|ca]"
"ipa topologysegment-find [domain|ca]"
# ipa-healthcheck
Internal server error HTTPSConnectionPool(host='server4.valid.tld', port=443): Max
retries exceeded with url: /ca/rest/certs/search?size=3 (Caused by
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
0x7f8ac490a8d0>: Failed to establish a new connection: [Errno -2] Name or service not
known',))
ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API:
403. (403)
ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API:
403. (403)
ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API:
403. (403)
ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API:
403. (403)
ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API:
403. (403)
ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API:
403. (403)
ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API:
403. (403)
ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API:
403. (403)
ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API:
403. (403)
ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API:
403. (403)
[
{
"source": "pki.server.healthcheck.clones.connectivity_and_data",
"check": "ClonesConnectivyAndDataCheck",
"result": "ERROR",
"uuid": "d6d3a36d-f2fd-4793-971f-9bacadfe5881",
"when": "20210910184505Z",
"duration": "1.538118",
"kw": {
"status": "ERROR: pki-tomcat : Internal error testing CA clone.
Host: server4.valid.tld Port: 443"
}
},
{
"source": "ipahealthcheck.dogtag.ca",
"check": "DogtagCertsConnectivityCheck",
"result": "ERROR",
"uuid": "fa1ac443-9ce2-457a-a814-2b127eff8541",
"when": "20210910184507Z",
"duration": "0.246410",
"kw": {
"msg": "Request for certificate failed, Certificate operation cannot
be completed: Request failed with status 403: Non-2xx response from CA REST API: 403.
(403)"
}
},
{
"source": "ipahealthcheck.ds.replication",
"check": "ReplicationCheck",
"result": "ERROR",
"uuid": "2ecf8b7f-78c7-4527-9d0b-716b1ba8061b",
"when": "20210910184508Z",
"duration": "0.742027",
"kw": {
"key": "DSREPLLE0003",
"items": [
"Replication",
"Agreement"
],
"msg": "The replication agreement (catoserver2.valid.tld) under
\"o=ipaca\" is not in synchronization.\nStatus message: error (18) can't
acquire replica (incremental update transient warning. backing off, will retry update
later.)"
}
},
{
"source": "ipahealthcheck.ds.replication",
"check": "ReplicationCheck",
"result": "ERROR",
"uuid": "498d7a58-68d4-44ad-966a-0d8e918df33c",
"when": "20210910184508Z",
"duration": "0.742055",
"kw": {
"key": "DSREPLLE0003",
"items": [
"Replication",
"Agreement"
],
"msg": "The replication agreement (catoserver3.valid.tld) under
\"o=ipaca\" is not in synchronization.\nStatus message: error (18) can't
acquire replica (incremental update transient warning. backing off, will retry update
later.)"
}
},
{
"source": "ipahealthcheck.ipa.certs",
"check": "IPACertRevocation",
"result": "ERROR",
"uuid": "519e1eb9-8229-4695-9f86-2c3d834543d1",
"when": "20210910184514Z",
"duration": "0.424361",
"kw": {
"key": "20210303190407",
"serial": 7,
"error": "Certificate operation cannot be completed: Request failed
with status 403: Non-2xx response from CA REST API: 403. (403)",
"msg": "Request for certificate serial number {serial} in request
{key} failed: {error}"
}
},
{
"source": "ipahealthcheck.ipa.certs",
"check": "IPACertRevocation",
"result": "ERROR",
"uuid": "7f3dd497-2125-4f64-bff3-52cd65291d9c",
"when": "20210910184514Z",
"duration": "0.528265",
"kw": {
"key": "20210303190402",
"serial": 5,
"error": "Certificate operation cannot be completed: Request failed
with status 403: Non-2xx response from CA REST API: 403. (403)",
"msg": "Request for certificate serial number {serial} in request
{key} failed: {error}"
}
},
{
"source": "ipahealthcheck.ipa.certs",
"check": "IPACertRevocation",
"result": "ERROR",
"uuid": "b242bb04-7a86-446b-b2c6-3c1c65994a21",
"when": "20210910184514Z",
"duration": "0.630944",
"kw": {
"key": "20210303190403",
"serial": 2,
"error": "Certificate operation cannot be completed: Request failed
with status 403: Non-2xx response from CA REST API: 403. (403)",
"msg": "Request for certificate serial number {serial} in request
{key} failed: {error}"
}
},
{
"source": "ipahealthcheck.ipa.certs",
"check": "IPACertRevocation",
"result": "ERROR",
"uuid": "5b6aad97-4a48-477c-bf45-503b6a2df426",
"when": "20210910184515Z",
"duration": "0.735810",
"kw": {
"key": "20210303190404",
"serial": 4,
"error": "Certificate operation cannot be completed: Request failed
with status 403: Non-2xx response from CA REST API: 403. (403)",
"msg": "Request for certificate serial number {serial} in request
{key} failed: {error}"
}
},
{
"source": "ipahealthcheck.ipa.certs",
"check": "IPACertRevocation",
"result": "ERROR",
"uuid": "4c68d780-aaab-4d28-8920-e0396433b969",
"when": "20210910184515Z",
"duration": "0.838743",
"kw": {
"key": "20210303190405",
"serial": 1,
"error": "Certificate operation cannot be completed: Request failed
with status 403: Non-2xx response from CA REST API: 403. (403)",
"msg": "Request for certificate serial number {serial} in request
{key} failed: {error}"
}
},
{
"source": "ipahealthcheck.ipa.certs",
"check": "IPACertRevocation",
"result": "ERROR",
"uuid": "8e8e7e65-3081-47b1-b3fd-d35ee444b7a6",
"when": "20210910184515Z",
"duration": "0.939950",
"kw": {
"key": "20210303190406",
"serial": 3,
"error": "Certificate operation cannot be completed: Request failed
with status 403: Non-2xx response from CA REST API: 403. (403)",
"msg": "Request for certificate serial number {serial} in request
{key} failed: {error}"
}
},
{
"source": "ipahealthcheck.ipa.certs",
"check": "IPACertRevocation",
"result": "ERROR",
"uuid": "e22c4c88-92dd-4326-ae54-9ce626348e5f",
"when": "20210910184515Z",
"duration": "0.992323",
"kw": {
"key": "20210303190409",
"serial": 58,
"error": "Certificate operation cannot be completed: Request failed
with status 403: Non-2xx response from CA REST API: 403. (403)",
"msg": "Request for certificate serial number {serial} in request
{key} failed: {error}"
}
},
{
"source": "ipahealthcheck.ipa.certs",
"check": "IPACertRevocation",
"result": "ERROR",
"uuid": "c885ae6c-4365-47ea-905c-e09429aa6f21",
"when": "20210910184515Z",
"duration": "1.091397",
"kw": {
"key": "20210303190408",
"serial": 8,
"error": "Certificate operation cannot be completed: Request failed
with status 403: Non-2xx response from CA REST API: 403. (403)",
"msg": "Request for certificate serial number {serial} in request
{key} failed: {error}"
}
},
{
"source": "ipahealthcheck.ipa.certs",
"check": "IPACertRevocation",
"result": "ERROR",
"uuid": "3c788561-f1a5-4d3e-8ad6-312fc4b335f3",
"when": "20210910184515Z",
"duration": "1.144757",
"kw": {
"key": "20201102193636",
"serial": 10,
"error": "Certificate operation cannot be completed: Request failed
with status 403: Non-2xx response from CA REST API: 403. (403)",
"msg": "Request for certificate serial number {serial} in request
{key} failed: {error}"
}
}
]
4:14 a.m.
I ran into similar issues after upgrading from FreeIPA 4.9.3 to 4.9.6 on Centos Stream 8
last week.
You could check /var/log/httpd/error_log - I had trouble with TLS 1.3 (leading to error
"Request failed with status 403: Non-2xx response from CA REST API: 403.") which
could be solved by disabling SSLProtocol in /etc/httpd/conf.d/ssl.conf.
There were other issues with disabling TLS 1.3 (see
https://bugzilla.redhat.com/show_bug.cgi?id=1775158) and due to lack of time, finally I
recovered from backup.
1:15 p.m.
Thank you. Setting requiredSecret to the same value as secret in
/etc/pki/pki-tomcat/server.xml fixed it for me on CentOS Stream 8. It stopped working
after upgrading FreeIPA from 4.9.3 to 4.9.6.
Seems I barely missed the version that uses "secret":
java -cp catalina.jar org.apache.catalina.util.ServerInfo
Server version: Apache Tomcat/9.0.30
Server built: Jun 29 2021 22:00:15 UTC
Server number: 9.0.30.0
OS Name: Linux
OS Version: 4.18.0-338.el8.x86_64
Architecture: amd64
JVM Version: 1.8.0_302-b08
JVM Vendor: Red Hat, Inc.
Now to figure out why sudo 1.9.8 segfaults when FreeIPA sudo rules are enabled...
952
days inactive
958
days old
freeipa-users@lists.fedorahosted.org
4 comments
4 participants
participants (4)
-
D Trom -
Dirk Silkenbaeumer -
pp -
Rob Crittenden