On Wed, Sep 16, 2020 at 11:29:05AM +1000, Fraser Tweedale via FreeIPA-users wrote:
On Tue, Sep 15, 2020 at 02:20:49PM -0000, Boris Sukhinin via
FreeIPA-users wrote:
> I try to add a new replica to a cluster of 3 freeipa servers. ipa-replica-install
--setup-ca fails with an error:
> [5/28]: configuring certificate server instance
> ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance: Command
'/usr/sbin/pkispawn -s CA -f /tmp/tmp_3KfOZ' returned non-zero exit status 1
> ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the
following files/directories for more information:
> ipaserver.install.dogtaginstance: CRITICAL /var/log/pki/pki-tomcat
> [error] RuntimeError: CA configuration failed.
>
> /var/log/pki/pki-tomcat/ca/debug gives more info:
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: === Processing ocsp_signing cert ===
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: === Processing sslserver cert ===
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]:
SystemConfigService.processKeyPair(sslserver)
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: SystemConfigService: san_server_cert
not found
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: SystemConfigService: loading existing
key pair from NSS database
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: ConfigurationUtils:
loadKeyPair(Server-Cert cert-pki-ca, )
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: SystemConfigService: storing key pair
into CS.cfg
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: ConfigurationUtils:
storeKeyPair(sslserver)
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]:
SystemConfigService.processCert(sslserver)
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: SystemConfigService: checking
sslserver cert in NSS database
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: configCert: caType is local
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: configCert: caType is remote
(revised)
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: ConfigurationUtils: updateConfig() for
certTag sslserver
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: updateConfig() done
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: configCert: remote CA
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: confgCert: tag: sslserver
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: CertRequestPanel: got public key
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: CertRequestPanel: got private key
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: ConfigurationUtils: For this Cloned
CA, always use its Master CA to generate the 'sslserver' certificate to avoid any
changes which may have been made to the X500Name directory string encoding order.
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: ConfigurationUtils: injectSAN: false
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: configRemoteCert: tag: sslserver :
setting profileId to: caInternalAuthServerCert
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: configRemoteCert: tag: sslserver
calculated profileId: caInternalAuthServerCert
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: CertUtil: content: {xmlOutput=[true],
cert_request_type=[pkcs10], profileId=[caInternalAuthServerCert],
cert_request=[...cut...], requestor_name=[CA-srvXXX.local.domain-8443],
sessionID=[6184760106499759096]}
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: ConfigurationUtils: POST
https://srvYYY.local.domain:443/ca/ee/ca/profileSubmit
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: CertUtil: status: 1
> [15/Sep/2020:15:42:06][http-bio-8443-exec-3]: CertUtil: error: Request 19990005 -
Server Internal Error
> java.io.IOException: Request 19990005 - Server Internal Error
> at
com.netscape.cms.servlet.csadmin.CertUtil.createRemoteCert(CertUtil.java:103)
> at
com.netscape.cms.servlet.csadmin.ConfigurationUtils.configRemoteCert(ConfigurationUtils.java:2737)
> at
com.netscape.cms.servlet.csadmin.ConfigurationUtils.configCert(ConfigurationUtils.java:2593)
> at
org.dogtagpki.server.rest.SystemConfigService.processCert(SystemConfigService.java:484)
> at
org.dogtagpki.server.rest.SystemConfigService.processCerts(SystemConfigService.java:303)
> at
org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:166)
> at
org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:101)
> <...long stacktrace...>
>
> LDAP access logs on existing server show that certificate request ID was reused. ADD
queries for request as well as for certificate have failed, but nevertheless the request
was overwritten with a third query:
> [15/Sep/2020:15:42:06.259655845 +0300] conn=2608 op=1804 ADD
dn="cn=19990005,ou=ca,ou=requests,o=ipaca"
> [15/Sep/2020:15:42:06.260525381 +0300] conn=2608 op=1804 RESULT err=68 tag=105
nentries=0 etime=0.0001251547
> [15/Sep/2020:15:42:06.295774388 +0300] conn=2608 op=1805 ADD
dn="cn=536805381,ou=certificateRepository,ou=ca,o=ipaca"
> [15/Sep/2020:15:42:06.296210847 +0300] conn=2608 op=1805 RESULT err=68 tag=105
nentries=0 etime=0.0000755866
> [15/Sep/2020:15:42:06.301651990 +0300] conn=2608 op=1806 MOD
dn="cn=19990005,ou=ca,ou=requests,o=ipaca"
> [15/Sep/2020:15:42:06.315983780 +0300] conn=2608 op=1806 RESULT err=0 tag=103
nentries=0 etime=0.0014827998 csn=5f60b69f000000110000
>
> I can confirm that each server has its own request ID range configured in
/etc/pki/pki-tomcat/ca/CS.cfg:
> 1. dbs.beginRequestNumber=19970001
> dbs.endRequestNumber=19980000
> 2. dbs.beginRequestNumber=19980001
> dbs.endRequestNumber=19990000
> 3. dbs.beginRequestNumber=19990001
> dbs.endRequestNumber=20000000
>
> So my questions are:
>
> 1. How does PKI track request IDs and prevent their reuse?
>
There are two aspects to this. One is replica range management.
You have already checked the replica ranges so that seems OK. More
details about replica range management in this blog post[1], if you
are interested.
[1]
https://frasertweedale.github.io/blog-redhat/posts/2019-07-26-dogtag-repl...
The other aspect is VLV indices. During startup, Dogtag uses a VLV
search to find the next unused identifier in its active range. If
VLV indicies are corrupt or incomplete, it can result in this
behaviour.
> 2. Does overwriting certificate request in LDAP affect certificate
> renewal or have any other negative consequences?
>
It depends on the verison of FreeIPA. Older versions used
serial-based renewal for system certificates. This can cause big
problems if there are conflicts or old requests have been
overwritten. Since freeipa-4.8.1 we perform a "fresh enrolment"
when renewing system certificates, to avoid such problems.
> 3. What's the best way to fix this problem? LDAP backups are
> available and restoring o=ipaca is an option. Domain database
> rollback is not an option though.
>
Next step is to stop Dogtag, rebuild VLV indices, restart Dogtag and
see if the problem went away. I'm not quite sure how to do this.
I'll reply again when I work it out - or maybe someone familiar with
DS can advise.
You can use the following script to reindex the VLV indices:
$ /bin/cp /usr/share/pki/ca/conf/vlvtasks.ldif .
$ sed -i "s/{instanceId}/pki-tomcat/g" vlvtasks.ldif
$ sed -i "s/{database}/ipaca/g" vlvtasks.ldif
$ ldapadd -x -D "cn=Directory Manager" -w $DM_PASS -f vlvtasks.ldif
Then wait for the task to complete successfully; check the progress
with this search:
$ ldapsearch -x -D "cn=Directory Manager" -w $DM_PASS \
-b "cn=index1160589769,cn=index,cn=tasks,cn=config"
Note that the task object is only retained for 10 seconds after the
task finishes. If you want to keep it a longer time, change the
`ttl' attribute in vlvtasks.ldif before creating the object.
After reindexing is complete, restart Dogtag.
Cheers,
Fraser