I think I (finally) figured out what went wrong, but I have no idea how to proceed. Somehow I missed an error during the setup, and so now I think whatever CA I had has been clobbered and rendered useless (since it was wiped and reinstalled after this error occurred):

Please don't read too much into any typos below -- I had to type this by hand as the originals are on an internal development network and this was actually faster than transferring files in this direction.

2017-03-29T12:01:22Z DEBUG cert valid True for "CN=zsipa.damascusgrp.com,O=damascusgrp.com"
2017-03-29T12:01:22Z DEBUG handshake complete, peer = 192.168.208.53:8443
2017-03-29T12:01:22Z DEBUG Protocol: TLS1.2
2017-03-29T12:01:22Z DEBUG Cipher: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2017-03-29T12:01:22Z DEBUG response status 204
2017-03-29T12:01:22Z DEBUG response headers {'set-cookie': 'JSESSIONID=7B6440F45777C030B70AC2CCEE7CE780; Path=/ca/; Secure; HttpOnlhy', 'expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'server': 'Apache-Coyote/1.1', 'cache-control': 'private', 'date': 'Wed, 29 Mar 2017 12:01:21 GMT', 'content-type': 'application/xml'}
2017-03-29T12:01:22Z DEBUG response body ''
2017-03-29T12:01:22Z DEBUG Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 449, in start_creation
    run_step(full_msg, method)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 439, in run_step
    method()
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1887, in import_included_profiles
    _create_dogtag_profile(profile_id, profile_data, overwrite=True)
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 2008, in _create_dogtag_profile
    profile_api.update_profile(profile_id, profile_data)
File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 2147, in update_profile
    body=profile_data
File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 2106, in _ssldo
    % {'status': status, 'explanation': explanation}
RemoteRetrieveError: Non-2xx reponse from CA REST API: 404.

2017-03-29T12:01:22Z DEBUG   [error] RemoteRetrieveError: Non-2xx response from CA REST API: 404.
2017-03-29T12:01:22Z DEBUG   File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 752, in run_script
    return_value = main_function()

File "/usr/sbin/ipa-ca-install", line 302, in main
    install(safe_options, options, filename)

File "/usr/sbin/ipa-ca-install", line 242, in install
    install_replica(safe_options, options, filename)

File "/usr/sbin/ipa-ca-install", line 204, in install_replica
    ca.install(True, config, options)

File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 118, in install
    install_step_0(standalone, replica_config, options)

File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 140, in install_step_0
    ra_p12=getattr(options, 'ra_p12', None)

File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1562, in install_replica_ca
    subject_base=config.subject_base)

File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 437, in configure_instance
    self.start_creation(runtime=210)

File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 449, in start_creation
    run_step(full_msg, method)

File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 439, in run_step
    method()

File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1887, in import_included_profiles
    _create_dogtag_profile(profile_id, profile_data, overwrite=True)

File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 2008, in _create_dogtag_profile
    profile_api.update_profile(profile_id, profile_data)

File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 2147, in update_profile
    body=profile_data

File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 2106, in _ssldo
    % {'status': status, 'explanation': explanation}

2017-03-29T12:01:22Z DEBUG The iopa-ca-install command failed, exception: RemoteRetrieveError: Non-2xx response from CA REST API: 404.

On 06/05/2017 11:07 AM, Bret Wortman wrote:

I've also just realized that replication appears to have ceased; I have entries in some IPA servers but not all.

[root@zsipa ~]# ipa-replica-manage list
Directory Manager password:

zsipa.damascusgrp.com: master
zsipa2.damascusgrp.com: master
zsipa3.damascusgrp.com: master
[root@zsipa ~]# ipa-replica-manage list zsipa.damascusgrp.com
Directory Manager password:

zsipa3.damascusgrp.com: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: Error (19) Replication error acquiring replica: Replica has different database generation ID, remote replica may need to be initialized (RUV error)
last update ended: 1970-01-01 00:00:00+00:00
[root@zsipa ~]#

Only zsipa3 is listed as a replica anywhere, and it's not a functioning one. I can set up replication between zsipa and zsipa2, but is there a good way to bring zsipa3 back in line as well?

The background is that we attempted to do a rolling update of our IPA servers by bringing in a new server, zsipa2, and then upgrading each of the other two from Fedora to Centos 7 and then initialized them as replicas of zsipa2. But apparently, this didn't work as we had thought. So add replication errors to the certificate issue I'm still trying to run to ground.

--

Bret Wortman

Damascus Products

ph/fax: 1-855-644-2783

Wrap Buddies now available for preorder!