[Freeipa-users] Unable to create replicas.

For the past couple months, I've been struggling to get replicas up and running. Have tried using containers and VMs, ended up rebuilding my FreeIPA install from the ground up to eliminate corruption as an issue. The failures are consistent, regardless of install options and appear to be related to replication itself. Initial replication works, but replication after that fails. Attached are the errors encountered during the ipa-replica-install command, along with the relevant log entries. The primary server is currently on a Fedora 35 VM running the following RPMs. freeipa-client-common-4.9.8-1.fc35.noarch freeipa-server-common-4.9.8-1.fc35.noarch freeipa-common-4.9.8-1.fc35.noarch freeipa-client-4.9.8-1.fc35.x86_64 freeipa-healthcheck-core-0.9-3.fc35.noarch freeipa-server-4.9.8-1.fc35.x86_64 freeipa-server-dns-4.9.8-1.fc35.noarch freeipa-server-trust-ad-4.9.8-1.fc35.x86_64 freeipa-selinux-4.9.8-1.fc35.noarch freeipa-healthcheck-0.9-3.fc35.noarch Here are the replica installs for the container and VM along with the relevant ipareplica-install.log entries. Container first, here's the output from ipa-replica-install command. [9/21]: configuring httpd Nothing to do for configure_httpd_wsgi_conf [10/21]: setting up httpd keytab [error] NotFound: wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information /var/log/ipareplica-install.log entries 2021-12-28T18:46:57Z DEBUG stderr=Keytab successfully retrieved and stored in: /var/lib/ipa/gssproxy/http.keytab 2021-12-28T18:46:57Z DEBUG Waiting up to 300 seconds for replication (ldap://primary.example.com:389) krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=ac counts,dc=example,dc=com (objectclass=*) 2021-12-28T18:47:06Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:47:16Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:47:26Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:47:36Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:47:46Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:47:56Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:48:06Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:48:16Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:48:26Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:48:36Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:48:46Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:48:56Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:49:06Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:49:16Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:49:26Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:49:36Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:49:46Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:49:56Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:50:06Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:50:16Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:50:26Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:50:36Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:50:46Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:50:56Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:51:06Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:51:16Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:51:26Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:51:36Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:51:46Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:51:56Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:51:57Z DEBUG Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/ipaserver/install/service.py", line 635, in start_creation run_step(full_msg, method) File "/usr/lib/python3.10/site-packages/ipaserver/install/service.py", line 621, in run_step method() File "/usr/lib/python3.10/site-packages/ipaserver/install/httpinstance.py", line 634, in request_service_keytab replication.wait_for_entry( File "/usr/lib/python3.10/site-packages/ipaserver/install/replication.py", line 208, in wait_for_entry raise errors.NotFound( ipalib.errors.NotFound: wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=roadfel dt,dc=com 2021-12-28T18:51:57Z DEBUG [error] NotFound: wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services ,cn=accounts,dc=example,dc=com 2021-12-28T18:51:57Z DEBUG File "/usr/lib/python3.10/site-packages/ipapython/admintool.py", line 180, in execute return_value = self.run() File "/usr/lib/python3.10/site-packages/ipapython/install/cli.py", line 342, in run return cfgr.run() File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 360, in run return self.execute() File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 386, in execute for rval in self._executor(): File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 655, in _configure next(executor) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 518, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 515, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.10/site-packages/ipapython/install/common.py", line 65, in _install for unused in self._installer(self.parent): File "/usr/lib/python3.10/site-packages/ipaserver/install/server/__init__.py", line 603, in main replica_install(self) File "/usr/lib/python3.10/site-packages/ipaserver/install/server/replicainstall.py", line 401, in decorated func(installer) File "/usr/lib/python3.10/site-packages/ipaserver/install/server/replicainstall.py", line 1315, in install install_http( File "/usr/lib/python3.10/site-packages/ipaserver/install/server/replicainstall.py", line 163, in install_http http.create_instance( File "/usr/lib/python3.10/site-packages/ipaserver/install/httpinstance.py", line 151, in create_instance self.start_creation() File "/usr/lib/python3.10/site-packages/ipaserver/install/service.py", line 635, in start_creation run_step(full_msg, method) File "/usr/lib/python3.10/site-packages/ipaserver/install/service.py", line 621, in run_step method() File "/usr/lib/python3.10/site-packages/ipaserver/install/httpinstance.py", line 634, in request_service_keytab replication.wait_for_entry( File "/usr/lib/python3.10/site-packages/ipaserver/install/replication.py", line 208, in wait_for_entry raise errors.NotFound( 2021-12-28T18:51:57Z DEBUG The ipa-replica-install command failed, exception: NotFound: wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:51:57Z ERROR wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com(a)EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com 2021-12-28T18:51:57Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information VM install output Done configuring ipa-otpd. Custodia uses 'primary.example.com' as master peer. Configuring ipa-custodia [1/4]: Generating ipa-custodia config file [2/4]: Generating ipa-custodia keys [3/4]: starting ipa-custodia [4/4]: configuring ipa-custodia to start on boot Done configuring ipa-custodia. Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. Incorrect number of results (0) searching for public key for host/primary.example.com(a)EXAMPLE.COM /var/log/ipareplica-install.log entries 2021-12-29T00:40:10Z DEBUG Done configuring ipa-custodia. 2021-12-29T00:40:10Z DEBUG service duration: ipa-custodia 2.37 sec 2021-12-29T00:40:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2021-12-29T00:40:10Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state' 2021-12-29T00:40:10Z DEBUG Waiting up to 300 seconds to see our keys appear on host ldap://primary.example.com 2021-12-29T00:40:10Z DEBUG File "/usr/lib/python3.10/site-packages/ipapython/admintool.py", line 180, in execute return_value = self.run() File "/usr/lib/python3.10/site-packages/ipapython/install/cli.py", line 342, in run return cfgr.run() File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 360, in run return self.execute() File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 386, in execute for rval in self._executor(): File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 655, in _configure next(executor) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 518, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 515, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.10/site-packages/ipapython/install/common.py", line 65, in _install for unused in self._installer(self.parent): File "/usr/lib/python3.10/site-packages/ipaserver/install/server/__init__.py", line 603, in main replica_install(self) File "/usr/lib/python3.10/site-packages/ipaserver/install/server/replicainstall.py", line 401, in decorated func(installer) File "/usr/lib/python3.10/site-packages/ipaserver/install/server/replicainstall.py", line 1345, in install ca.install(False, config, options, custodia=custodia) File "/usr/lib/python3.10/site-packages/ipaserver/install/ca.py", line 270, in install install_step_0(standalone, replica_config, options, custodia=custodia) File "/usr/lib/python3.10/site-packages/ipaserver/install/ca.py", line 306, in install_step_0 custodia.get_ca_keys( File "/usr/lib/python3.10/site-packages/ipaserver/install/custodiainstance.py", line 296, in get_ca_keys self._get_keys(cacerts_file, cacerts_pwd, data) File "/usr/lib/python3.10/site-packages/ipaserver/install/custodiainstance.py", line 252, in _get_keys cli = self._get_custodia_client() File "/usr/lib/python3.10/site-packages/ipaserver/install/custodiainstance.py", line 241, in _get_custodia_client return CustodiaClient( File "/usr/lib/python3.10/site-packages/ipaserver/secrets/client.py", line 70, in __init__ self._server_keys(), self._client_keys() File "/usr/lib/python3.10/site-packages/ipaserver/secrets/client.py", line 80, in _server_keys sk = JWK(**json_decode(self.ikk.find_key(principal, KEY_USAGE_SIG))) File "/usr/lib/python3.10/site-packages/ipaserver/secrets/kem.py", line 224, in find_key return conn.get_key(usage, kid) File "/usr/lib/python3.10/site-packages/ipaserver/secrets/kem.py", line 78, in get_key raise ValueError("Incorrect number of results (%d) searching for " 2021-12-29T00:40:10Z DEBUG The ipa-replica-install command failed, exception: ValueError: Incorrect number of results (0) searching for public key for host/primary.example.com(a)EXAMPLE.COM 2021-12-29T00:40:10Z ERROR Incorrect number of results (0) searching for public key for host/primary.example.com(a)EXAMPLE.COM 2021-12-29T00:40:10Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information