Run nmap from each server at each other server in your replication cluster. Double-check the firewall allows access between servers. As a bad-form test, disable all firewalls and run a sync test.

The install error indicates things are not setup correctly. Any errors in install should be seen as blockers and a --uninstall should be run before any new attempts. The number of moving parts in a freeipa install is large and all must be perfect. When in doubt a dnf reinstall may be a good starting point.

The logs you provided suggest that there's no network connection between servers. Either name resolution failure or keytab failure caused by incorrect names are suspect. I'm hoping the example.com domain is a deliberate sanitizer version as that may be resolvable.

On December 28, 2021 7:47:00 PM EST, Chris Roadfeldt via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
For the past couple months, I've been struggling to get replicas up and running. Have tried using containers and VMs, ended up rebuilding my FreeIPA install from the ground up to eliminate corruption as an issue. The failures are consistent, regardless of install options and appear to be related to replication itself. Initial replication works, but replication after that fails. Attached are the errors encountered during the ipa-replica-install command, along with the relevant log entries.

The primary server is currently on a Fedora 35 VM running the following RPMs.
freeipa-client-common-4.9.8-1.fc35.noarch
freeipa-server-common-4.9.8-1.fc35.noarch
freeipa-common-4.9.8-1.fc35.noarch
freeipa-client-4.9.8-1.fc35.x86_64
freeipa-healthcheck-core-0.9-3.fc35.noarch
freeipa-server-4.9.8-1.fc35.x86_64
freeipa-server-dns-4.9.8-1.fc35.noarch
freeipa-server-trust-ad-4.9.8-1.fc35.x86_64
freeipa-selinux-4.9.8-1.fc35.noarch
freeipa-healthcheck-0.9-3.fc35.noarch


Here are the replica installs for the container and VM along with the relevant ipareplica-install.log entries.


Container first, here's the output from ipa-replica-install command.

[9/21]: configuring httpd
Nothing to do for configure_httpd_wsgi_conf
[10/21]: setting up httpd keytab
[error] NotFound: wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

/var/log/ipareplica-install.log entries

2021-12-28T18:46:57Z DEBUG stderr=Keytab successfully retrieved and stored in: /var/lib/ipa/gssproxy/http.keytab

2021-12-28T18:46:57Z DEBUG Waiting up to 300 seconds for replication (ldap://primary.example.com:389) krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=ac
counts,dc=example,dc=com (objectclass=*)
2021-12-28T18:47:06Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:47:16Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:47:26Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:47:36Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:47:46Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:47:56Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:48:06Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:48:16Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:48:26Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:48:36Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:48:46Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:48:56Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:49:06Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:49:16Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:49:26Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:49:36Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:49:46Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:49:56Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:50:06Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:50:16Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:50:26Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:50:36Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:50:46Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:50:56Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:51:06Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:51:16Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:51:26Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:51:36Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:51:46Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:51:56Z DEBUG Still waiting for replication of krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:51:57Z DEBUG Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/ipaserver/install/service.py", line 635, in start_creation
run_step(full_msg, method)
File "/usr/lib/python3.10/site-packages/ipaserver/install/service.py", line 621, in run_step
method()
File "/usr/lib/python3.10/site-packages/ipaserver/install/httpinstance.py", line 634, in request_service_keytab
replication.wait_for_entry(
File "/usr/lib/python3.10/site-packages/ipaserver/install/replication.py", line 208, in wait_for_entry
raise errors.NotFound(
ipalib.errors.NotFound: wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=roadfel
dt,dc=com

2021-12-28T18:51:57Z DEBUG [error] NotFound: wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services
,cn=accounts,dc=example,dc=com
2021-12-28T18:51:57Z DEBUG File "/usr/lib/python3.10/site-packages/ipapython/admintool.py", line 180, in execute
return_value = self.run()
File "/usr/lib/python3.10/site-packages/ipapython/install/cli.py", line 342, in run
return cfgr.run()
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 360, in run
return self.execute()
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 386, in execute
for rval in self._executor():
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 431, in __runner
exc_handler(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 421, in __runner
step()
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 655, in _configure
next(executor)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 431, in __runner
exc_handler(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 518, in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 515, in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 421, in __runner
step()
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.10/site-packages/ipapython/install/common.py", line 65, in _install
for unused in self._installer(self.parent):
File "/usr/lib/python3.10/site-packages/ipaserver/install/server/__init__.py", line 603, in main
replica_install(self)
File "/usr/lib/python3.10/site-packages/ipaserver/install/server/replicainstall.py", line 401, in decorated
func(installer)
File "/usr/lib/python3.10/site-packages/ipaserver/install/server/replicainstall.py", line 1315, in install
install_http(
File "/usr/lib/python3.10/site-packages/ipaserver/install/server/replicainstall.py", line 163, in install_http
http.create_instance(
File "/usr/lib/python3.10/site-packages/ipaserver/install/httpinstance.py", line 151, in create_instance
self.start_creation()
File "/usr/lib/python3.10/site-packages/ipaserver/install/service.py", line 635, in start_creation
run_step(full_msg, method)
File "/usr/lib/python3.10/site-packages/ipaserver/install/service.py", line 621, in run_step
method()
File "/usr/lib/python3.10/site-packages/ipaserver/install/httpinstance.py", line 634, in request_service_keytab
replication.wait_for_entry(
File "/usr/lib/python3.10/site-packages/ipaserver/install/replication.py", line 208, in wait_for_entry
raise errors.NotFound(

2021-12-28T18:51:57Z DEBUG The ipa-replica-install command failed, exception: NotFound: wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:51:57Z ERROR wait_for_entry timeout on ldap://primary.example.com:389 for krbprincipalname=HTTP/replica1.example.com@EXAMPLE.COM,cn=services,cn=accounts,dc=example,dc=com
2021-12-28T18:51:57Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

VM install output

Done configuring ipa-otpd.
Custodia uses 'primary.example.com' as master peer.
Configuring ipa-custodia
[1/4]: Generating ipa-custodia config file
[2/4]: Generating ipa-custodia keys
[3/4]: starting ipa-custodia
[4/4]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

Incorrect number of results (0) searching for public key for host/primary.example.com@EXAMPLE.COM


/var/log/ipareplica-install.log entries

2021-12-29T00:40:10Z DEBUG Done configuring ipa-custodia.
2021-12-29T00:40:10Z DEBUG service duration: ipa-custodia 2.37 sec
2021-12-29T00:40:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2021-12-29T00:40:10Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2021-12-29T00:40:10Z DEBUG Waiting up to 300 seconds to see our keys appear on host ldap://primary.example.com
2021-12-29T00:40:10Z DEBUG File "/usr/lib/python3.10/site-packages/ipapython/admintool.py", line 180, in execute
return_value = self.run()
File "/usr/lib/python3.10/site-packages/ipapython/install/cli.py", line 342, in run
return cfgr.run()
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 360, in run
return self.execute()
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 386, in execute
for rval in self._executor():
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 431, in __runner
exc_handler(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 421, in __runner
step()
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 655, in _configure
next(executor)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 431, in __runner
exc_handler(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 518, in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 515, in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 421, in __runner
step()
File "/usr/lib/python3.10/site-packages/ipapython/install/core.py", line 418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3.10/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.10/site-packages/ipapython/install/common.py", line 65, in _install
for unused in self._installer(self.parent):
File "/usr/lib/python3.10/site-packages/ipaserver/install/server/__init__.py", line 603, in main
replica_install(self)
File "/usr/lib/python3.10/site-packages/ipaserver/install/server/replicainstall.py", line 401, in decorated
func(installer)
File "/usr/lib/python3.10/site-packages/ipaserver/install/server/replicainstall.py", line 1345, in install
ca.install(False, config, options, custodia=custodia)
File "/usr/lib/python3.10/site-packages/ipaserver/install/ca.py", line 270, in install
install_step_0(standalone, replica_config, options, custodia=custodia)
File "/usr/lib/python3.10/site-packages/ipaserver/install/ca.py", line 306, in install_step_0
custodia.get_ca_keys(
File "/usr/lib/python3.10/site-packages/ipaserver/install/custodiainstance.py", line 296, in get_ca_keys
self._get_keys(cacerts_file, cacerts_pwd, data)
File "/usr/lib/python3.10/site-packages/ipaserver/install/custodiainstance.py", line 252, in _get_keys
cli = self._get_custodia_client()
File "/usr/lib/python3.10/site-packages/ipaserver/install/custodiainstance.py", line 241, in _get_custodia_client
return CustodiaClient(
File "/usr/lib/python3.10/site-packages/ipaserver/secrets/client.py", line 70, in __init__
self._server_keys(), self._client_keys()
File "/usr/lib/python3.10/site-packages/ipaserver/secrets/client.py", line 80, in _server_keys
sk = JWK(**json_decode(self.ikk.find_key(principal, KEY_USAGE_SIG)))
File "/usr/lib/python3.10/site-packages/ipaserver/secrets/kem.py", line 224, in find_key
return conn.get_key(usage, kid)
File "/usr/lib/python3.10/site-packages/ipaserver/secrets/kem.py", line 78, in get_key
raise ValueError("Incorrect number of results (%d) searching for "

2021-12-29T00:40:10Z DEBUG The ipa-replica-install command failed, exception: ValueError: Incorrect number of results (0) searching for public key for host/primary.example.com@EXAMPLE.COM
2021-12-29T00:40:10Z ERROR Incorrect number of results (0) searching for public key for host/primary.example.com@EXAMPLE.COM
2021-12-29T00:40:10Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

--
Computers amplify human error
Super computers are really cool