Hi guys.

I'm trying to migrate IPA from Centos 8 over to Centos 9 but I fail.
If the path I try is supported & should work then, first, 'restore' failed with:
...
Restoring umask to 18
CalledProcessError(Command ['/usr/sbin/ipactl', 'start'] returned non-zero exit status 1: 'IPA version error: data needs to be upgraded (expected version \'4.10.1-6.el9\', current version \'4.9.8-7.module_el8.6.0+1103+a004f6a8\')\nAutomatically running upgrade, for details see /var/log/ipaupgrade.log\nBe patient, this may take a few minutes.\nAutomatic upgrade failed: Error caught updating nsDS5ReplicatedAttributeList: Server is unwilling to perform: Entry and attributes are managed by topology plugin.No direct modifications allowed.\nError caught updating nsDS5ReplicatedAttributeListTotal: Server is unwilling to perform: Entry and attributes are managed by topology plugin.No direct modifications allowed.\nUpdate complete\nUpgrading the configuration of the IPA services\n[Verifying that root certificate is published]\n[Migrate CRL publish directory]\nPublish directory already set to new location\nForcing update of template /usr/share/ipa/ipa-pki-proxy.conf.template\nUpgraded /etc/httpd/conf.d/ipa-pki-proxy.conf to version 17\n[Ensuring ephemeralRequest is enabled in KRA]\nephemeralRequest is already enabled\n[Verifying that KDC configuration is using ipa-kdb backend]\n[Fix DS schema file syntax]\n[Removing RA cert from DS NSS database]\n[Enable sidgen and extdom plugins by default]\n[Updating HTTPD service IPA configuration]\n[Updating HTTPD service IPA WSGI configuration]\nNothing to do for configure_httpd_wsgi_conf\n[Migrating from mod_nss to mod_ssl]\nAlready migrated to mod_ssl\n[Moving HTTPD service keytab to gssproxy]\n[Removing self-signed CA]\n[Removing Dogtag 9 CA]\n[Set OpenSSL engine for BIND]\n[Checking for deprecated KDC configuration files]\n[Checking for deprecated backups of Samba configuration files]\ndnssec-validation yes\n[Add missing CA DNS records]\nunable to resolve host name c8kubermaster1.private.lot. to IP address, ipa-ca DNS record will be incomplete\nIPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.\nUnexpected error - see /var/log/ipaupgrade.log for details:\nCalledProcessError: CalledProcessError(Command [\'/bin/systemctl\', \'start\', \'named.service\'] returned non-zero exit status 1: \'Job for named.service failed because the control process exited with error code.\\nSee "systemctl status named.service" and "journalctl -xeu named.service" for details.\\n\')\nThe ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information\n\nSee the upgrade log for more details and/or run /usr/sbin/ipa-server-upgrade again\nAborting ipactl\n')

so I try:
-> $ ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/9]: saving configuration
  [2/9]: disabling listeners
  [3/9]: enabling DS global lock
  [4/9]: disabling Schema Compat
  [5/9]: starting directory server
  [error] CalledProcessError: CalledProcessError(Command ['/bin/systemctl', 'start', 'dirsrv@PRIVATE-LOT.service'] returned non-zero exit status 1: 'Job for dirsrv@PRIVATE-LOT.service failed because a fatal signal was delivered causing the control process to dump core.\nSee "systemctl status dirsrv@PRIVATE-LOT.service" and "journalctl -xeu dirsrv@PRIVATE-LOT.service" for details.\n')
  [cleanup]: stopping directory server
  [cleanup]: restoring configuration
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: CalledProcessError(Command ['/bin/systemctl', 'start', 'dirsrv@PRIVATE-LOT.service'] returned non-zero exit status 1: 'Job for dirsrv@PRIVATE-LOT.service failed because a fatal signal was delivered causing the control process to dump core.\nSee "systemctl status dirsrv@PRIVATE-LOT.service" and "journalctl -xeu dirsrv@PRIVATE-LOT.service" for details.\n')
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

-> $ journalctl -lf -u dirsrv@PRIVATE-LOT.service
Mar 17 16:19:03 c8kubermaster2.private.lot ns-slapd[14967]: [17/Mar/2023:16:19:03.748676397 +0000] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=private,dc=lot--no CoS Templates found, which should be added before the CoS Definition.
Mar 17 16:19:03 c8kubermaster2.private.lot ns-slapd[14967]: [17/Mar/2023:16:19:03.764528091 +0000] - ERR - libdb - BDB2506 file userRoot/replication_changelog.db has LSN 12/7510992, past end of log at 12/2536210
Mar 17 16:19:03 c8kubermaster2.private.lot ns-slapTrd[14967]: [17/Mar/2023:16:19:03.768119982 +0000] - ERR - libdb - BDB2507 Commonly caused by moving a database from one database environment
Mar 17 16:19:03 c8kubermaster2.private.lot ns-slapd[14967]: [17/Mar/2023:16:19:03.771501904 +0000] - ERR - libdb - BDB2508 to another without clearing the database LSNs, or by removing all of
Mar 17 16:19:03 c8kubermaster2.private.lot ns-slapd[14967]: [17/Mar/2023:16:19:03.774956063 +0000] - ERR - libdb - BDB2509 the log files from a database environment
Mar 17 16:19:03 c8kubermaster2.private.lot ns-slapd[14967]: ns-slapd: ldap/servers/plugins/replication/cl5_api.c:1268: cldb_SetReplicaDB: Assertion `cldb' failed.
Mar 17 16:19:03 c8kubermaster2.private.lot systemd-coredump[14993]: [🡕] Process 14967 (ns-slapd) of user 389 dumped core.
Mar 17 16:19:03 c8kubermaster2.private.lot systemd[1]: dirsrv@PRIVATE-LOT.service: Main process exited, code=dumped, status=6/ABRT
Mar 17 16:19:03 c8kubermaster2.private.lot systemd[1]: dirsrv@PRIVATE-LOT.service: Failed with result 'core-dump'.
Mar 17 16:19:03 c8kubermaster2.private.lot systemd[1]: Failed to start 389 Directory Server PRIVATE-LOT..

If such simple process should work then please share your thoughts on what is failing here which can be fixed.

Alternatively, trying the most obvious method - adding new master to existing domain - fails if the new member/master I want to make CA, without CA new master installs/adds.
fails:
...
  [3/30]: creating ACIs for admin
  [4/30]: creating installation admin user
Unable to log in as uid=admin-c9kmaster1.private.lot,ou=people,o=ipaca on ldap://c8kubermaster2.private.lot:389
[hint] tune with replication_wait_timeout
  [error] NotFound: uid=admin-c9kmaster1.private.lot,ou=people,o=ipaca did not replicate to ldap://c8kubermaster2.private.lot:389

and from log file:
...
2023-03-17T17:32:51Z ERROR Unable to log in as uid=admin-c9kmaster1.private.lot,ou=people,o=ipaca on ldap://c8kubermaster2.private.lot:389
2023-03-17T17:32:51Z INFO [hint] tune with replication_wait_timeout
2023-03-17T17:32:51Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 686, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 672, in run_step
    method()
  File "/usr/lib/python3.9/site-packages/ipaserver/install/dogtaginstance.py", line 789, in setup_admin
    raise errors.NotFound(
ipalib.errors.NotFound: uid=admin-c9kmaster1.private.lot,ou=people,o=ipaca did not replicate to ldap://c8kubermaster2.private.lot:389

2023-03-17T17:32:51Z DEBUG   [error] NotFound: uid=admin-c9kmaster1.private.lot,ou=people,o=ipaca did not replicate to ldap://c8kubermaster2.private.lot:389
2023-03-17T17:32:51Z DEBUG Removing /root/.dogtag/pki-tomcat/ca
2023-03-17T17:32:51Z DEBUG   File "/usr/lib/python3.9/site-packages/ipapython/admintool.py", line 180, in execute
...