I have made some progress since my last post which was entitled " Unable to start directory server after updates" A little background- this server was running as the master for over a year and was integrated with Active Directory. AFAIK, all DNS was setup properly, otherwise how could you integrate with AD without having DNS setup properly? #rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server package freeipa-server is not installed package freeipa-client is not installed ipa-server-4.9.2-4.module_el8.4.0+846+96522ed7.x86_64 ipa-client-4.9.2-4.module_el8.4.0+846+96522ed7.x86_64 389-ds-base-1.4.3.16-19.module_el8.4.0+884+e9b25896.x86_64 pki-ca-10.10.5-3.module_el8.4.0+816+beb6e9a3.noarch krb5-server-1.18.2-8.el8.x86_64 Here are the steps i have taken so far- 1.Centos was updated from 8.2 to 8.4.2105 and now the problems began 2. Directory Server would not start so.. 3. IPA was updated/ugraded from 4.8.4 to 4.9.2 (successfully) 4. Directory server now worked not but named-pkcs11 service would not start. 5. Ipactl start -f reported that ipa-dnskeysyncd was started when actually it was having issues. A review of journal logs after turning on some debugging revealed that- The unit ipa-dnskeysyncd.service has entered the 'failed' state with result 'exit-code'. 6. At this point I found someone who was able to fix the ipa-dnskeysyncd by running ipa-dns-install again. 7. named-pkcs11 and ipa-dnskeysyncd are now fixed but ipa-ods-exporter is now broken after this install See the install sequence below: [root@utility ipa]# ipa-dns-install --dnssec-master The log file for this installation can be found in /var/log/ipaserver-dns-install.log ============================================================================== This program will setup DNS for the IPA Server. This includes: * Configure DNS (bind) * Configure SoftHSM (required by DNSSEC) * Configure ipa-dnskeysyncd (required by DNSSEC) * Configure ipa-ods-exporter (required by DNSSEC key master) * Configure OpenDNSSEC (required by DNSSEC key master) * Generate DNSSEC master key (required by DNSSEC key master) NOTE: DNSSEC zone signing is not enabled by default Plan carefully, replacing DNSSEC key master is not recommended To accept the default shown in brackets, press the Enter key. Do you want to setup this IPA server as DNSSEC key master? [no]: yes Do you want to configure DNS forwarders? [yes]: Following DNS servers are configured in /etc/resolv.conf: 127.0.0.1 Do you want to configure these servers as DNS forwarders? [yes]: no Enter an IP address for a DNS forwarder, or press Enter to skip: 172.30.50.10 DNS forwarder 172.30.50.10 added. You may add another. Enter an IP address for a DNS forwarder, or press Enter to skip: DNS forwarders: 172.30.50.10 Checking DNS forwarders, please wait ... Do you want to search for missing reverse zones? [yes]: The following operations may take some minutes to complete. Please wait until the prompt is returned. Configuring DNS (named) [1/8]: generating rndc key file [2/8]: setting up our own record [3/8]: adding NS record to the zones [4/8]: setting up kerberos principal [5/8]: setting up named.conf [6/8]: setting up server configuration [7/8]: configuring named to start on boot [8/8]: changing resolv.conf to point to ourselves Done configuring DNS (named). Restarting the web server to pick up resolv.conf changes Configuring DNS key synchronization service (ipa-dnskeysyncd) [1/7]: checking status [2/7]: setting up bind-dyndb-ldap working directory [3/7]: setting up kerberos principal [4/7]: setting up SoftHSM [5/7]: adding DNSSEC containers [6/7]: creating replica keys [7/7]: configuring ipa-dnskeysyncd to start on boot Done configuring DNS key synchronization service (ipa-dnskeysyncd). Configuring IPA OpenDNSSEC exporter daemon (ipa-ods-exporter) [1/5]: setting up DNS Key Exporter [2/5]: setting up kerberos principal [3/5]: disabling default signer daemon [4/5]: starting DNS Key Exporter [5/5]: configuring DNS Key Exporter to start on boot Done configuring IPA OpenDNSSEC exporter daemon (ipa-ods-exporter). Configuring OpenDNSSEC enforcer daemon (ods-enforcerd) [1/7]: setting up configuration files [2/7]: setting up ownership and file mode bits [3/7]: generating master key [4/7]: setting up OpenDNSSEC [5/7]: setting up ipa-dnskeysyncd [6/7]: starting OpenDNSSEC enforcer [7/7]: configuring OpenDNSSEC enforcer to start on boot Done configuring OpenDNSSEC enforcer daemon (ods-enforcerd). Restarting ipa-dnskeysyncd Restarting named Updating DNS system records ============================================================================== Setup complete Global DNS configuration in LDAP server is empty You can use 'dnsconfig-mod' command to set global DNS options that would override settings in local named.conf files You must make sure these network ports are open: TCP Ports: * 53: bind UDP Ports: * 53: bind [root@utility ipa]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-ods-exporter Service: STOPPED ods-enforcerd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@utility ipa]# ipactl start Existing service file detected! Assuming stale, cleaning and proceeding Starting Directory Service Starting krb5kdc Service Starting kadmin Service Starting named Service Starting httpd Service Starting ipa-custodia Service Starting pki-tomcatd Service Starting smb Service Starting winbind Service Starting ipa-otpd Service Starting ipa-ods-exporter Service Starting ods-enforcerd Service Starting ipa-dnskeysyncd Service ipa: INFO: The ipactl command was successful [root@utility ipa]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-ods-exporter Service: RUNNING ods-enforcerd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@utility ipa]# ipactl stop Stopping ipa-dnskeysyncd Service Stopping ods-enforcerd Service Stopping ipa-ods-exporter Service Stopping ipa-otpd Service Stopping winbind Service Stopping smb Service Stopping pki-tomcatd Service Stopping ipa-custodia Service Stopping httpd Service Stopping named Service Stopping kadmin Service Stopping krb5kdc Service Stopping Directory Service ipa: INFO: The ipactl command was successful [root@utility ipa]# ipactl start Starting Directory Service Starting krb5kdc Service Starting kadmin Service Starting named Service Starting httpd Service Starting ipa-custodia Service Starting pki-tomcatd Service Starting smb Service Starting winbind Service Starting ipa-otpd Service Starting ipa-ods-exporter Service Starting ods-enforcerd Service Starting ipa-dnskeysyncd Service ipa: INFO: The ipactl command was successful [root@utility ipa]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-ods-exporter Service: STOPPED ods-enforcerd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@utility log]# systemctl status ipa-ods-exporter ● ipa-ods-exporter.service - IPA OpenDNSSEC Signer replacement Loaded: loaded (/usr/lib/systemd/system/ipa-ods-exporter.service; disabled; vendor preset: disabled) Active: inactive (dead) since Sun 2021-09-05 05:55:32 CDT; 8min ago Process: 5089 ExecStart=/usr/libexec/ipa/ipa-ods-exporter (code=exited, status=0/SUCCESS) Main PID: 5089 (code=exited, status=0/SUCCESS) Sep 05 05:55:32 utility.idm.nac-issa.org ipa-ods-exporter[5089]: ipa-ods-exporter: DEBUG new zone keys in local HSM: set() Sep 05 05:55:32 utility.idm.nac-issa.org ipa-ods-exporter[5089]: ipa-ods-exporter: INFO master2ldap_zone_keys: keys in local HSM & LDAP: > Sep 05 05:55:32 utility.idm.nac-issa.org ipa-ods-exporter[5089]: ipa-ods-exporter: INFO master2ldap_zone_keys: keys in local HSM & LDAP: > Sep 05 05:55:32 utility.idm.nac-issa.org ipa-ods-exporter[5089]: ipa-ods-exporter: DEBUG accepting new connection Sep 05 05:55:32 utility.idm.nac-issa.org ipa-ods-exporter[5089]: ipa-ods-exporter: DEBUG accepted new connection <ipaserver.dnssec._ods21.> Sep 05 05:55:32 utility.idm.nac-issa.org ipa-ods-exporter[5089]: ipa-ods-exporter: DEBUG received command "ipa-hsm-update" from systemd so> Sep 05 05:55:32 utility.idm.nac-issa.org ipa-ods-exporter[5089]: ipa-ods-exporter: DEBUG parse_command returned msg: HSM synchronization f> Sep 05 05:55:32 utility.idm.nac-issa.org ipa-ods-exporter[5089]: ipa-ods-exporter: DEBUG parse_command returned cmd: ipa-hsm-update Sep 05 05:55:32 utility.idm.nac-issa.org ipa-ods-exporter[5089]: ipa-ods-exporter: INFO HSM synchronization finished, skipping zone synch> Sep 05 05:55:32 utility.idm.nac-issa.org systemd[1]: ipa-ods-exporter.service: Succeeded. [root@utility log]# systemctl start ipa-ods-exporter [root@utility log]# systemctl status ipa-ods-exporter ● ipa-ods-exporter.service - IPA OpenDNSSEC Signer replacement Loaded: loaded (/usr/lib/systemd/system/ipa-ods-exporter.service; disabled; vendor preset: disabled) Active: active (running) since Sun 2021-09-05 06:05:05 CDT; 2s ago Main PID: 5170 (ipa-ods-exporte) Tasks: 1 (limit: 37317) Memory: 72.8M CGroup: /system.slice/ipa-ods-exporter.service └─5170 /usr/libexec/platform-python -I /usr/libexec/ipa/ipa-ods-exporter Sep 05 06:05:07 utility.idm.nac-issa.org ipa-ods-exporter[5170]: ipa-ods-exporter: DEBUG enabled replica key ids: {'0xa75863f538583b5d6d74> Sep 05 06:05:07 utility.idm.nac-issa.org ipa-ods-exporter[5170]: ipa-ods-exporter: DEBUG processing master key data: 0xd2c3520e317cf8ae4b1> Sep 05 06:05:07 utility.idm.nac-issa.org ipa-ods-exporter[5170]: ipa-ods-exporter: DEBUG master key 0xd2c3520e317cf8ae4b1194a52a3c2e1b is > Sep 05 06:05:07 utility.idm.nac-issa.org ipa-ods-exporter[5170]: ipa-ods-exporter: DEBUG processing master key data: 0x284dcba6251d8d68f46> Sep 05 06:05:07 utility.idm.nac-issa.org ipa-ods-exporter[5170]: ipa-ods-exporter: DEBUG master key 0x284dcba6251d8d68f464e447537f3bb0 is > Sep 05 06:05:07 utility.idm.nac-issa.org ipa-ods-exporter[5170]: ipa-ods-exporter: DEBUG zone keys in LDAP: set() Sep 05 06:05:07 utility.idm.nac-issa.org ipa-ods-exporter[5170]: ipa-ods-exporter: DEBUG zone keys in local HSM: set() Sep 05 06:05:07 utility.idm.nac-issa.org ipa-ods-exporter[5170]: ipa-ods-exporter: DEBUG new zone keys in local HSM: set() Sep 05 06:05:07 utility.idm.nac-issa.org ipa-ods-exporter[5170]: ipa-ods-exporter: INFO master2ldap_zone_keys: keys in local HSM & LDAP: > Sep 05 06:05:07 utility.idm.nac-issa.org ipa-ods-exporter[5170]: ipa-ods-exporter: INFO master2ldap_zone_keys: keys in local HSM & LDAP: > [root@utility log]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-ods-exporter Service: RUNNING ods-enforcerd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@utility log]# systemctl status ipa-ods-exporter ● ipa-ods-exporter.service - IPA OpenDNSSEC Signer replacement Loaded: loaded (/usr/lib/systemd/system/ipa-ods-exporter.service; disabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Sun 2021-09-05 06:06:12 CDT; 7s ago Process: 5316 ExecStart=/usr/libexec/ipa/ipa-ods-exporter (code=exited, status=1/FAILURE) Main PID: 5316 (code=exited, status=1/FAILURE) [root@utility log]# systemctl start ipa-ods-exporter [root@utility log]# systemctl status ipa-ods-exporter ● ipa-ods-exporter.service - IPA OpenDNSSEC Signer replacement Loaded: loaded (/usr/lib/systemd/system/ipa-ods-exporter.service; disabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Sun 2021-09-05 06:07:15 CDT; 3s ago Process: 5330 ExecStart=/usr/libexec/ipa/ipa-ods-exporter (code=exited, status=1/FAILURE) Main PID: 5330 (code=exited, status=1/FAILURE) [root@utility log]# systemctl stop ipa-ods-exporter Warning: Stopping ipa-ods-exporter.service, but it can still be activated by: ipa-ods-exporter.socket [root@utility log]# systemctl start ipa-ods-exporter.socket [root@utility log]# systemctl status ipa-ods-exporter ● ipa-ods-exporter.service - IPA OpenDNSSEC Signer replacement Loaded: loaded (/usr/lib/systemd/system/ipa-ods-exporter.service; disabled; vendor preset: disabled) Active: inactive (dead) (Result: exit-code) since Sun 2021-09-05 06:08:09 CDT; 37s ago Process: 5330 ExecStart=/usr/libexec/ipa/ipa-ods-exporter (code=exited, status=1/FAILURE) Main PID: 5330 (code=exited, status=1/FAILURE) Sep 05 06:08:09 utility.idm.nac-issa.org systemd[1]: Stopped IPA OpenDNSSEC Signer replacement. [root@utility log]# systemctl start ipa-ods-exporter [root@utility log]# systemctl status ipa-ods-exporter ● ipa-ods-exporter.service - IPA OpenDNSSEC Signer replacement Loaded: loaded (/usr/lib/systemd/system/ipa-ods-exporter.service; disabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Sun 2021-09-05 06:09:01 CDT; 169ms ago Process: 5348 ExecStart=/usr/libexec/ipa/ipa-ods-exporter (code=exited, status=1/FAILURE) Main PID: 5348 (code=exited, status=1/FAILURE) [root@utility log]# journalctl -xe Sep 05 06:09:03 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirrors.vcea.wsu.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: validating mirror.vcu.edu/CNAME: bad cache hit (edu/DS) Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirror.vcu.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: validating mirrors.cmich.edu/A: bad cache hit (edu/DS) Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirrors.cmich.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: validating mirrors.lug.mtu.edu/A: bad cache hit (edu/DS) Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirrors.lug.mtu.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: *validating edu/DNSKEY: bad cache hit (edu/DS) Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirror.cs.pitt.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: validating edu/DNSKEY: bad cache hit (edu/DS) Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'bay.uchicago.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: validating centos.s.uw.edu/CNAME: bad cache hit (edu/DS) Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'centos.s.uw.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: validating ftp.usf.edu/A: bad cache hit (edu/DS) Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'ftp.usf.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: validating mirrors.vcea.wsu.edu/A: bad cache hit (edu/DS) Sep 05 06:09:06 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirrors.vcea.wsu.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: validating gnome.org/DNSKEY: bad cache hit (gnome.org/DS) Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: validating gnome.org/DNSKEY: bad cache hit (gnome.org/DS) Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'extensions.gnome.org/AAAA/IN': 172.30.50.10#53 Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'extensions.gnome.org/A/IN': 172.30.50.10#53 Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: validating mirror.vcu.edu/CNAME: bad cache hit (edu/DS) Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirror.vcu.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: validating mirrors.cmich.edu/A: bad cache hit (edu/DS) Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirrors.cmich.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: *validating edu/DNSKEY: bad cache hit (edu/DS) Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirror.cs.pitt.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: validating mirrors.lug.mtu.edu/A: bad cache hit (edu/DS) Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: validating edu/DNSKEY: bad cache hit (edu/DS) Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirrors.lug.mtu.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:10 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'bay.uchicago.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:11 utility.idm.nac-issa.org named-pkcs11[4487]: validating ftp.usf.edu/A: bad cache hit (edu/DS) Sep 05 06:09:11 utility.idm.nac-issa.org named-pkcs11[4487]: validating centos.s.uw.edu/CNAME: bad cache hit (edu/DS) Sep 05 06:09:11 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'ftp.usf.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:11 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'centos.s.uw.edu/A/IN': 172.30.50.10#53 Sep 05 06:09:11 utility.idm.nac-issa.org named-pkcs11[4487]: validating mirrors.vcea.wsu.edu/A: bad cache hit (edu/DS) Sep 05 06:09:11 utility.idm.nac-issa.org named-pkcs11[4487]: broken trust chain resolving 'mirrors.vcea.wsu.edu/A/IN': 172.30.50.10#53 [root@utility log]# reboot