[Freeipa-users] IPA web login: 401 "Login failed due to an unknown reason."

We are unable to login to the FreeIPA web console. However, it is able to tell when I use an incorrect password (shows "The password you entered is incorrect.") Also one of the CentOS servers getting ssh login credentials from our ipa server is using my old password (expired several weeks ago.) These lines are from httpd error.log ipa: INFO: 401 Unauthorized: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) SSL Library Error: -12269 The server has rejected your certificate as expired I've spent quite a bit of time researching the issue. At first I thought it was because a recent upgrade to FreeIPA (CentOS 7) ipa-server-4.6.6-11.el7.centos.x86_64 But when I looked back in /var/log/messages I can see the error(s) appear to have started occurring before the upgrade. from journalctl: Jun 03 22:41:52 ipa.michiganlabs.com systemd[1]: Started IPA key daemon. Jun 03 22:41:54 ipa.michiganlabs.com python2[14471]: GSSAPI client step 1 Jun 03 22:41:54 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: ipa-dnskeysyncd: INFO LDAP bind... Jun 03 22:41:54 ipa.michiganlabs.com python2[14471]: GSSAPI client step 1 Jun 03 22:41:54 ipa.michiganlabs.com python2[14471]: GSSAPI client step 1 Jun 03 22:41:54 ipa.michiganlabs.com python2[14471]: GSSAPI client step 2 Jun 03 22:41:54 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: ipa-dnskeysyncd: INFO Commencing sync process Jun 03 22:41:54 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: ipaserver.dnssec.keysyncer: INFO Initial LDAP dump is done, sychronizing with ODS and BIND Jun 03 22:41:55 ipa.michiganlabs.com python2[14481]: GSSAPI client step 1 Jun 03 22:41:55 ipa.michiganlabs.com python2[14481]: GSSAPI client step 1 Jun 03 22:41:55 ipa.michiganlabs.com python2[14481]: GSSAPI client step 1 Jun 03 22:41:55 ipa.michiganlabs.com python2[14481]: ObjectStore.cpp(59): Failed to enumerate object store in /var/lib/softhsm/tokens/ Jun 03 22:41:55 ipa.michiganlabs.com python2[14481]: SoftHSM.cpp(476): Could not load the object store Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: Traceback (most recent call last): Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: File "/usr/libexec/ipa/ipa-dnskeysyncd", line 116, in <module> Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: while ldap_connection.syncrepl_poll(all=1, msgid=ldap_search): Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: File "/usr/lib64/python2.7/site-packages/ldap/syncrepl.py", line 405, in syncrepl_poll Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: self.syncrepl_refreshdone() Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: File "/usr/lib/python2.7/site-packages/ipaserver/dnssec/keysyncer.py", line 126, in syncrepl_refreshdon Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: self.hsm_replica_sync() Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: File "/usr/lib/python2.7/site-packages/ipaserver/dnssec/keysyncer.py", line 192, in hsm_replica_sync Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: ipautil.run([paths.IPA_DNSKEYSYNCD_REPLICA]) Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 563, in run Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: raise CalledProcessError(p.returncode, arg_string, str(output)) Jun 03 22:41:55 ipa.michiganlabs.com ipa-dnskeysyncd[14471]: subprocess.CalledProcessError: Command '/usr/libexec/ipa/ipa-dnskeysync-replica' returned non-zero exit Jun 03 22:41:55 ipa.michiganlabs.com systemd[1]: ipa-dnskeysyncd.service: main process exited, code=exited, status=1/FAILURE Jun 03 22:41:55 ipa.michiganlabs.com systemd[1]: Unit ipa-dnskeysyncd.service entered failed state. Jun 03 22:41:55 ipa.michiganlabs.com systemd[1]: ipa-dnskeysyncd.service failed. Jun 03 22:42:55 ipa.michiganlabs.com systemd[1]: ipa-dnskeysyncd.service holdoff time over, scheduling restart. Jun 03 22:42:55 ipa.michiganlabs.com systemd[1]: Stopped IPA key daemon. possibly relevant output of Number of certificates and requests being tracked: 8. Request ID '20170928130908': status: CA_UNCONFIGURED ca-error: Unable to determine principal name for signing request. stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-IPA-MICHIGANLABS-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-IPA-MICHIGANLABS-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-IPA-MICHIGANLABS-COM',nickname='Server-Cert' CA: IPA issuer: subject: expires: unknown pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv IPA-MICHIGANLABS-COM track: yes auto-renew: yes possibly relevant SELinux info: type=PROCTITLE msg=audit(1591756304.759:2543): proctitle=2F7573722F62696E2F707974686F6E32002F7573722F6C6962657865632F6970612F6970612D646E736B657973796E632D7265706C696361 type=SYSCALL msg=audit(1591756304.759:2543): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=de4478 a2=90800 a3=0 items=0 ppid=22017 pid=22027 auid=4294967295 uid=994 gid=25 euid=994 suid=994 fsuid=994 egid=25 sgid=25 fsgid=25 tty=(none) ses=4294967295 comm="ipa-dnskeysync-" exe="/usr/bin/python2.7" subj=system_u:system_r:ipa_dnskey_t:s0 key=(null) type=AVC msg=audit(1591756304.759:2543): avc: denied { read } for pid=22027 comm="ipa-dnskeysync-" name="tokens" dev="dm-6" ino=1373802 scontext=system_u:system_r:ipa_dnskey_t:s0 tcontext=system_u:object_r:named_cache_t:s0 tclass=dir permissive=0 output of Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful NOTE: it hangs for like at least 30 seconds when trying to display "ipa-dnskeysyncd Service: RUNNING" This server was setup slightly less than two years ago using this as a basis of instructions: https://www.digitalocean.com/community/tutorials/how-to-set-up-centralize... Also there was this note: "ipa.michiganlabs.com - Delegate ipa. dns to FreeIPA server" Thanks for any assistance