[Freeipa-users] Client part of server install failing - KRB5CCNAME not defined in HTTP request environment

I am having an issue attempting to install IPA Server. The server component install processes correctly, but when it comes to set up the client components it fails: 2020-04-28T22:41:42Z DEBUG failed to find session_cookie in persistent storage for principal &#39;host/ipa.mydomain.com(a)MYDOMAIN.COM&#39; 2020-04-28T22:41:42Z INFO trying https://ipa.mydomain.com/ipa/json 2020-04-28T22:41:42Z DEBUG Created connection context.rpcclient_1954644240 2020-04-28T22:41:42Z INFO [try 1]: Forwarding 'schema' to json server 'https://ipa.mydomain.com/ipa/json' 2020-04-28T22:41:42Z DEBUG New HTTP connection (ipa.mydomain.com) 2020-04-28T22:41:53Z DEBUG HTTP connection destroyed (ipa.mydomain.com) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 732, in single_request response.msg) ProtocolError: <ProtocolError for ipa.mydomain.com/ipa/json: 500 Internal Server Error> 2020-04-28T22:41:53Z DEBUG Destroyed connection context.rpcclient_1954644240 2020-04-28T22:41:53Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run return cfgr.run() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 360, in run return self.execute() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 386, in execute for rval in self._executor(): File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 655, in _configure next(executor) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 515, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 65, in _install for unused in self._installer(self.parent): File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line 3671, in main install(self) File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line 2392, in install _install(options) File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line 2734, in _install api.finalize() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 739, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 431, in __do_if_not_done getattr(self, name)() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 619, in load_plugins for package in self.packages: File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 949, in packages ipaclient.remote_plugins.get_package(self), File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py", line 134, in get_package plugins = schema.get_package(server_info, client) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 553, in get_package schema = Schema(client) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 401, in __init__ fingerprint, ttl = self._fetch(client, ignore_cache=read_failed) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 426, in _fetch schema = client.forward(u'schema', **kwargs)['result'] File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1190, in forward raise NetworkError(uri=server, error=e.errmsg) 2020-04-28T22:41:53Z DEBUG The ipa-client-install command failed, exception: NetworkError: cannot connect to 'https://ipa.mydomain.com/ipa/json': Internal Server Error The relevant services appear to be running certmonger.service loaded active running Certificate monitoring and dirsrv(a)MYDOMAIN-COM.service loaded active running 389 Directory S gssproxy.service loaded active running GSSAPI Proxy Daemon httpd.service loaded active running The Apache HTTP Server ipa-custodia.service loaded active running IPA Custodia Service ipa-dnskeysyncd.service loaded active running IPA key daemon ipa.service loaded active exited Identity, Policy, Audit kadmin.service loaded active running Kerberos 5 Password-changin krb5kdc.service loaded active running Kerberos 5 KDC named-pkcs11.service loaded active running Berkeley Internet Name Doma ntpd.service loaded active running Network Time Service oddjobd.service loaded active running privileged operations for u pki-tomcatd(a)pki-tomcat.service loaded active running PKI Tomcat Server pki-t I can use kinit to obtain a ticket for admin, but any ipa command that I attempt to run gives an error along the following lines ipa: DEBUG: failed to find session_cookie in persistent storage for principal &#39;admin(a)MYDOMAIN.COM&#39; ipa: INFO: trying https://ipa.mydomain.com/ipa/json ipa: DEBUG: Created connection context.rpcclient_1964217648 ipa: INFO: [try 1]: Forwarding 'schema' to json server 'https://ipa.mydomain .com/ipa/json' ipa: DEBUG: New HTTP connection (ipa.mydomain.com) ipa: DEBUG: HTTP connection destroyed (ipa.mydomain.com) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 732, in single_request response.msg) ProtocolError: <ProtocolError for ipa.mydomain.com/ipa/json: 500 Internal Server Error> ipa: DEBUG: Destroyed connection context.rpcclient_1964217648 ipa: ERROR: cannot connect to 'https://ipa.mydomain.com/ipa/json': Internal Server Error In the httpd error log, I see the same error for every ipa command issued [Wed Apr 29 14:51:19.119357 2020] [:error] [pid 8505] ipa: ERROR: 500 Internal Server Error: KerberosWSGIExecutioner.__call__: KRB5CCNAME not defined in HTTP request environment [Wed Apr 29 14:51:19.120223 2020] [:error] [pid 8505] [remote 192.168.0.2:16498] mod_wsgi (pid=8505): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Wed Apr 29 14:51:19.120335 2020] [:error] [pid 8505] [remote 192.168.0.2:16498] RuntimeError: response has not been started The same error is present at the time the install failed. The Kerberos ticket is valid as ldapsearch works using it [root@ipa1 ~]# ldapsearch -h ipa.mydomain.com -b ou=people,o=ipaca -Y GSSAPI -s sub "(uid=admin)" dn uid SASL/GSSAPI authentication started SASL username: admin(a)MYDOMAIN.COM SASL SSF: 256 SASL data security layer installed. # extended LDIF # # LDAPv3 # base <ou=people,o=ipaca> with scope subtree # filter: (uid=admin) # requesting: dn uid # # admin, people, ipaca dn: uid=admin,ou=people,o=ipaca uid: admin # search result search: 4 result: 0 Success # numResponses: 2 # numEntries: 1 but doesn't without it [root@ipa1 ~]# ldapsearch -h ipa.mydomain.com -b ou=people,o=ipaca -s sub "(uid=admin)" dn uid SASL/EXTERNAL authentication started ldap_sasl_interactive_bind_s: Unknown authentication method (-6) additional info: SASL(-4): no mechanism available: Does anyone have any ideas? I'm tearing my hair out here!