DNS and kerberos seem to be working fine (and have been for a long while). All `ipa`
commands fail:
```
# kinit admin
Password for admin@$REALM:
# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
# ipa help topics
ipa: ERROR: cannot connect to 'any of the configured servers':
https://$MASTER/ipa/json, https://$REPLICA/ipa/json
```
(yes, the firewall is open)
Attempting to login via the WebUI with user/pass, says `Authenticating...`, then prints
red text: An unknown error occurred. (or something to that effect).
The apache error log shows:
```
[Tue Nov 06 07:46:46.388297 2018] [:error] [pid 23816] ipa: INFO: *** PROCESS START ***
[Tue Nov 06 07:46:46.862410 2018] [:error] [pid 23815] ipa: INFO: *** PROCESS START ***
[Tue Nov 06 07:48:55.510961 2018] [:error] [pid 23816] ipa: ERROR: 500 Internal Server
Error: KerberosWSGIExecutioner.__call__: KRB5CCNAME not defined in HTTP request
environment
[Tue Nov 06 07:48:55.512943 2018] [:error] [pid 23816] [remote $MASTER_IP:52342] mod_wsgi
(pid=23816): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
[Tue Nov 06 07:48:55.513207 2018] [:error] [pid 23816] [remote $MASTER_IP:52342]
RuntimeError: response has not been started
[Tue Nov 06 17:09:21.111120 2018] [:error] [pid 23815] ipa: ERROR: 500 Internal Server
Error: KerberosWSGIExecutioner.__call__: KRB5CCNAME not defined in HTTP request
environment
[Tue Nov 06 17:09:21.113133 2018] [:error] [pid 23815] [remote $MASTER_IP:52342] mod_wsgi
(pid=23815): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
[Tue Nov 06 17:09:21.113410 2018] [:error] [pid 23815] [remote $MASTER_IP:52342]
RuntimeError: response has not been started
[Tue Nov 06 17:17:28.498098 2018] [auth_gssapi:error] [pid 23819] [client $CLIENT:36060]
NO AUTH DATA Client did not send any authentication headers, referer:
https://$MASTER/ipa/ui/
[Tue Nov 06 17:17:28.522306 2018] [auth_gssapi:error] [pid 23819] [client $CLIENT:36060]
NO AUTH DATA Client did not send any authentication headers, referer:
https://$MASTER/ipa/ui/
[Tue Nov 06 17:17:35.408453 2018] [:error] [pid 23815] [remote $CLIENT:24687] mod_wsgi
(pid=23815): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
[Tue Nov 06 17:17:35.408776 2018] [:error] [pid 23815] [remote $CLIENT:24687] Traceback
(most recent call last):
[Tue Nov 06 17:17:35.408944 2018] [:error] [pid 23815] [remote $CLIENT:24687] File
"/usr/share/ipa/wsgi.py", line 51, in application
[Tue Nov 06 17:17:35.409572 2018] [:error] [pid 23815] [remote $CLIENT:24687] return
api.Backend.wsgi_dispatch(environ, start_response)
[Tue Nov 06 17:17:35.409666 2018] [:error] [pid 23815] [remote $CLIENT:24687] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 262, in
__call__
[Tue Nov 06 17:17:35.471519 2018] [:error] [pid 23815] [remote $CLIENT:24687] return
self.route(environ, start_response)
[Tue Nov 06 17:17:35.471701 2018] [:error] [pid 23815] [remote $CLIENT:24687] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 274, in route
[Tue Nov 06 17:17:35.471923 2018] [:error] [pid 23815] [remote $CLIENT:24687] return
app(environ, start_response)
[Tue Nov 06 17:17:35.472027 2018] [:error] [pid 23815] [remote $CLIENT:24687] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 929, in
__call__
[Tue Nov 06 17:17:35.472163 2018] [:error] [pid 23815] [remote $CLIENT:24687]
self.kinit(user_principal, password, ipa_ccache_name)
[Tue Nov 06 17:17:35.472244 2018] [:error] [pid 23815] [remote $CLIENT:24687] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 965, in kinit
[Tue Nov 06 17:17:35.472378 2018] [:error] [pid 23815] [remote $CLIENT:24687]
pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM],
[Tue Nov 06 17:17:35.472461 2018] [:error] [pid 23815] [remote $CLIENT:24687] File
"/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 125, in
kinit_armor
[Tue Nov 06 17:17:35.474208 2018] [:error] [pid 23815] [remote $CLIENT:24687]
run(args, env=env, raiseonerr=True, capture_error=True)
[Tue Nov 06 17:17:35.474308 2018] [:error] [pid 23815] [remote $CLIENT:24687] File
"/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 512, in run
[Tue Nov 06 17:17:35.480086 2018] [:error] [pid 23815] [remote $CLIENT:24687] raise
CalledProcessError(p.returncode, arg_string, str(output))
[Tue Nov 06 17:17:35.480364 2018] [:error] [pid 23815] [remote $CLIENT:24687]
CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_23815 -X
X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit
status 1
```
I'm not above trying to troubleshoot this a little, but honestly it's probably
faster to reinstall both master and replica. The problem isn't a bug, it was most
certainly my blundering.
Being able to recover the 20-30 DNS entries (somehow) would be super nice. If I could
recover the 5-10 host-details, even better. I don't care too much about my three
users, they can just be told to re-enter their passwords :D
In case it's important, this is Centos 7, 32-bit, running on a Raspberry Pi 3. I had
to use the Oracle Java, and hand-edit a pki-related-file.py (somewhere) to tweak a startup
timeout. Otherwise it was working brilliantly for a long time, until I screwed it up.