Recently we have noticed that our master server is borderline
unusable. The webui will time out with the message 'Web UI got in unrecoverable state
during "metadata" phase.' Running commands on the command line take
minutes to run. While trouble shooting this I found these errors in the http error_log.
DOMAIN.NET is our redacted name.
[Fri Mar 01 00:28:10.054324 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 200, in
read_input
[Fri Mar 01 00:28:10.054326 2019] [:error] [pid 70749] return
environ['wsgi.input'].read(length).decode('utf-8')
[Fri Mar 01 00:28:10.054329 2019] [:error] [pid 70749] IOError: request data read error
[Fri Mar 01 00:28:10.054474 2019] [:error] [pid 70749] ipa: INFO: [jsonserver_kerb]
admin(a)DOMAIN.NET: None: InternalError
[Fri Mar 01 00:28:10.070493 2019] [:error] [pid 70749] ipa: ERROR: non-public: IOError:
request data read error
[Fri Mar 01 00:28:10.070514 2019] [:error] [pid 70749] Traceback (most recent call
last):
[Fri Mar 01 00:28:10.070517 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 360, in
wsgi_execute
[Fri Mar 01 00:28:10.070520 2019] [:error] [pid 70749] data = read_input(environ)
[Fri Mar 01 00:28:10.070528 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 200, in
read_input
[Fri Mar 01 00:28:10.070531 2019] [:error] [pid 70749] return
environ['wsgi.input'].read(length).decode('utf-8')
[Fri Mar 01 00:28:10.070533 2019] [:error] [pid 70749] IOError: request data read error
[Fri Mar 01 00:28:10.070698 2019] [:error] [pid 70749] ipa: INFO: [xmlserver]
admin(a)DOMAIN.NET: None: InternalError
[Fri Mar 01 00:28:10.086787 2019] [:error] [pid 70749] ipa: ERROR: non-public: IOError:
request data read error
[Fri Mar 01 00:28:10.086809 2019] [:error] [pid 70749] Traceback (most recent call
last):
[Fri Mar 01 00:28:10.086812 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 360, in
wsgi_execute
[Fri Mar 01 00:28:10.086815 2019] [:error] [pid 70749] data = read_input(environ)
[Fri Mar 01 00:28:10.086817 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 200, in
read_input
[Fri Mar 01 00:28:10.086820 2019] [:error] [pid 70749] return
environ['wsgi.input'].read(length).decode('utf-8')
[Fri Mar 01 00:28:10.086823 2019] [:error] [pid 70749] IOError: request data read error
[Fri Mar 01 00:28:10.086968 2019] [:error] [pid 70749] ipa: INFO: [jsonserver_kerb]
admin(a)DOMAIN.NET: None: InternalError
[Fri Mar 01 00:28:10.103055 2019] [:error] [pid 70749] ipa: ERROR: non-public: IOError:
request data read error
[Fri Mar 01 00:28:10.103075 2019] [:error] [pid 70749] Traceback (most recent call
last):
[Fri Mar 01 00:28:10.103077 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 360, in
wsgi_execute
[Fri Mar 01 00:28:10.103080 2019] [:error] [pid 70749] data = read_input(environ)
[Fri Mar 01 00:28:10.103083 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 200, in
read_input
[Fri Mar 01 00:28:10.103085 2019] [:error] [pid 70749] return
environ['wsgi.input'].read(length).decode('utf-8')
[Fri Mar 01 00:28:10.103088 2019] [:error] [pid 70749] IOError: request data read error
[Fri Mar 01 00:28:10.103245 2019] [:error] [pid 70749] ipa: INFO: [xmlserver]
admin(a)DOMAIN.NET: None: InternalError
[Fri Mar 01 00:28:10.119618 2019] [:error] [pid 70749] ipa: INFO: [jsonserver_kerb]
admin(a)DOMAIN.NET: ping(): SUCCESS
[Fri Mar 01 00:28:10.138727 2019] [:error] [pid 70749] ipa: INFO: [jsonserver_session]
admin(a)DOMAIN.NET: ping(): SUCCESS
[Fri Mar 01 00:28:10.142020 2019] [:warn] [pid 70986] [client 172.31.59.230:50746] failed
to set perms (3140) on file (/var/run/ipa/ccaches/admin(a)DOMAIN.NET)!, referer:
https://ipa-1.domain.net/ipa/xml
This is a decent isolating test because ping doesn't do any activity
beyond returning some static data. This rules out a lot of things to
check (LDAP, Kerberos).
In this case it is failing waiting to read the POST data from wsgi.
Are the versions of mod_wsgi and httpd the same between the working and
non-working masters?
What release of IPA is this on what distribution?
rob