1:08 a.m.
Recently we have noticed that our master server is borderline unusable. The webui will
time out with the message 'Web UI got in unrecoverable state during
"metadata" phase.' Running commands on the command line take minutes to
run. While trouble shooting this I found these errors in the http error_log. DOMAIN.NET
is our redacted name.
[Fri Mar 01 00:28:10.054324 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 200, in
read_input
[Fri Mar 01 00:28:10.054326 2019] [:error] [pid 70749] return
environ['wsgi.input'].read(length).decode('utf-8')
[Fri Mar 01 00:28:10.054329 2019] [:error] [pid 70749] IOError: request data read error
[Fri Mar 01 00:28:10.054474 2019] [:error] [pid 70749] ipa: INFO: [jsonserver_kerb]
admin(a)DOMAIN.NET: None: InternalError
[Fri Mar 01 00:28:10.070493 2019] [:error] [pid 70749] ipa: ERROR: non-public: IOError:
request data read error
[Fri Mar 01 00:28:10.070514 2019] [:error] [pid 70749] Traceback (most recent call last):
[Fri Mar 01 00:28:10.070517 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 360, in
wsgi_execute
[Fri Mar 01 00:28:10.070520 2019] [:error] [pid 70749] data = read_input(environ)
[Fri Mar 01 00:28:10.070528 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 200, in
read_input
[Fri Mar 01 00:28:10.070531 2019] [:error] [pid 70749] return
environ['wsgi.input'].read(length).decode('utf-8')
[Fri Mar 01 00:28:10.070533 2019] [:error] [pid 70749] IOError: request data read error
[Fri Mar 01 00:28:10.070698 2019] [:error] [pid 70749] ipa: INFO: [xmlserver]
admin(a)DOMAIN.NET: None: InternalError
[Fri Mar 01 00:28:10.086787 2019] [:error] [pid 70749] ipa: ERROR: non-public: IOError:
request data read error
[Fri Mar 01 00:28:10.086809 2019] [:error] [pid 70749] Traceback (most recent call last):
[Fri Mar 01 00:28:10.086812 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 360, in
wsgi_execute
[Fri Mar 01 00:28:10.086815 2019] [:error] [pid 70749] data = read_input(environ)
[Fri Mar 01 00:28:10.086817 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 200, in
read_input
[Fri Mar 01 00:28:10.086820 2019] [:error] [pid 70749] return
environ['wsgi.input'].read(length).decode('utf-8')
[Fri Mar 01 00:28:10.086823 2019] [:error] [pid 70749] IOError: request data read error
[Fri Mar 01 00:28:10.086968 2019] [:error] [pid 70749] ipa: INFO: [jsonserver_kerb]
admin(a)DOMAIN.NET: None: InternalError
[Fri Mar 01 00:28:10.103055 2019] [:error] [pid 70749] ipa: ERROR: non-public: IOError:
request data read error
[Fri Mar 01 00:28:10.103075 2019] [:error] [pid 70749] Traceback (most recent call last):
[Fri Mar 01 00:28:10.103077 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 360, in
wsgi_execute
[Fri Mar 01 00:28:10.103080 2019] [:error] [pid 70749] data = read_input(environ)
[Fri Mar 01 00:28:10.103083 2019] [:error] [pid 70749] File
"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 200, in
read_input
[Fri Mar 01 00:28:10.103085 2019] [:error] [pid 70749] return
environ['wsgi.input'].read(length).decode('utf-8')
[Fri Mar 01 00:28:10.103088 2019] [:error] [pid 70749] IOError: request data read error
[Fri Mar 01 00:28:10.103245 2019] [:error] [pid 70749] ipa: INFO: [xmlserver]
admin(a)DOMAIN.NET: None: InternalError
[Fri Mar 01 00:28:10.119618 2019] [:error] [pid 70749] ipa: INFO: [jsonserver_kerb]
admin(a)DOMAIN.NET: ping(): SUCCESS
[Fri Mar 01 00:28:10.138727 2019] [:error] [pid 70749] ipa: INFO: [jsonserver_session]
admin(a)DOMAIN.NET: ping(): SUCCESS
[Fri Mar 01 00:28:10.142020 2019] [:warn] [pid 70986] [client 172.31.59.230:50746] failed
to set perms (3140) on file (/var/run/ipa/ccaches/admin(a)DOMAIN.NET)!, referer:
https://ipa-1.domain.net/ipa/xml