ipa-client-install randomly fails with slapi_access_allowed does not allow WRITE to ipaProtectedOperation; write_keys!
2:58 p.m.
Hi there,
We have IPA (VERSION: 4.9.10, API_VERSION: 2.248) running on Alma Linux on 8.7 with total
of 4 replicas. We're running in a cloud, so have an automated process in place where
new instances automatically enrol to IPA when launching (they all use the same IPA user
and fetch the password from a secrets manager). For a while now we have been seeing
instances fail to enrol to IPA on random occasions, which is more pronounced when multiple
instances are starting at the same time.
Each instance runs ipa-client-install, like below, when it starts:
ipa-client-install --mkhomedir --ssh-trust-dns --domain=example.com -w${PASSW}
-phost-enrollment --unattended --force-join --no-dns-sshfp
This sometimes fails with the following:
Starting external process
args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmpdqzuq_ts',
'-A', '-n', 'CA certificate 1', '-t', 'C,,',
'-a', '-f', '/tmp/tmpdqzuq_ts/pwdfile.txt']
Process finished, return code=0
stdout=
stderr=
failed to find session_cookie in persistent storage for principal
'host/ip-172-26-1-238.xxx(a)EXAMPLE.COM'
trying https://ipa2.example.com/ipa/json
New HTTP connection (ipa2.example.com)
HTTP connection destroyed (ipa2.example.com)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py",
line 120, in get_package
plugins = api._remote_plugins
AttributeError: 'API' object has no attribute '_remote_plugins'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 644, in
get_auth_info
response = self._sec_context.step()
File "<decorator-gen-15>", line 2, in step
File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 167, in
check_last_err
return func(self, *args, **kwargs)
File "<decorator-gen-5>", line 2, in step
File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 127, in
catch_and_return_token
return func(self, *args, **kwargs)
File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 521, in
step
return self._initiator_step(token=token)
File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 542, in
_initiator_step
token)
File "gssapi/raw/sec_contexts.pyx", line 244, in
gssapi.raw.sec_contexts.init_sec_context
gssapi.raw.misc.GSSError: Major (851968): Unspecified GSS failure. Minor code may provide
more information, Minor (2529639068): Cannot contact any KDC for realm
'EXAMPLE.COM'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in
single_request
self.get_auth_info()
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in
get_auth_info
self._handle_exception(e, service=service)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in
_handle_exception
raise errors.KerberosError(message=unicode(e))
ipalib.errors.KerberosError: Major (851968): Unspecified GSS failure. Minor code may
provide more information, Minor (2529639068): Cannot contact any KDC for realm
'EXAMPLE.COM'
File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in
execute
return_value = self.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 344, in
run
return cfgr.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 360,
in run
return self.execute()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 386,
in execute
for rval in self._executor():
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431,
in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460,
in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421,
in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418,
in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in
run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in
run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 655,
in _configure
next(executor)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431,
in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460,
in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518,
in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515,
in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421,
in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418,
in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in
run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in
run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 65,
in _install
for unused in self._installer(self.parent):
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
3961, in main
install(self)
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
2655, in install
_install(options)
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
2972, in _install
api.finalize()
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 753, in
finalize
self.__do_if_not_done('load_plugins')
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 432, in
__do_if_not_done
getattr(self, name)()
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 632, in
load_plugins
for package in self.packages:
File "/usr/lib/python3.6/site-packages/ipalib/__init__.py", line 952, in
packages
ipaclient.remote_plugins.get_package(self),
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py",
line 128, in get_package
plugins = schema.get_package(server_info, client)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 546, in get_package
schema = Schema(client)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 395, in __init__
fingerprint, ttl = self._fetch(client, ignore_cache=read_failed)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 407, in _fetch
client.connect(verbose=False)
File "/usr/lib/python3.6/site-packages/ipalib/backend.py", line 69, in
connect
conn = self.create_connection(*args, **kw)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1064, in
create_connection
command([],
{}
)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1276, in _call
return self.__request(name, args)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1243, in
__request
verbose=self.__verbose >= 3,
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in
single_request
self.get_auth_info()
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in
get_auth_info
self._handle_exception(e, service=service)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in
_handle_exception
raise errors.KerberosError(message=unicode(e))
The ipa-client-install command failed, exception: KerberosError: Major (851968):
Unspecified GSS failure. Minor code may provide more information, Minor (2529639068):
Cannot contact any KDC for realm 'EXAMPLE.COM'
Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor
(2529639068): Cannot contact any KDC for realm 'EXAMPLE.COM'
The ipa-client-install command failed. See /var/log/ipaclient-install.log for more
information
This program will set up IPA client.
Version 4.9.10
On IPA server the following pops up in logs:
ERR - is_allowed_to_access_attr - [file ipa_pwd_extop.c, line 825]: slapi_access_allowed
does not allow WRITE to ipaProtectedOperation;write_keys!
ERR - ipapwd_getkeytab - [file ipa_pwd_extop.c, line 1714]: Not allowed to set keytab on
[host/ip-172-26-1-238.xxx(a)EXAMPLE.COM]!
This doesn't happen every time - even when multiple instances are launched from the
same image, some will fail and some will enrol successfully. It's worse when instances
are in different cloud region that IPA (even when they are very close, network-wise, so
latency shouldn't be an issue), but can still happen within the same region. For some
reason, this has also become worse when we switched from forcing a specific IPA server
(--server to ipa-client-install) to DNS auto-discovery.
We commonly have situations where 5 instances try to launch at mostly the same time and
try to enrol using 2 replicas - and all 5 will fail, both IPAs showing the same errors (as
above).
We've run out of ideas of what to debug and how, so any clues would be appreciated.
3:25 p.m.
New subject: ipa-client-install randomly fails with slapi_access_allowed does not allow WRITE to ipaProtectedOperation; write_keys!
In some cases the error message from ipa-client-install is different (but still thrown at
certuril):
Starting external process
args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmpiat7ggvf',
'-A', '-n', 'CA certificate 1', '-t', 'C,,',
'-a', '-f', '/tmp/tmpiat7ggvf/pwdfile.txt']
Process finished, return code=0
stdout=
stderr=
failed to find session_cookie in persistent storage for principal
'host/ip-172-22-1-106.xxx(a)EXAMPLE.COM'
trying https://ipa2.example.com/ipa/json
New HTTP connection (ipa2.example.com)
HTTP connection destroyed (ipa2.example.com)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py",
line 120, in get_package
plugins = api._remote_plugins
AttributeError: 'API' object has no attribute '_remote_plugins'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 644, in
get_auth_info
response = self._sec_context.step()
File "<decorator-gen-15>", line 2, in step
File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 167, in
check_last_err
return func(self, *args, **kwargs)
File "<decorator-gen-5>", line 2, in step
File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 127, in
catch_and_return_token
return func(self, *args, **kwargs)
File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 521, in
step
return self._initiator_step(token=token)
File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 542, in
_initiator_step
token)
File "gssapi/raw/sec_contexts.pyx", line 244, in
gssapi.raw.sec_contexts.init_sec_context
gssapi.raw.misc.GSSError: Major (851968): Unspecified GSS failure. Minor code may provide
more information, Minor (2529638932): TGT has been revoked
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in
single_request
self.get_auth_info()
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in
get_auth_info
self._handle_exception(e, service=service)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in
_handle_exception
raise errors.KerberosError(message=unicode(e))
ipalib.errors.KerberosError: Major (851968): Unspecified GSS failure. Minor code may
provide more information, Minor (2529638932): TGT has been revoked
File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in
execute
return_value = self.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 344, in
run
return cfgr.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 360,
in run
return self.execute()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 386,
in execute
for rval in self._executor():
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431,
in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460,
in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421,
in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418,
in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in
run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in
run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 655,
in _configure
next(executor)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431,
in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460,
in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518,
in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515,
in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421,
in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418,
in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in
run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in
run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 65,
in _install
for unused in self._installer(self.parent):
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
3961, in main
install(self)
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
2655, in install
_install(options)
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
2972, in _install
api.finalize()
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 753, in
finalize
self.__do_if_not_done('load_plugins')
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 432, in
__do_if_not_done
getattr(self, name)()
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 632, in
load_plugins
for package in self.packages:
File "/usr/lib/python3.6/site-packages/ipalib/__init__.py", line 952, in
packages
ipaclient.remote_plugins.get_package(self),
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py",
line 128, in get_package
plugins = schema.get_package(server_info, client)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 546, in get_package
schema = Schema(client)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 395, in __init__
fingerprint, ttl = self._fetch(client, ignore_cache=read_failed)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 407, in _fetch
client.connect(verbose=False)
File "/usr/lib/python3.6/site-packages/ipalib/backend.py", line 69, in
connect
conn = self.create_connection(*args, **kw)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1064, in
create_connection
command([],
{}
)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1276, in _call
return self.__request(name, args)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1243, in
__request
verbose=self.__verbose >= 3,
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in
single_request
self.get_auth_info()
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in
get_auth_info
self._handle_exception(e, service=service)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in
_handle_exception
raise errors.KerberosError(message=unicode(e))
The ipa-client-install command failed, exception: KerberosError: Major (851968):
Unspecified GSS failure. Minor code may provide more information, Minor (2529638932): TGT
has been revoked
Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor
(2529638932): TGT has been revoked
The ipa-client-install command failed. See /var/log/ipaclient-install.log for more
information
This program will set up IPA client.
Version 4.9.10
However the error on IPA server side is the same.
3:20 a.m.
New subject: ipa-client-install randomly fails with slapi_access_allowed does not allow WRITE to ipaProtectedOperation; write_keys!
On ma, 21 marras 2022, Paulina Budzon via FreeIPA-users wrote:
In some cases the error message from ipa-client-install is different
(but still thrown at certuril):
Starting external process
args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmpiat7ggvf',
'-A', '-n', 'CA certificate 1', '-t', 'C,,',
'-a', '-f', '/tmp/tmpiat7ggvf/pwdfile.txt']
Process finished, return code=0
stdout=
stderr=
certutil returned 0, so it is just fine. The output below is unrelated
to certutil use; you can ignore certutil part. Please see more below.
failed to find session_cookie in persistent storage for principal
'host/ip-172-22-1-106.xxx(a)EXAMPLE.COM'
trying https://ipa2.example.com/ipa/json
New HTTP connection (ipa2.example.com)
HTTP connection destroyed (ipa2.example.com)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py",
line 120, in get_package
plugins = api._remote_plugins
AttributeError: 'API' object has no attribute '_remote_plugins'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 644, in
get_auth_info
response = self._sec_context.step()
File "<decorator-gen-15>", line 2, in step
File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 167, in
check_last_err
return func(self, *args, **kwargs)
File "<decorator-gen-5>", line 2, in step
File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 127, in
catch_and_return_token
return func(self, *args, **kwargs)
File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 521, in
step
return self._initiator_step(token=token)
File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 542, in
_initiator_step
token)
File "gssapi/raw/sec_contexts.pyx", line 244, in
gssapi.raw.sec_contexts.init_sec_context
gssapi.raw.misc.GSSError: Major (851968): Unspecified GSS failure. Minor code may provide
more information, Minor (2529638932): TGT has been revoked
This is coming from an attempt to get a Kerberos service ticket using
credentials for the user you are using to enroll this machine. Since you
are passing '-w$password' and not any specific principal, this means it
is the machine itself, hence we see
failed to find session_cookie in persistent storage for principal
'host/ip-172-22-1-106.xxx(a)EXAMPLE.COM'
this is fine at that point because we need a session cookie to talk to
IPA server's API endpoint and we don't have any yet. So we attempt to
kinit with that password you passed and fail.
'TGT has been revoked' error comes from your KDC on IPA master. Please check
/var/log/krb5kdc.log on IPA server you connected to for this deployment.
There should be one of explaining messages prior to rejection. It might
be prefixed with 'PAC issue:' string
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in
single_request
self.get_auth_info()
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in
get_auth_info
self._handle_exception(e, service=service)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in
_handle_exception
raise errors.KerberosError(message=unicode(e))
ipalib.errors.KerberosError: Major (851968): Unspecified GSS failure. Minor code may
provide more information, Minor (2529638932): TGT has been revoked
File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in
execute
return_value = self.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 344, in
run
return cfgr.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 360,
in run
return self.execute()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 386,
in execute
for rval in self._executor():
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431,
in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460,
in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421,
in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418,
in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in
run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in
run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 655,
in _configure
next(executor)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431,
in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460,
in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518,
in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515,
in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421,
in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418,
in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in
run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in
run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 65,
in _install
for unused in self._installer(self.parent):
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
3961, in main
install(self)
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
2655, in install
_install(options)
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
2972, in _install
api.finalize()
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 753, in
finalize
self.__do_if_not_done('load_plugins')
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 432, in
__do_if_not_done
getattr(self, name)()
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 632, in
load_plugins
for package in self.packages:
File "/usr/lib/python3.6/site-packages/ipalib/__init__.py", line 952, in
packages
ipaclient.remote_plugins.get_package(self),
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py",
line 128, in get_package
plugins = schema.get_package(server_info, client)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 546, in get_package
schema = Schema(client)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 395, in __init__
fingerprint, ttl = self._fetch(client, ignore_cache=read_failed)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 407, in _fetch
client.connect(verbose=False)
File "/usr/lib/python3.6/site-packages/ipalib/backend.py", line 69, in
connect
conn = self.create_connection(*args, **kw)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1064, in
create_connection
command([],
{}
)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1276, in _call
return self.__request(name, args)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1243, in
__request
verbose=self.__verbose >= 3,
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in
single_request
self.get_auth_info()
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in
get_auth_info
self._handle_exception(e, service=service)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in
_handle_exception
raise errors.KerberosError(message=unicode(e))
The ipa-client-install command failed, exception: KerberosError: Major (851968):
Unspecified GSS failure. Minor code may provide more information, Minor (2529638932): TGT
has been revoked
Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor
(2529638932): TGT has been revoked
The ipa-client-install command failed. See /var/log/ipaclient-install.log for more
information
This program will set up IPA client.
Version 4.9.10
However the error on IPA server side is the same.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
4:04 a.m.
New subject: ipa-client-install randomly fails with slapi_access_allowed does not allow WRITE to ipaProtectedOperation; write_keys!
Thanks for your help!
This is coming from an attempt to get a Kerberos service ticket
using
credentials for the user you are using to enroll this machine. Since you
are passing '-w$password' and not any specific principal, this means it
is the machine itself, hence we see
I'm passing -phost-enrollment (host-enrollment is the user for the password in -w),
should I be adding something more?
'TGT has been revoked' error comes from your KDC on IPA
master. Please check
/var/log/krb5kdc.log on IPA server you connected to for this deployment.
There should be one of explaining messages prior to rejection. It might
be prefixed with 'PAC issue:' string
There's nothing around this exact time, the only bits regarding a specific failed host
that I could find are:
krb5kdc[4526](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18),
aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), a
es128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 172.22.2.123: NEEDED_PREAUTH:
host/ip-172-22-2-123.xxx(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Additional
pre-authentication required
krb5kdc[4526](info): closing down fd 4
krb5kdc[4525](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18),
aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17),
aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 172.22.2.123: ISSUE: authtime
1669105826, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18),
ses=aes256-cts-hmac-sha1-96(18)}, host/ip-172-22-2-123.xxx(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
(Different host in the logs above than before, but we had this issue again this morning
and it was easier to look up that older logs).
In terms of errors, the only one I could find in the logs was:
krb5kdc[27486](Error): PAC issue: ipadb_get_principal failed.
This was logged right before the whole set of instances got their errors.
I did also notice that the some of the same hostnames exist in older Kerberos logs
(hostnames will get repeated in our cloud env every now and then), could this be the
cause? A host with a previously used hostname trying to enrol again? We have an automated
process in place that calls host-del to IPA when an instance is terminated to delete it
and its data from IPA, but maybe we should be clearing something from Kerberos directly
too?
4:33 a.m.
New subject: ipa-client-install randomly fails with slapi_access_allowed does not allow WRITE to ipaProtectedOperation; write_keys!
On ti, 22 marras 2022, Paulina Budzon via FreeIPA-users wrote:
Thanks for your help!
> This is coming from an attempt to get a Kerberos service ticket using
> credentials for the user you are using to enroll this machine. Since you
> are passing '-w$password' and not any specific principal, this means it
> is the machine itself, hence we see
I'm passing -phost-enrollment (host-enrollment is the user for the password in -w),
should I be adding something more?
> 'TGT has been revoked' error comes from your KDC on IPA master. Please check
> /var/log/krb5kdc.log on IPA server you connected to for this deployment.
> There should be one of explaining messages prior to rejection. It might
> be prefixed with 'PAC issue:' string
There's nothing around this exact time, the only bits regarding a specific failed host
that I could find are:
krb5kdc[4526](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18),
aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), a
es128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 172.22.2.123: NEEDED_PREAUTH:
host/ip-172-22-2-123.xxx(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Additional
pre-authentication required
krb5kdc[4526](info): closing down fd 4
krb5kdc[4525](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18),
aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17),
aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 172.22.2.123: ISSUE: authtime
1669105826, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18),
ses=aes256-cts-hmac-sha1-96(18)}, host/ip-172-22-2-123.xxx(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
(Different host in the logs above than before, but we had this issue again this morning
and it was easier to look up that older logs).
In terms of errors, the only one I could find in the logs was:
krb5kdc[27486](Error): PAC issue: ipadb_get_principal failed.
Can you please share with me the log lines around this one?
Also,
ipa user-show --all --raw host-enrollment
ipa trustconfig-show --all --raw
You can send them privately, if needed.
This was logged right before the whole set of instances got their errors.
I did also notice that the some of the same hostnames exist in older
Kerberos logs (hostnames will get repeated in our cloud env every now
and then), could this be the cause? A host with a previously used
hostname trying to enrol again? We have an automated process in place
that calls host-del to IPA when an instance is terminated to delete it
and its data from IPA, but maybe we should be clearing something from
Kerberos directly too?
If host is already enrolled, this would cause a problem to enroll unless
you pass --force to ipa-client-install. This is unrelated to the issues
you are seeing as you are getting the error in a different stage of the
installer.
--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
6:38 a.m.
New subject: ipa-client-install randomly fails with slapi_access_allowed does not allow WRITE to ipaProtectedOperation; write_keys!
For reference to @freeipa-users, since I very much don’t like open threads that moved to
private and were left unanswered.
Big thanks to Alexander for helping with debugging. It seems we are affected by
https://pagure.io/freeipa/issue/9228 <https://pagure.io/freeipa/issue/9228>;. To
confirm this: we don’t have much in terms of Kerberos logs on the IPA server that the host
initially enrolled to, but we can see "PAC issue: ipadb_get_principal failed” and
"TGT has been revoked” errors for this host in Kerberos logs on the second IPA in
this region, which I understand is a typical sign of this issue.
@Alexander - do you know if forcing —server to ipa-client-install would help as a
temporary work-around to force the installation to only use a specific server?
7:49 a.m.
New subject: ipa-client-install randomly fails with slapi_access_allowed does not allow WRITE to ipaProtectedOperation; write_keys!
Paulina Budzoń via FreeIPA-users wrote:
For reference to @freeipa-users, since I very much don’t like open
threads that moved to private and were left unanswered.
Big thanks to Alexander for helping with debugging. It seems we are
affected by https://pagure.io/freeipa/issue/9228. To confirm this: we
don’t have much in terms of Kerberos logs on the IPA server that the
host initially enrolled to, but we can see "PAC issue:
ipadb_get_principal failed” and "TGT has been revoked” errors for this
host in Kerberos logs on the second IPA in this region, which I
understand is a typical sign of this issue.
@Alexander - do you know if forcing —server to ipa-client-install would
help as a temporary work-around to force the installation to only use a
specific server?
I think it should help. The downside is that the resulting configuration
will be pinned to that one server. You'd need to go in afterward and
manually tweak the configuration on each client to use DNS discovery
again (at least krb5.conf and sssd.conf).
rob
532
days inactive
533
days old
freeipa-users@lists.fedorahosted.org
6 comments
4 participants
participants (4)
-
Alexander Bokovoy -
Paulina Budzon -
Paulina Budzoń
-
Rob Crittenden