Full client log below. It turns out that what I was suspecting (issue with
regards to the cross-realm TGT) was a step in the right direction. It turns
out that my story is a bit more complicated indeed.
My IPA server has a direct trust relationship with another AD realm/domain
(say "d2.domain" which has a bidirectional trust relationship with
"ad.domain"). So far so good, however I learned this morning that our
Windows guys replaced some outdated "ad2.domain" DCs with brand new ones
some days ago and, for some obscure reason, the network security stuff had
not been updated on my side. Consequently, my IPA server was unable to
speak with the new DCs deserving "ad2.domain thus was also unable to
process anything related to "ad.domain", including cross-realm TGTs
implying that later AD realm/domain. Our network admins should fix that
next week and everyting should come back in good working order. In the
meantime not bug or a software configuration or a DNS issue. Re-doing the
trust with "ad2.domain" would have directly pointed the problem
("Communication error").
Thank you Rob for taking the time to investigate my concern.
==== [Start of /var/log/ipaclient-install.log (w/ sensitive info redacted)
] ====
2021-07-23T14:30:40Z DEBUG Logging to /var/log/ipaclient-install.log
2021-07-23T14:30:40Z DEBUG ipa-client-install was invoked with arguments []
and options: {'no_dns_sshfp': False, 'force': True, 'verbose':
False,
'ip_addresses': None, 'configure_firefox': False, 'realm_name':
'IPA.DOMAIN', 'force_ntpd': False, 'on_master': False,
'no_nisdomain':
False, 'ssh_trust_dns': False, 'principal': 'ul-val-s-enroll',
'keytab':
None, 'no_ntp': True, 'domain_name': ipa.domain',
'request_cert': False,
'fixed_primary': False, 'no_ac': False, 'no_sudo': False,
'ca_cert_files':
None, 'all_ip_addresses': False, 'kinit_attempts': None,
'ntp_servers':
None, 'enable_dns_updates': False, 'no_sshd': False, 'no_sssd':
False,
'no_krb5_offline_passwords': False, 'servers':
['idmsrv01.ad.domain'],
'no_ssh': False, 'force_join': True, 'firefox_dir': None,
'unattended':
False, 'quiet': False, 'nisdomain': None, 'prompt_password':
False,
'host_name': None, 'permit': False, 'automount_location': None,
'preserve_sssd': False, 'mkhomedir': True, 'log_file': None,
'uninstall':
False}
2021-07-23T14:30:40Z DEBUG IPA version 4.6.8-5.el7.centos.6
2021-07-23T14:30:40Z DEBUG Loading Index file from
'/var/lib/ipa-client/sysrestore/sysrestore.index'
2021-07-23T14:30:40Z DEBUG Starting external process
2021-07-23T14:30:40Z DEBUG args=/usr/sbin/selinuxenabled
2021-07-23T14:30:40Z DEBUG Process finished, return code=0
2021-07-23T14:30:40Z DEBUG stdout=
2021-07-23T14:30:40Z DEBUG stderr=
2021-07-23T14:30:40Z DEBUG [IPA Discovery]
2021-07-23T14:30:40Z DEBUG Starting IPA discovery with domain=ipa.domain,
servers=['idmsrv01.ad.domain'], hostname=clientvm01.ad.domain
2021-07-23T14:30:40Z DEBUG Server and domain forced
2021-07-23T14:30:40Z DEBUG [Kerberos realm search]
2021-07-23T14:30:40Z DEBUG Kerberos realm forced
2021-07-23T14:30:40Z DEBUG [LDAP server check]
2021-07-23T14:30:40Z DEBUG Verifying that idmsrv01.ad.domain (realm
IPA.DOMAIN) is an IPA server
2021-07-23T14:30:40Z DEBUG Init LDAP connection to:
ldap://idmsrv01.ad.domain:389
2021-07-23T14:30:40Z DEBUG Search LDAP server for IPA base DN
2021-07-23T14:30:40Z DEBUG Check if naming context 'dc=ipa,dc=domain' is
for IPA
2021-07-23T14:30:40Z DEBUG Naming context 'dc=ipa,dc=domain' is a valid IPA
context
2021-07-23T14:30:40Z DEBUG Search for (objectClass=krbRealmContainer) in
dc=ipa,dc=domain (sub)
2021-07-23T14:30:40Z DEBUG Found: cn=IPA.DOMAIN,cn=kerberos,dc=ipa,dc=domain
2021-07-23T14:30:40Z DEBUG Discovery result: Success;
server=idmsrv01.ad.domain, domain=ipa.domain, kdc=idmsrv01.ad.domain,
basedn=dc=ipa,dc=domain
2021-07-23T14:30:40Z DEBUG Validated servers: idmsrv01.ad.domain
2021-07-23T14:30:40Z DEBUG will use discovered domain: ipa.domain
2021-07-23T14:30:40Z DEBUG Using servers from command line, disabling DNS
discovery
2021-07-23T14:30:40Z DEBUG will use provided server: idmsrv01.ad.domain
2021-07-23T14:30:40Z INFO Autodiscovery of servers for failover cannot work
with this configuration.
2021-07-23T14:30:40Z INFO If you proceed with the installation, services
will be configured to always access the discovered server for all
operations and will not fail over to other servers in case of failure.
2021-07-23T14:30:41Z DEBUG will use discovered realm: IPA.DOMAIN
2021-07-23T14:30:41Z DEBUG will use discovered basedn: dc=ipa,dc=domain
2021-07-23T14:30:41Z INFO Client hostname: clientvm01.ad.domain
2021-07-23T14:30:41Z DEBUG Hostname source: Machine's FQDN
2021-07-23T14:30:41Z INFO Realm: IPA.DOMAIN
2021-07-23T14:30:41Z DEBUG Realm source: Discovered from LDAP DNS records
in idmsrv01.ad.domain
2021-07-23T14:30:41Z INFO DNS Domain: ipa.domain
2021-07-23T14:30:41Z DEBUG DNS Domain source: Forced
2021-07-23T14:30:41Z INFO IPA Server: idmsrv01.ad.domain
2021-07-23T14:30:41Z DEBUG IPA Server source: Provided as option
2021-07-23T14:30:41Z INFO BaseDN: dc=ipa,dc=domain
2021-07-23T14:30:41Z DEBUG BaseDN source: From IPA server
ldap://idmsrv01.ad.domain:389
2021-07-23T14:30:48Z DEBUG Loading Index file from
'/var/lib/ipa-client/sysrestore/sysrestore.index'
2021-07-23T14:30:48Z DEBUG Loading StateFile from
'/var/lib/ipa-client/sysrestore/sysrestore.state'
2021-07-23T14:30:48Z DEBUG Starting external process
2021-07-23T14:30:48Z DEBUG args=/usr/sbin/ipa-rmkeytab -k /etc/krb5.keytab
-r IPA.DOMAIN
2021-07-23T14:30:48Z DEBUG Process finished, return code=5
2021-07-23T14:30:48Z DEBUG stdout=
2021-07-23T14:30:48Z DEBUG stderr=realm not found
2021-07-23T14:30:48Z INFO Skipping synchronizing time with NTP server.
2021-07-23T14:30:48Z DEBUG Starting external process
2021-07-23T14:30:48Z DEBUG args=/usr/bin/keyctl get_persistent @s 0
2021-07-23T14:30:48Z DEBUG Process finished, return code=0
2021-07-23T14:30:48Z DEBUG stdout=96233181
2021-07-23T14:30:48Z DEBUG stderr=
2021-07-23T14:30:48Z DEBUG Enabling persistent keyring CCACHE
2021-07-23T14:30:48Z DEBUG Writing Kerberos configuration to /tmp/tmpO564dk:
2021-07-23T14:30:48Z DEBUG #File modified by ipa-client-install
includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/
[libdefaults]
default_realm = IPA.DOMAIN
dns_lookup_realm = false
dns_lookup_kdc = false
rdns = false
dns_canonicalize_hostname = false
ticket_lifetime = 24h
forwardable = true
udp_preference_limit = 0
default_ccache_name = KEYRING:persistent:%{uid}
[realms]
IPA.DOMAIN = {
kdc = idmsrv01.ad.domain:88
master_kdc = idmsrv01.ad.domain:88
admin_server = idmsrv01.ad.domain:749
kpasswd_server = idmsrv01.ad.domain:464
default_domain = ipa.domain
pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
}
[domain_realm]
.ipa.domain = IPA.DOMAIN
ipa.domain = IPA.DOMAIN
clientvm01.ad.domain = IPA.DOMAIN
.ad.domain = IPA.DOMAIN
ad.domain = IPA.DOMAIN
2021-07-23T14:30:52Z DEBUG Initializing principal
enrollmentaccount(a)IPA.DOMAIN using password
2021-07-23T14:30:52Z DEBUG Starting external process
2021-07-23T14:30:52Z DEBUG args=/usr/bin/kinit enrollmentaccount(a)IPA.DOMAIN
-c /tmp/krbccQTje0m/ccache
2021-07-23T14:30:52Z DEBUG Process finished, return code=0
2021-07-23T14:30:52Z DEBUG stdout=Password for enrollmentaccount(a)IPA.DOMAIN:
2021-07-23T14:30:52Z DEBUG stderr=
2021-07-23T14:30:52Z DEBUG trying to retrieve CA cert via LDAP from
idmsrv01.ad.domain
2021-07-23T14:30:53Z DEBUG get_ca_certs_from_ldap() error: Insufficient
access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Server krbtgt/AD.DOMAIN(a)IPA.DOMAIN
not found in Kerberos database)
2021-07-23T14:30:53Z DEBUG Insufficient access: SASL(-1): generic failure:
GSSAPI Error: Unspecified GSS failure. Minor code may provide more
information (Server krbtgt/AD.DOMAIN(a)IPA.DOMAIN not found in Kerberos
database)
2021-07-23T14:30:57Z WARNING Downloading the CA certificate via HTTP, this
is INSECURE
2021-07-23T14:30:57Z DEBUG trying to retrieve CA cert via HTTP from
http://idmsrv01.ad.domain/ipa/config/ca.crt
2021-07-23T14:30:57Z DEBUG Starting external process
2021-07-23T14:30:57Z DEBUG args=/usr/bin/curl -o -
http://idmsrv01.ad.domain/ipa/config/ca.crt
2021-07-23T14:30:57Z DEBUG Process finished, return code=0
2021-07-23T14:30:57Z DEBUG stdout=-----BEGIN CERTIFICATE-----
**[PEM CERTIFICATES REDACTED]**
-----END CERTIFICATE-----
2021-07-23T14:30:57Z DEBUG stderr= % Total % Received % Xferd Average
Speed Time Time Time Current
Dload Upload Total Spent Left
Speed
^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0^M100 7754 100 7754 0 0 1056k 0 --:--:-- --:--:--
--:--:-- 1081k
2021-07-23T14:30:57Z INFO Successfully retrieved CA cert
Subject: CN=AAA Certificate Services,O=Comodo CA
Limited,L=Salford,ST=Greater Manchester,C=GB
Issuer: CN=AAA Certificate Services,O=Comodo CA
Limited,L=Salford,ST=Greater Manchester,C=GB
Valid From: 2004-01-01 00:00:00
Valid Until: 2028-12-31 23:59:59
Subject: CN=USERTrust RSA Certification Authority,O=The USERTRUST
Network,L=Jersey City,ST=New Jersey,C=US
Issuer: CN=AAA Certificate Services,O=Comodo CA
Limited,L=Salford,ST=Greater Manchester,C=GB
Valid From: 2019-03-12 00:00:00
Valid Until: 2028-12-31 23:59:59
Subject: CN=USERTrust RSA Certification Authority,O=The USERTRUST
Network,L=Jersey City,ST=New Jersey,C=US
Issuer: CN=USERTrust RSA Certification Authority,O=The USERTRUST
Network,L=Jersey City,ST=New Jersey,C=US
Valid From: 2010-02-01 00:00:00
Valid Until: 2038-01-18 23:59:59
Subject: CN=Sectigo RSA Organization Validation Secure Server
CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB
Issuer: CN=USERTrust RSA Certification Authority,O=The USERTRUST
Network,L=Jersey City,ST=New Jersey,C=US
Valid From: 2018-11-02 00:00:00
Valid Until: 2030-12-31 23:59:59
2021-07-23T14:30:57Z DEBUG Starting external process
2021-07-23T14:30:57Z DEBUG args=/usr/sbin/ipa-join -s idmsrv01.ad.domain -b
dc=ipa,dc=domain -h clientvm01.ad.domain -f
2021-07-23T14:30:57Z DEBUG Process finished, return code=17
2021-07-23T14:30:57Z DEBUG stdout=
2021-07-23T14:30:57Z DEBUG stderr=HTTP response code is 401, not 200
2021-07-23T14:30:57Z ERROR Joining realm failed: HTTP response code is 401,
not 200
2021-07-23T14:30:57Z INFO Use ipa-getkeytab to obtain a host principal for
this server.
2021-07-23T14:30:57Z DEBUG Starting external process
2021-07-23T14:30:57Z DEBUG args=/usr/bin/kdestroy
2021-07-23T14:30:57Z DEBUG Process finished, return code=0
2021-07-23T14:30:57Z DEBUG stdout=
2021-07-23T14:30:57Z DEBUG stderr=
2021-07-23T14:30:57Z DEBUG Initializing principal
host/clientvm01.ad.domain(a)IPA.DOMAIN using keytab /etc/krb5.keytab
2021-07-23T14:30:57Z DEBUG using ccache /etc/ipa/.dns_ccache
2021-07-23T14:30:57Z INFO Please make sure the following ports are opened
in the firewall settings:
TCP: 80, 88, 389
UDP: 88 (at least one of TCP/UDP ports 88 has to be open)
Also note that following ports are necessary for ipa-client working
properly after enrollment:
TCP: 464
UDP: 464, 123 (if NTP enabled)
2021-07-23T14:30:57Z ERROR Failed to obtain host TGT: Major (851968):
Unspecified GSS failure. Minor code may provide more information, Minor
(2529639107): No credentials cache found
2021-07-23T14:30:57Z WARNING Installation failed. Force set so not rolling
back changes.
2021-07-23T14:30:57Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in
execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line
319, in run
return cfgr.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
360, in run
return self.execute()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
386, in execute
for rval in self._executor():
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
431, in __runner
exc_handler(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
460, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
421, in __runner
step()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line
81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line
59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
655, in _configure
next(executor)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
431, in __runner
exc_handler(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
460, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
518, in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
515, in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
421, in __runner
step()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line
81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line
59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line
65, in _install
for unused in self._installer(self.parent):
File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line
3670, in main
install(self)
File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line
2391, in install
_install(options)
File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line
2644, in _install
raise ScriptError(rval=CLIENT_INSTALL_ERROR)
2021-07-23T14:30:57Z DEBUG The ipa-client-install command failed,
exception: ScriptError:
2021-07-23T14:30:57Z ERROR The ipa-client-install command failed. See
/var/log/ipaclient-install.log for more information
==== [End of of /var/log/ipaclient-install.log] ====
Le jeu. 22 juil. 2021 à 18:25, Rob Crittenden <rcritten(a)redhat.com> a
écrit :
Adrien Dessemond via FreeIPA-users wrote:
> Hi,
>
> I do have an issue with my latest Linux (CentOS 7) virtual machines
> (names have been redacted):
>
> 1. They start their life in an AD domain (Windows 2019 DCs) and
> everything is working as expected (for the rest of the explanation, I
> will use "AD.DOMAIN" for this realm)
> 2. I have an IPA Server, which, despite having a FQDN ending in
> "ad.domain" is enrolled in its own Kerberos domain (IPA.DOMAIN) =>
> idmsrv01.ad.domain
> 2.On my client machine I do => realm leave
> 3. Then I try to make it rejoin my IPA domain/realm =>
>
> # export KRB5_TRACE=/dev/stdout
> # ipa-client-install --server idmsrv01.ad.domain --domain idm.domain
> --realm IPA.DOMAIN --principal enrollmentaccount --mkhomedir --no-ntp
> --force-join
>
> And here is the issue :
>
> (...)
> Using existing certificate '/etc/ipa/ca.crt'.
> Client hostname: myvm01.ad.domain
> Realm: IPA.DOMAIN
> DNS Domain: ipa.domain
> IPA Server: idmsrv01.ad.domain
> BaseDN: dc=ipa,dc=domain
>
> Skipping synchronizing time with NTP server.
> [19420] 1626829415.775243: ccselect can't find appropriate cache for
> server principal ldap/idmsrv01.ad.domain(a)AD.DOMAIN
> [19420] 1626829415.775244: Getting credentials
> enrollmentaccount(a)IPA.DOMAIN -> ldap/idmsrv01.ad.domain(a)AD.DOMAIN using
> ccache FILE:/tmp/krbccoz6hPK/ccache
> [19420] 1626829415.775245: Retrieving enrollmentaccount(a)IPA.DOMAIN ->
> ldap/idmsrv01.ad.domain(a)AD.DOMAIN from FILE:/tmp/krbccoz6hPK/ccache with
> result: -1765328243/Matching credential not found (filename:
> /tmp/krbccoz6hPK/ccache)
> [19420] 1626829415.775246: Retrieving enrollmentaccount(a)IPA.DOMAIN ->
> krbtgt/AD.DOMAIN(a)AD.DOMAIN from FILE:/tmp/krbccoz6hPK/ccache with
> result: -1765328243/Matching credential not found (filename:
> /tmp/krbccoz6hPK/ccache)
> [19420] 1626829415.775247: Retrieving enrollmentaccount(a)IPA.DOMAIN ->
> krbtgt/IPA.DOMAIN(a)IPA.DOMAIN from FILE:/tmp/krbccoz6hPK/ccache with
> result: 0/Success
> [19420] 1626829415.775248: Starting with TGT for client realm:
> enrollmentaccount(a)IPA.DOMAIN -> krbtgt/IPA.DOMAIN(a)IPA.DOMAIN
> [19420] 1626829415.775249: Retrieving enrollmentaccount(a)IPA.DOMAIN ->
> krbtgt/AD.DOMAIN(a)AD.DOMAIN from FILE:/tmp/krbccoz6hPK/ccache with
> result: -1765328243/Matching credential not found (filename:
> /tmp/krbccoz6hPK/ccache)
> [19420] 1626829415.775250: Requesting TGT krbtgt/AD.DOMAIN(a)IPA.DOMAIN
> using TGT krbtgt/IPA.DOMAIN(a)IPA.DOMAIN
> [19420] 1626829415.775251: Generated subkey for TGS request:
aes256-cts/3835
> [19420] 1626829415.775252: etypes requested in TGS request: aes256-cts,
> aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac,
> camellia128-cts, camellia256-cts
> [19420] 1626829415.775254: Encoding request body and padata into FAST
> request
> [19420] 1626829415.775255: Sending request (1565 bytes) to IPA.DOMAIN
> [19420] 1626829415.775256: Resolving hostname idmsrv01.ad.domain
> [19420] 1626829415.775257: Initiating TCP connection to stream
> 10.xxx.xxx.xxx:88
> [19420] 1626829415.775258: Sending TCP request to stream
10.xxx.xxx.xxx:88
> [19420] 1626829415.775259: Received answer (482 bytes) from stream
> 10.xxx.xxx.xxx:88
> [19420] 1626829415.775260: Terminating TCP connection to stream
> 10.xxx.xxx.xxx:88
> [19420] 1626829415.775261: Response was from master KDC
> [19420] 1626829415.775262: Decoding FAST response
> [19420] 1626829415.775263: TGS request result: -1765328377/Server
> krbtgt/AD.DOMAIN(a)IPA.DOMAIN not found in Kerberos database
> Unable to download CA cert from LDAP but found preexisting cert, using
it.
>
> Joining realm failed: HTTP response code is 401, not 200
>
> Installation failed. Rolling back changes.
> (...)
>
> So basically my interpretation is : the IPA client is trying to
> initiate a Kerberos authentication which fails and being unable to do
> that it is unable to get a valid credential for the IPA server which in
> turn results in a 401 error when doing an XML RPC call on it. The
> account used for enrollment is not locked, not expired and is having the
> correct roles especially the enrollment capability.
>
> Understanding "TGS request result: -1765328377/Server
> krbtgt/AD.DOMAIN(a)IPA.DOMAIN" seems the key to my issue but I have no
> clue on what to do at this point.
>
> - My IPA server has no direct trust to AD.DOMAIN.
> - Client and server are both using IPA 4.6.8
> - Server certificates seem correct and not expired.
>
> Any idea on what is going on / what to look at next?
This is not a recommended DNS configuration [1]. Having the IPA domain
name different from the realm is one thing, but it points to an active
AD realm. Anything that does DNS SRV discovery is going to find the
wrong KDC which is what appears to be happening here.
Can you provide the full client install log? That may provide more
details on what is happening.
rob
[1]
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/...
--
Adrien Dessemond / VE2AKS