I am running into a strange issue with a few user accounts where
logging into the web interface gives them the error message "Login failed due to an
unknown reason”. It also prevents them from SSH’ing into IPA bound systems using
passwords. Pubkeys work fine (as long as it is manually added to the local accounts) and
any services I have bound to it (Gitlab, Mattermost, Owncloud, etc) seem to work fine. I
’think’ this is kerberos related since the only services that are using it is SSH and
probably the IPA web interface. Here is the apache error log for it:
[Thu Jan 13 09:15:38.688228 2022] [wsgi:error] [pid 579266:tid 139812542121728] [remote
xx.xxx.xx.xxx:52162] ipa: INFO: 401 Unauthorized: Major (851968): Unspecified GSS failure.
Minor code may provide more information, Minor (2598844948): TGT has been revoked
I ’think’ the message "TGT has been revoked” is due to the 401 error, since the user
is not showing as being authorized to login. However, this user is enabled and I have
tried a number of things to try to fix it:
1. Disable/Re-enable account
2. Reset passwords
3. Kinit username (seems to get a ticket, but logins still do not work)
4. Run the account migration task (using the web gui)
5. Restart the IPA server and services
6. Re-initialize the IPA server from another master
Also, I can confirm that the passwords are correct since a failed password error message
shows up differently and other services are using it correctly. Going down the Kerberos
path, here is the krb5kdc log file:
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](info): AS_REQ (7 etypes
{aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26),
aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25),
DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM
for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Additional pre-authentication required
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](info): AS_REQ (7 etypes
{aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26),
aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25),
DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: ISSUE: authtime 1642094138, etypes
{rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18),
ses=aes256-cts-hmac-sha1-96(18)}, WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38
ipa.example.com krb5kdc[579225](info): AS_REQ (7 etypes
{aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26),
aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25),
DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: NEEDED_PREAUTH: testuser(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Additional pre-authentication required
Jan 13 09:15:38
ipa.example.com krb5kdc[579225](info): closing down fd 12
Jan 13 09:15:38
ipa.example.com krb5kdc[579225](info): AS_REQ (7 etypes
{aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26),
aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25),
DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: ISSUE: authtime 1642094138, etypes
{rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18),
ses=aes256-cts-hmac-sha1-96(18)}, testuser(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
Jan 13 09:15:38
ipa.example.com krb5kdc[579225](info): closing down fd 12
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](Error): PAC issue: PAC record claims
domain SID different to local domain SID or any trusted domain SID: local
[S-1-5-21-997841278-3584560916-1456654135], PAC
[S-1-5-21-2108153867-2082035330-3701898995]
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](info): TGS_REQ : handle_authdata
(-1765328364)
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](info): TGS_REQ (7 etypes
{aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26),
aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25),
DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: HANDLE_AUTHDATA: authtime 1642094138, etypes
{rep=UNSUPPORTED:(0)} testuser(a)EXAMPLE.COM for HTTP/ipa.example.com(a)EXAMPLE.COM, TGT has
been revoked
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](Error): PAC issue: PAC record claims
domain SID different to local domain SID or any trusted domain SID: local
[S-1-5-21-997841278-3584560916-1456654135], PAC
[S-1-5-21-2108153867-2082035330-3701898995]
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](info): TGS_REQ : handle_authdata
(-1765328364)
Jan 13 09:15:38
ipa.example.com krb5kdc[579226](info): TGS_REQ (7 etypes
{aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26),
aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25),
DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: HANDLE_AUTHDATA: authtime 1642094138, etypes
{rep=UNSUPPORTED:(0)} testuser(a)EXAMPLE.COM for HTTP/ipa.example.com(a)EXAMPLE.COM, TGT has
been revoked
I only see two errors that might be related:
"PAC record claims domain SID different to local domain SID or any trusted domain
SID”
"DEPRECATED:arcfour-hmac(23)”
However, those might just be red herrings or something else that is unrelated.
No, that's exactly your problem. Your domain has SID
S-1-5-21-997841278-3584560916-1456654135 but user account has domain SID
S-1-5-21-2108153867-2082035330-3701898995. This is against a policy: all
accounts in the same domain should have the SID from this domain.
Perhaps these users were migrated from earlier IPA installation (test
deployment?) with 'ipa migrate-ds'?
A fix is either to re-create a user from scratch or replace
ipaNTSecurityIdentifier value in the user's LDAP entry with the right one.
A correct way to replace that would be a bit complicated -- you need to
remove both
objectclass: ipaNTUserAttrs
ipaNTSecurityIdentifier: <value>
at the same time because this object class has MUST on
ipaNTSecurityIdentifier.
If you have groups with SIDs from wrong domains, do the same for them
with these attributes:
objectclass: ipaNTGroupAttrs
ipaNTSecurityIdentifier: <value>
After removal of the values from all 'offending' entries, run
kinit admin
ipa config-mod --add-sids --enable-sid
This will force re-issue of SIDs to accounts where they are missing. It
might take quite a lot of time and 389-ds on that IPA master will be
restarted in a process.
So far, there are only a small number of accounts that have this problem, but more seem to
be popping up on a daily basis. The only fix I have found is the nuclear option, where I
completely remove the account and then add it back in with the same UID/GID, group
memberships and policies. After that it seems to work fine. However, I would rather not
want to do this to all accounts since that would be a logistical nightmare.
Are there any suggestions for either troubleshooting or fixing this problem with a lighter
approach? Is it possible to reset or regenerate the users kerberos authentication?
Thanks,
Dan West
Systems Administrator
Galois Inc.
http://galois.com
--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland