I am running into a strange issue with a few user accounts where logging into the web interface gives them the error message "Login failed due to an unknown reason”.  It also prevents them from SSH’ing into IPA bound systems using passwords.  Pubkeys work fine (as long as it is manually added to the local accounts) and any services I have bound to it (Gitlab, Mattermost, Owncloud, etc) seem to work fine.  I ’think’ this is kerberos related since the only services that are using it is SSH and probably the IPA web interface.  Here is the apache error log for it:

[Thu Jan 13 09:15:38.688228 2022] [wsgi:error] [pid 579266:tid 139812542121728] [remote xx.xxx.xx.xxx:52162] ipa: INFO: 401 Unauthorized: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2598844948): TGT has been revoked

I ’think’ the message "TGT has been revoked” is due to the 401 error, since the user is not showing as being authorized to login.  However, this user is enabled and I have tried a number of things to try to fix it:

1. Disable/Re-enable account
2. Reset passwords
3. Kinit username (seems to get a ticket, but logins still do not work)
4. Run the account migration task (using the web gui)
5. Restart the IPA server and services
6. Re-initialize the IPA server from another master

Also, I can confirm that the passwords are correct since a failed password error message shows up differently and other services are using it correctly.  Going down the Kerberos path, here is the krb5kdc log file:

Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: ISSUE: authtime 1642094138, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: NEEDED_PREAUTH: testuser@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: ISSUE: authtime 1642094138, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, testuser@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](Error): PAC issue: PAC record claims domain SID different to local domain SID or any trusted domain SID: local [S-1-5-21-997841278-3584560916-1456654135], PAC [S-1-5-21-2108153867-2082035330-3701898995]
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ : handle_authdata (-1765328364)
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: HANDLE_AUTHDATA: authtime 1642094138, etypes {rep=UNSUPPORTED:(0)} testuser@EXAMPLE.COM for HTTP/ipa.example.com@EXAMPLE.COM, TGT has been revoked
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](Error): PAC issue: PAC record claims domain SID different to local domain SID or any trusted domain SID: local [S-1-5-21-997841278-3584560916-1456654135], PAC [S-1-5-21-2108153867-2082035330-3701898995]
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ : handle_authdata (-1765328364)
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: HANDLE_AUTHDATA: authtime 1642094138, etypes {rep=UNSUPPORTED:(0)} testuser@EXAMPLE.COM for HTTP/ipa.example.com@EXAMPLE.COM, TGT has been revoked

I only see two errors that might be related:

"PAC record claims domain SID different to local domain SID or any trusted domain SID”
"DEPRECATED:arcfour-hmac(23)”

However, those might just be red herrings or something else that is unrelated.

So far, there are only a small number of accounts that have this problem, but more seem to be popping up on a daily basis.  The only fix I have found is the nuclear option, where I completely remove the account and then add it back in with the same UID/GID, group memberships and policies.  After that it seems to work fine.  However, I would rather not want to do this to all accounts since that would be a logistical nightmare.

Are there any suggestions for either troubleshooting or fixing this problem with a lighter approach?  Is it possible to reset or regenerate the users kerberos authentication?

Thanks,

Dan West
Systems Administrator
Galois Inc.
http://galois.com