Yes, that was the issue.  I had migrated from an older FreeIPA instance to a newer one using "ipa migrate-ds" this past summer.  I’m not sure why it was just now causing problems, though.  Looking at the “ipaNTSecurityIdentifier” for all the accounts gave me a pretty good idea as to which users were affected by this, so I can surgically correct them.  Here is my quick one-off fix for it:

ldapmodify -Y GSSAPI <<EOF
dn: uid=testuser,cn=users,cn=accounts,dc=example,dc=com
changetype: modify
delete: ipaNTHash
-
delete: objectclass
objectclass: ipaNTUserAttrs
-
delete: ipaNTSecurityIdentifier

dn: cn=testuser,cn=groups,cn=accounts,dc=example,dc=com
changetype: modify
delete: objectclass
objectclass: ipaNTGroupAttrs
-
delete: ipaNTSecurityIdentifier
EOF

Thanks for the assist.


Dan West
Systems Administrator
Galois Inc.
http://galois.com




On Jan 13, 2022, at 10:05 AM, Alexander Bokovoy <abokovoy@redhat.com> wrote:

On to, 13 tammi 2022, Dan West via FreeIPA-users wrote:
I am running into a strange issue with a few user accounts where logging into the web interface gives them the error message "Login failed due to an unknown reason”. It also prevents them from SSH’ing into IPA bound systems using passwords.  Pubkeys work fine (as long as it is manually added to the local accounts) and any services I have bound to it (Gitlab, Mattermost, Owncloud, etc) seem to work fine.  I ’think’ this is kerberos related since the only services that are using it is SSH and probably the IPA web interface.  Here is the apache error log for it:

[Thu Jan 13 09:15:38.688228 2022] [wsgi:error] [pid 579266:tid 139812542121728] [remote xx.xxx.xx.xxx:52162] ipa: INFO: 401 Unauthorized: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2598844948): TGT has been revoked

I ’think’ the message "TGT has been revoked” is due to the 401 error, since the user is not showing as being authorized to login.  However, this user is enabled and I have tried a number of things to try to fix it:

1. Disable/Re-enable account
2. Reset passwords
3. Kinit username (seems to get a ticket, but logins still do not work)
4. Run the account migration task (using the web gui)
5. Restart the IPA server and services
6. Re-initialize the IPA server from another master

Also, I can confirm that the passwords are correct since a failed password error message shows up differently and other services are using it correctly.  Going down the Kerberos path, here is the krb5kdc log file:

Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: ISSUE: authtime 1642094138, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: NEEDED_PREAUTH: testuser@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: ISSUE: authtime 1642094138, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, testuser@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](Error): PAC issue: PAC record claims domain SID different to local domain SID or any trusted domain SID: local [S-1-5-21-997841278-3584560916-1456654135], PAC [S-1-5-21-2108153867-2082035330-3701898995]
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ : handle_authdata (-1765328364)
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: HANDLE_AUTHDATA: authtime 1642094138, etypes {rep=UNSUPPORTED:(0)} testuser@EXAMPLE.COM for HTTP/ipa.example.com@EXAMPLE.COM, TGT has been revoked
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](Error): PAC issue: PAC record claims domain SID different to local domain SID or any trusted domain SID: local [S-1-5-21-997841278-3584560916-1456654135], PAC [S-1-5-21-2108153867-2082035330-3701898995]
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ : handle_authdata (-1765328364)
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: HANDLE_AUTHDATA: authtime 1642094138, etypes {rep=UNSUPPORTED:(0)} testuser@EXAMPLE.COM for HTTP/ipa.example.com@EXAMPLE.COM, TGT has been revoked

I only see two errors that might be related:

"PAC record claims domain SID different to local domain SID or any trusted domain SID”
"DEPRECATED:arcfour-hmac(23)”

However, those might just be red herrings or something else that is unrelated.

No, that's exactly your problem. Your domain has SID
S-1-5-21-997841278-3584560916-1456654135 but user account has domain SID
S-1-5-21-2108153867-2082035330-3701898995. This is against a policy: all
accounts in the same domain should have the SID from this domain.

Perhaps these users were migrated from earlier IPA installation (test
deployment?) with 'ipa migrate-ds'?

A fix is either to re-create a user from scratch or replace
ipaNTSecurityIdentifier value in the user's LDAP entry with the right one.

A correct way to replace that would be a bit complicated -- you need to
remove both

objectclass: ipaNTUserAttrs
ipaNTSecurityIdentifier: <value>

at the same time because this object class has MUST on
ipaNTSecurityIdentifier.

If you have groups with SIDs from wrong domains, do the same for them
with these attributes:

objectclass: ipaNTGroupAttrs
ipaNTSecurityIdentifier: <value>

After removal of the values from all 'offending' entries, run

kinit admin
ipa config-mod --add-sids --enable-sid

This will force re-issue of SIDs to accounts where they are missing. It
might take quite a lot of time and 389-ds on that IPA master will be
restarted in a process.


So far, there are only a small number of accounts that have this problem, but more seem to be popping up on a daily basis.  The only fix I have found is the nuclear option, where I completely remove the account and then add it back in with the same UID/GID, group memberships and policies.  After that it seems to work fine. However, I would rather not want to do this to all accounts since that would be a logistical nightmare.

Are there any suggestions for either troubleshooting or fixing this problem with a lighter approach?  Is it possible to reset or regenerate the users kerberos authentication?

Thanks,

Dan West
Systems Administrator
Galois Inc.
http://galois.com








-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland