Do you have a keytab named /var/local/keytabs/0.keytab ?
It looks like gss-proxy attempts to acquire creds but uses ost/client.zz.example.com@ZZ.EXAMPLE.COM to try to obtain a TGT, but AD KDCs are picky and do not allow to use the SPN as the initiator they want to see a request from client$@AD.EXAMPLE.COM instead.
So gss-proxy returns an error and then rpc.gssd falls back and tries to directly obtain a credential and succeeds (ie the creds are not obtained through gss-proxy in this case).
I do not know if this should be in any way a problem because you are not trying to use impersonation, you are trying to use actual keytabs for users. So you should try to walk in amount point as a user and post the gss-proxy/rpc.gssd errors when that happen.
Root is probably squashed and is generally not a good user for debugging as rpc.gssd falls back trying to use machine credentials for root.
Simo.
On Thu, 2021-06-24 at 12:09 -0400, John Bazik wrote:
Sure, here's the klist output:
Ticket cache: FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM Default principal: CLIENT$@AD.EXAMPLE.COM
Valid starting Expires Service principal 06/24/2021 11:53:49 06/24/2021 21:53:49 krbtgt/AD.EXAMPLE.COM@AD.EXAMPLE.COM renew until 07/01/2021 11:53:49 06/24/2021 11:53:49 06/24/2021 21:53:49 nfs/nfs.example.com@AD.EXAMPLE.COM renew until 07/01/2021 11:53:49
And here is larger snippet from syslog with gssproxy debug = 2:
Jun 24 11:54:08 client rpc.gssd[6512]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt2fe) Jun 24 11:54:08 client rpc.gssd[6512]: krb5_use_machine_creds: uid 0 tgtname (null) Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [ ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [ ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure. Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [ ] } context_handle: <Null> output_token: <Null> ) Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [ ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [ ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure. Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [ ] } context_handle: <Null> output_token: <Null> ) Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: doing error downcall
John
On 6/24/21 10:19 AM, Simo Sorce wrote:
Ok two points, can you raise the debug level of gssproy and see what it prints ?
Also can you klist the contents of /tmp/krb5ccmachine_AD.EXAMPLE.COM ?
Thanks, Simo.
On Wed, 2021-06-23 at 23:46 -0400, John Bazik wrote:
I've recently switched from using k5start to gssproxy to allow my users to access NFSv4 mounts with sec=krb5, using keytabs I manage for them. I have just one service configured in gssproxy:
[service/nfs-client] mechs = krb5 cred_store = keytab:/etc/krb5.keytab cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U cred_store = client_keytab:/var/local/keytabs/%u.keytab cred_usage = initiate allow_any_uid = yes trusted = yes euid = 0
I thought everything was working great, but now I find that I can't mount remote filesystems when gssproxy is running. If I stop gssproxy, mount works. If I change sec=krb5 to sec=sys, mount works. It seems clear that gssproxy is preventing mount from working. When I run mount -a, I get errors like this:
mount.nfs: access denied by server while mounting [...]
When I add -vvv to rpc.gssd, this is what I see in syslog (anonymized):
rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 rpc.gssd[6512]: creating tcp client for server nfs.example.com rpc.gssd[6512]: DEBUG: port already set to 2049 rpc.gssd[6512]: creating context with server nfs@nfs.example.com rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com rpc.gssd[6512]: doing error downcall
I'm running version 0.8.0, as distributed with Debian Buster (I worked around the systemd ordering cycle bug in that version by using the upstream unit file). The fileserver is run by a different group and kerberos is AD.
Googling for answers, I found others describe similar problems, but no solutions that make sense to me. Help!
John _______________________________________________ gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure