Do you have a keytab named /var/local/keytabs/0.keytab ?

It looks like gss-proxy attempts to acquire creds but uses ost/client.zz.example.com@ZZ.EXAMPLE.COM to try to obtain a TGT, but AD KDCs are picky and do not allow to use the SPN as the initiator they want to see a request from client$@AD.EXAMPLE.COM instead.

So gss-proxy returns an error and then rpc.gssd falls back and tries to directly obtain a credential and succeeds (ie the creds are not obtained through gss-proxy in this case).

I do not know if this should be in any way a problem because you are not trying to use impersonation, you are trying to use actual keytabs for users. So you should try to walk in amount point as a user and post the gss-proxy/rpc.gssd errors when that happen.

Root is probably squashed and is generally not a good user for debugging as rpc.gssd falls back trying to use machine credentials for root.

Simo.

On Thu, 2021-06-24 at 12:09 -0400, John Bazik wrote:

Sure, here's the klist output:

Ticket cache: FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM Default principal: CLIENT$@AD.EXAMPLE.COM

Valid starting Expires Service principal 06/24/2021 11:53:49 06/24/2021 21:53:49 krbtgt/AD.EXAMPLE.COM@AD.EXAMPLE.COM renew until 07/01/2021 11:53:49 06/24/2021 11:53:49 06/24/2021 21:53:49 nfs/nfs.example.com@AD.EXAMPLE.COM renew until 07/01/2021 11:53:49

And here is larger snippet from syslog with gssproxy debug = 2:

Jun 24 11:54:08 client rpc.gssd[6512]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt2fe) Jun 24 11:54:08 client rpc.gssd[6512]: krb5_use_machine_creds: uid 0 tgtname (null) Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [ ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [ ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure. Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [ ] } context_handle: <Null> output_token: <Null> ) Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [ ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [ ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure. Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [ ] } context_handle: <Null> output_token: <Null> ) Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: doing error downcall

John

On 6/24/21 10:19 AM, Simo Sorce wrote:

...

Ok two points, can you raise the debug level of gssproy and see what it prints ?

Also can you klist the contents of /tmp/krb5ccmachine_AD.EXAMPLE.COM ?

Thanks, Simo.

On Wed, 2021-06-23 at 23:46 -0400, John Bazik wrote:

...

I've recently switched from using k5start to gssproxy to allow my users to access NFSv4 mounts with sec=krb5, using keytabs I manage for them. I have just one service configured in gssproxy:

[service/nfs-client] mechs = krb5 cred_store = keytab:/etc/krb5.keytab cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U cred_store = client_keytab:/var/local/keytabs/%u.keytab cred_usage = initiate allow_any_uid = yes trusted = yes euid = 0

I thought everything was working great, but now I find that I can't mount remote filesystems when gssproxy is running. If I stop gssproxy, mount works. If I change sec=krb5 to sec=sys, mount works. It seems clear that gssproxy is preventing mount from working. When I run mount -a, I get errors like this:

mount.nfs: access denied by server while mounting [...]

When I add -vvv to rpc.gssd, this is what I see in syslog (anonymized):

rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com
rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com'
rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com'
rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM'
rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM'
rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464
rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464
rpc.gssd[6512]: creating tcp client for server nfs.example.com
rpc.gssd[6512]: DEBUG: port already set to 2049
rpc.gssd[6512]: creating context with server nfs@nfs.example.com
rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com
rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com
rpc.gssd[6512]: doing error downcall

I'm running version 0.8.0, as distributed with Debian Buster (I worked around the systemd ordering cycle bug in that version by using the upstream unit file). The fileserver is run by a different group and kerberos is AD.

Googling for answers, I found others describe similar problems, but no solutions that make sense to me. Help!

John _______________________________________________ gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

So think more about your config,given what you want to achieve you should probably removed euid = 0 and trusted = yes

rpc.gssd changes the uid used for the process, and this tells gss-proxy what user (and hence) what keytab to use), however euid = 0 is forcing to accept only connections from root, and trusted is saying root can impersonated anyone, but you do not want that.

I do not know if this will help the actual problem, but it is worth getting anything cleared so we do not chase ghosts.

Simo.

On Thu, 2021-06-24 at 13:58 -0400, John Bazik wrote:

So, no, there is no 0.keytab, and there is no AD account for root, so it would make no sense to have one.

We normally rely on root using machine credentials to do mounts. I don't understand why that stops working when gssproxy is running.

John

On 6/24/21 12:41 PM, Simo Sorce wrote:

...

Do you have a keytab named /var/local/keytabs/0.keytab ?

It looks like gss-proxy attempts to acquire creds but uses ost/client.zz.example.com@ZZ.EXAMPLE.COM to try to obtain a TGT, but AD KDCs are picky and do not allow to use the SPN as the initiator they want to see a request from client$@AD.EXAMPLE.COM instead.

So gss-proxy returns an error and then rpc.gssd falls back and tries to directly obtain a credential and succeeds (ie the creds are not obtained through gss-proxy in this case).

I do not know if this should be in any way a problem because you are not trying to use impersonation, you are trying to use actual keytabs for users. So you should try to walk in amount point as a user and post the gss-proxy/rpc.gssd errors when that happen.

Root is probably squashed and is generally not a good user for debugging as rpc.gssd falls back trying to use machine credentials for root.

Simo.

On Thu, 2021-06-24 at 12:09 -0400, John Bazik wrote:

...

Sure, here's the klist output:

Ticket cache: FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM Default principal: CLIENT$@AD.EXAMPLE.COM

Valid starting Expires Service principal 06/24/2021 11:53:49 06/24/2021 21:53:49 krbtgt/AD.EXAMPLE.COM@AD.EXAMPLE.COM renew until 07/01/2021 11:53:49 06/24/2021 11:53:49 06/24/2021 21:53:49 nfs/nfs.example.com@AD.EXAMPLE.COM renew until 07/01/2021 11:53:49

And here is larger snippet from syslog with gssproxy debug = 2:

Jun 24 11:54:08 client rpc.gssd[6512]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt2fe) Jun 24 11:54:08 client rpc.gssd[6512]: krb5_use_machine_creds: uid 0 tgtname (null) Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [ ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [ ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure. Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [ ] } context_handle: <Null> output_token: <Null> ) Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [ ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [ ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration Jun 24 11:54:08 client gssproxy[32163]: GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure. Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [ ] } context_handle: <Null> output_token: <Null> ) Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: doing error downcall

John

On 6/24/21 10:19 AM, Simo Sorce wrote:

...

Ok two points, can you raise the debug level of gssproy and see what it prints ?

Also can you klist the contents of /tmp/krb5ccmachine_AD.EXAMPLE.COM ?

Thanks, Simo.

On Wed, 2021-06-23 at 23:46 -0400, John Bazik wrote:

...

I've recently switched from using k5start to gssproxy to allow my users to access NFSv4 mounts with sec=krb5, using keytabs I manage for them. I have just one service configured in gssproxy:

[service/nfs-client] mechs = krb5 cred_store = keytab:/etc/krb5.keytab cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U cred_store = client_keytab:/var/local/keytabs/%u.keytab cred_usage = initiate allow_any_uid = yes trusted = yes euid = 0

I thought everything was working great, but now I find that I can't mount remote filesystems when gssproxy is running. If I stop gssproxy, mount works. If I change sec=krb5 to sec=sys, mount works. It seems clear that gssproxy is preventing mount from working. When I run mount -a, I get errors like this:

 mount.nfs: access denied by server while mounting [...]

When I add -vvv to rpc.gssd, this is what I see in syslog (anonymized):

 rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com
 rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com'
 rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com'
 rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM'
 rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM'
 rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464
 rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464
 rpc.gssd[6512]: creating tcp client for server nfs.example.com
 rpc.gssd[6512]: DEBUG: port already set to 2049
 rpc.gssd[6512]: creating context with server nfs@nfs.example.com
 rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
 rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com
 rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com
 rpc.gssd[6512]: doing error downcall

I'm running version 0.8.0, as distributed with Debian Buster (I worked around the systemd ordering cycle bug in that version by using the upstream unit file). The fileserver is run by a different group and kerberos is AD.

Googling for answers, I found others describe similar problems, but no solutions that make sense to me. Help!

John _______________________________________________ gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

I should add that user authentication with NFS works fine with my original config. It is only root executing mount that stopped working.

John

On 6/24/21 3:03 PM, John Bazik wrote:

I wondered about euid, but gssproxy.conf(5) says:

The "euid" parameter is imperative, any section without it will be discarded.

And when I delete those two lines from the configuration, gssproxy does not run:

Jun 24 14:50:32 client gssproxy[596]: [2021/06/24 18:50:32]: Debug Enabled (level: 2)   Jun 24 14:50:32 client gssproxy[596]: [2021/06/24 18:50:32]: Config file(s) not found!   Jun 24 14:50:32 client gssproxy: Option 'euid' is missing from [service/nfs-client].   Jun 24 14:50:32 client gssproxy: Error reading configuration 22: Invalid argument   Jun 24 14:50:32 client systemd[1]: gssproxy.service: Control process exited, code=exited, status=1/FAILURE   Jun 24 14:50:32 client systemd[1]: gssproxy.service: Failed with result 'exit-code'.

This is how I test that it's working (when gssproxy is running):

# su -c 'ls /home/jbazik' jbazik

So, you are right, I'm not interested in having root impersonate users.  I just want users to get keytab-authenticated.

John

On 6/24/21 2:21 PM, Simo Sorce wrote:

...

So think more about your config,given what you want to achieve you should probably removed euid = 0 and trusted = yes

rpc.gssd changes the uid used for the process, and this tells gss-proxy what user (and hence) what keytab to use), however euid = 0 is forcing to accept only connections from root, and trusted is saying root can impersonated anyone, but you do not want that.

I do not know if this will help the actual problem, but it is worth getting anything cleared so we do not chase ghosts.

Simo.

On Thu, 2021-06-24 at 13:58 -0400, John Bazik wrote:

...

So, no, there is no 0.keytab, and there is no AD account for root, so it would make no sense to have one.

We normally rely on root using machine credentials to do mounts.  I don't understand why that stops working when gssproxy is running.

John

On 6/24/21 12:41 PM, Simo Sorce wrote:

...

Do you have a keytab named /var/local/keytabs/0.keytab ?

It looks like gss-proxy attempts to acquire creds but uses ost/client.zz.example.com@ZZ.EXAMPLE.COM to try to obtain a TGT, but AD KDCs are picky and do not allow to use the SPN as the initiator they want to see a request from client$@AD.EXAMPLE.COM instead.

So gss-proxy returns an error and then rpc.gssd falls back and tries to directly obtain a credential and succeeds (ie the creds are not obtained through gss-proxy in this case).

I do not know if this should be in any way a problem because you are not trying to use impersonation, you are trying to use actual keytabs for users. So you should try to walk in amount point as a user and post the gss-proxy/rpc.gssd errors when that happen.

Root is probably squashed and is generally not a good user for debugging as rpc.gssd falls back trying to use machine credentials for root.

Simo.

On Thu, 2021-06-24 at 12:09 -0400, John Bazik wrote:

...

Sure, here's the klist output:

Ticket cache: FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM Default principal: CLIENT$@AD.EXAMPLE.COM

Valid starting       Expires              Service principal 06/24/2021 11:53:49  06/24/2021 21:53:49  krbtgt/AD.EXAMPLE.COM@AD.EXAMPLE.COM            renew until 07/01/2021 11:53:49 06/24/2021 11:53:49  06/24/2021 21:53:49  nfs/nfs.example.com@AD.EXAMPLE.COM            renew until 07/01/2021 11:53:49

And here is larger snippet from syslog with gssproxy debug = 2:

Jun 24 11:54:08 client rpc.gssd[6512]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt2fe) Jun 24 11:54:08 client rpc.gssd[6512]: krb5_use_machine_creds: uid 0 tgtname (null) Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [  ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [  ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure.  Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [  ] } context_handle: <Null> output_token: <Null> ) Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [  ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [  ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure.  Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [  ] } context_handle: <Null> output_token: <Null> ) Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com Jun 24 11:54:08 client rpc.gssd[6512]: doing error downcall

John

On 6/24/21 10:19 AM, Simo Sorce wrote:

...

Ok two points, can you raise the debug level of gssproy and see what it prints ?

Also can you klist the contents of /tmp/krb5ccmachine_AD.EXAMPLE.COM ?

Thanks, Simo.

On Wed, 2021-06-23 at 23:46 -0400, John Bazik wrote: > I've recently switched from using k5start to gssproxy to allow my users to access NFSv4 mounts with sec=krb5, using keytabs I manage for them.  I have just one service configured in gssproxy: > > [service/nfs-client] >       mechs = krb5 >       cred_store = keytab:/etc/krb5.keytab >       cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U >       cred_store = client_keytab:/var/local/keytabs/%u.keytab >       cred_usage = initiate >       allow_any_uid = yes >       trusted = yes >       euid = 0 > > I thought everything was working great, but now I find that I can't mount remote filesystems when gssproxy is running.  If I stop gssproxy, mount works.  If I change sec=krb5 to sec=sys, mount works.  It seems clear that gssproxy is preventing mount from working.  When I run mount -a, I get errors like this: > >       mount.nfs: access denied by server while mounting [...] > > When I add -vvv to rpc.gssd, this is what I see in syslog (anonymized): > >       rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com >       rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' >       rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' >       rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' >       rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' >       rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 >       rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 >       rpc.gssd[6512]: creating tcp client for server nfs.example.com >       rpc.gssd[6512]: DEBUG: port already set to 2049 >       rpc.gssd[6512]: creating context with server nfs@nfs.example.com >       rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com >       rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com >       rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com >       rpc.gssd[6512]: doing error downcall > > I'm running version 0.8.0, as distributed with Debian Buster (I worked around the systemd ordering cycle bug in that version by using the upstream unit file).  The fileserver is run by a different group and kerberos is AD. > > Googling for answers, I found others describe similar problems, but no solutions that make sense to me.  Help! > > John > _______________________________________________ > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

It's in nfs-utils 1.3.4. Looks like upstream (linux-nfs.org) is at 2.5.4.

John

On 6/24/21 3:21 PM, Simo Sorce wrote:

That's really odd given rpc.gssd seem to fall-back just fine and obtains a ticket. Sounds more like a bug in rpc.gssd, maybe the fallback path has been broken ...

What version of rpc.gssd is this ?

On Thu, 2021-06-24 at 15:07 -0400, John Bazik wrote:

...

I should add that user authentication with NFS works fine with my original config. It is only root executing mount that stopped working.

John

On 6/24/21 3:03 PM, John Bazik wrote:

...

I wondered about euid, but gssproxy.conf(5) says:

The "euid" parameter is imperative, any section without it will be discarded.

And when I delete those two lines from the configuration, gssproxy does not run:

Jun 24 14:50:32 client gssproxy[596]: [2021/06/24 18:50:32]: Debug Enabled (level: 2)   Jun 24 14:50:32 client gssproxy[596]: [2021/06/24 18:50:32]: Config file(s) not found!   Jun 24 14:50:32 client gssproxy: Option 'euid' is missing from [service/nfs-client].   Jun 24 14:50:32 client gssproxy: Error reading configuration 22: Invalid argument   Jun 24 14:50:32 client systemd[1]: gssproxy.service: Control process exited, code=exited, status=1/FAILURE   Jun 24 14:50:32 client systemd[1]: gssproxy.service: Failed with result 'exit-code'.

This is how I test that it's working (when gssproxy is running):

# su -c 'ls /home/jbazik' jbazik

So, you are right, I'm not interested in having root impersonate users.  I just want users to get keytab-authenticated.

John

On 6/24/21 2:21 PM, Simo Sorce wrote:

...

So think more about your config,given what you want to achieve you should probably removed euid = 0 and trusted = yes

rpc.gssd changes the uid used for the process, and this tells gss-proxy what user (and hence) what keytab to use), however euid = 0 is forcing to accept only connections from root, and trusted is saying root can impersonated anyone, but you do not want that.

I do not know if this will help the actual problem, but it is worth getting anything cleared so we do not chase ghosts.

Simo.

On Thu, 2021-06-24 at 13:58 -0400, John Bazik wrote:

...

So, no, there is no 0.keytab, and there is no AD account for root, so it would make no sense to have one.

We normally rely on root using machine credentials to do mounts.  I don't understand why that stops working when gssproxy is running.

John

On 6/24/21 12:41 PM, Simo Sorce wrote:

...

Do you have a keytab named /var/local/keytabs/0.keytab ?

It looks like gss-proxy attempts to acquire creds but uses ost/client.zz.example.com@ZZ.EXAMPLE.COM to try to obtain a TGT, but AD KDCs are picky and do not allow to use the SPN as the initiator they want to see a request from client$@AD.EXAMPLE.COM instead.

So gss-proxy returns an error and then rpc.gssd falls back and tries to directly obtain a credential and succeeds (ie the creds are not obtained through gss-proxy in this case).

I do not know if this should be in any way a problem because you are not trying to use impersonation, you are trying to use actual keytabs for users. So you should try to walk in amount point as a user and post the gss-proxy/rpc.gssd errors when that happen.

Root is probably squashed and is generally not a good user for debugging as rpc.gssd falls back trying to use machine credentials for root.

Simo.

On Thu, 2021-06-24 at 12:09 -0400, John Bazik wrote: > Sure, here's the klist output: > > Ticket cache: FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM > Default principal: CLIENT$@AD.EXAMPLE.COM > > Valid starting       Expires              Service principal > 06/24/2021 11:53:49  06/24/2021 21:53:49  krbtgt/AD.EXAMPLE.COM@AD.EXAMPLE.COM >            renew until 07/01/2021 11:53:49 > 06/24/2021 11:53:49  06/24/2021 21:53:49  nfs/nfs.example.com@AD.EXAMPLE.COM >            renew until 07/01/2021 11:53:49 > > And here is larger snippet from syslog with gssproxy debug = 2: > > Jun 24 11:54:08 client rpc.gssd[6512]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt2fe) > Jun 24 11:54:08 client rpc.gssd[6512]: krb5_use_machine_creds: uid 0 tgtname (null) > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' > Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' > Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [  ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) > Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com > Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 > Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [  ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure.  Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [  ] } context_handle: <Null> output_token: <Null> ) > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' > Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' > Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [  ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) > Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com > Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 > Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [  ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure.  Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [  ] } context_handle: <Null> output_token: <Null> ) > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com > Jun 24 11:54:08 client rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com > Jun 24 11:54:08 client rpc.gssd[6512]: doing error downcall > > John > > On 6/24/21 10:19 AM, Simo Sorce wrote: >> Ok two points, >> can you raise the debug level of gssproy and see what it prints ? >> >> Also can you klist the contents of /tmp/krb5ccmachine_AD.EXAMPLE.COM ? >> >> Thanks, >> Simo. >> >> On Wed, 2021-06-23 at 23:46 -0400, John Bazik wrote: >>> I've recently switched from using k5start to gssproxy to allow my users to access NFSv4 mounts with sec=krb5, using keytabs I manage for them.  I have just one service configured in gssproxy: >>> >>> [service/nfs-client] >>>       mechs = krb5 >>>       cred_store = keytab:/etc/krb5.keytab >>>       cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U >>>       cred_store = client_keytab:/var/local/keytabs/%u.keytab >>>       cred_usage = initiate >>>       allow_any_uid = yes >>>       trusted = yes >>>       euid = 0 >>> >>> I thought everything was working great, but now I find that I can't mount remote filesystems when gssproxy is running.  If I stop gssproxy, mount works.  If I change sec=krb5 to sec=sys, mount works.  It seems clear that gssproxy is preventing mount from working.  When I run mount -a, I get errors like this: >>> >>>       mount.nfs: access denied by server while mounting [...] >>> >>> When I add -vvv to rpc.gssd, this is what I see in syslog (anonymized): >>> >>>       rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com >>>       rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' >>>       rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' >>>       rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' >>>       rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' >>>       rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 >>>       rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 >>>       rpc.gssd[6512]: creating tcp client for server nfs.example.com >>>       rpc.gssd[6512]: DEBUG: port already set to 2049 >>>       rpc.gssd[6512]: creating context with server nfs@nfs.example.com >>>       rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com >>>       rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com >>>       rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com >>>       rpc.gssd[6512]: doing error downcall >>> >>> I'm running version 0.8.0, as distributed with Debian Buster (I worked around the systemd ordering cycle bug in that version by using the upstream unit file).  The fileserver is run by a different group and kerberos is AD. >>> >>> Googling for answers, I found others describe similar problems, but no solutions that make sense to me.  Help! >>> >>> John >>> _______________________________________________ >>> gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org >>> To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org >>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... >>> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure >> > _______________________________________________ > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

Ok my theory is that even though rpc.gssd obtains a credential that is valid, for some reason it later acts as if that is not the case. This causes interposition by gss-proxy which can obtain some creds (but fot eh worng principal), threfore gss-proxy assumes to have valid creds and keep interposing the later operation to try to setup a session with the server, which fails (no ticket) and therefore an error is returned.

I see a way to work around this. symlink the correct keytab as .../clients/0.keytab so that hopefully the correct creds are used. Additionally migh set the principal name to be used in this case as krb5_principal = client$@REALM (hopefully this won't break regular user auth).

This is so that gss-proxy can successfully interpose root using the machine account. The longer term fix would be to figure out why gss-proxy is invoked after initial credentials are obtained via fallback, I assume an issue in making libgssapi_crb5 see the correct ccache (FILE:/tmp/whatever_machine)

HTH, Simo.

On Thu, 2021-06-24 at 15:51 -0400, John Bazik wrote:

What I meant was that user auth, using gssproxy and keytabs, works fine. It's the root fallback that breaks.

John

On 6/24/21 3:21 PM, Simo Sorce wrote:

...

That's really odd given rpc.gssd seem to fall-back just fine and obtains a ticket. Sounds more like a bug in rpc.gssd, maybe the fallback path has been broken ...

What version of rpc.gssd is this ?

On Thu, 2021-06-24 at 15:07 -0400, John Bazik wrote:

...

I should add that user authentication with NFS works fine with my original config. It is only root executing mount that stopped working.

John

On 6/24/21 3:03 PM, John Bazik wrote:

...

I wondered about euid, but gssproxy.conf(5) says:

The "euid" parameter is imperative, any section without it will be discarded.

And when I delete those two lines from the configuration, gssproxy does not run:

Jun 24 14:50:32 client gssproxy[596]: [2021/06/24 18:50:32]: Debug Enabled (level: 2)   Jun 24 14:50:32 client gssproxy[596]: [2021/06/24 18:50:32]: Config file(s) not found!   Jun 24 14:50:32 client gssproxy: Option 'euid' is missing from [service/nfs-client].   Jun 24 14:50:32 client gssproxy: Error reading configuration 22: Invalid argument   Jun 24 14:50:32 client systemd[1]: gssproxy.service: Control process exited, code=exited, status=1/FAILURE   Jun 24 14:50:32 client systemd[1]: gssproxy.service: Failed with result 'exit-code'.

This is how I test that it's working (when gssproxy is running):

# su -c 'ls /home/jbazik' jbazik

So, you are right, I'm not interested in having root impersonate users.  I just want users to get keytab-authenticated.

John

On 6/24/21 2:21 PM, Simo Sorce wrote:

...

So think more about your config,given what you want to achieve you should probably removed euid = 0 and trusted = yes

rpc.gssd changes the uid used for the process, and this tells gss-proxy what user (and hence) what keytab to use), however euid = 0 is forcing to accept only connections from root, and trusted is saying root can impersonated anyone, but you do not want that.

I do not know if this will help the actual problem, but it is worth getting anything cleared so we do not chase ghosts.

Simo.

On Thu, 2021-06-24 at 13:58 -0400, John Bazik wrote:

...

So, no, there is no 0.keytab, and there is no AD account for root, so it would make no sense to have one.

We normally rely on root using machine credentials to do mounts.  I don't understand why that stops working when gssproxy is running.

John

On 6/24/21 12:41 PM, Simo Sorce wrote: > > Do you have a keytab named /var/local/keytabs/0.keytab ? > > It looks like gss-proxy attempts to acquire creds but uses > ost/client.zz.example.com@ZZ.EXAMPLE.COM to try to obtain a TGT, but AD > KDCs are picky and do not allow to use the SPN as the initiator they > want to see a request from client$@AD.EXAMPLE.COM instead. > > So gss-proxy returns an error and then rpc.gssd falls back and tries to > directly obtain a credential and succeeds (ie the creds are not > obtained through gss-proxy in this case). > > I do not know if this should be in any way a problem because you are > not trying to use impersonation, you are trying to use actual keytabs > for users. So you should try to walk in amount point as a user and post > the gss-proxy/rpc.gssd errors when that happen. > > Root is probably squashed and is generally not a good user for > debugging as rpc.gssd falls back trying to use machine credentials for > root. > > Simo. > > > On Thu, 2021-06-24 at 12:09 -0400, John Bazik wrote: > > Sure, here's the klist output: > > > > Ticket cache: FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM > > Default principal: CLIENT$@AD.EXAMPLE.COM > > > > Valid starting       Expires              Service principal > > 06/24/2021 11:53:49  06/24/2021 21:53:49  krbtgt/AD.EXAMPLE.COM@AD.EXAMPLE.COM > >            renew until 07/01/2021 11:53:49 > > 06/24/2021 11:53:49  06/24/2021 21:53:49  nfs/nfs.example.com@AD.EXAMPLE.COM > >            renew until 07/01/2021 11:53:49 > > > > And here is larger snippet from syslog with gssproxy debug = 2: > > > > Jun 24 11:54:08 client rpc.gssd[6512]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt2fe) > > Jun 24 11:54:08 client rpc.gssd[6512]: krb5_use_machine_creds: uid 0 tgtname (null) > > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' > > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' > > Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' > > Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' > > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [  ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) > > Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com > > Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 > > Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [  ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure.  Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [  ] } context_handle: <Null> output_token: <Null> ) > > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com > > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com > > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com > > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' > > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' > > Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' > > Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' > > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [  ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) > > Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com > > Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 > > Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [  ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure.  Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [  ] } context_handle: <Null> output_token: <Null> ) > > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com > > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com > > Jun 24 11:54:08 client rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com > > Jun 24 11:54:08 client rpc.gssd[6512]: doing error downcall > > > > John > > > > On 6/24/21 10:19 AM, Simo Sorce wrote: > > > Ok two points, > > > can you raise the debug level of gssproy and see what it prints ? > > > > > > Also can you klist the contents of /tmp/krb5ccmachine_AD.EXAMPLE.COM ? > > > > > > Thanks, > > > Simo. > > > > > > On Wed, 2021-06-23 at 23:46 -0400, John Bazik wrote: > > > > I've recently switched from using k5start to gssproxy to allow my users to access NFSv4 mounts with sec=krb5, using keytabs I manage for them.  I have just one service configured in gssproxy: > > > > > > > > [service/nfs-client] > > > >       mechs = krb5 > > > >       cred_store = keytab:/etc/krb5.keytab > > > >       cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U > > > >       cred_store = client_keytab:/var/local/keytabs/%u.keytab > > > >       cred_usage = initiate > > > >       allow_any_uid = yes > > > >       trusted = yes > > > >       euid = 0 > > > > > > > > I thought everything was working great, but now I find that I can't mount remote filesystems when gssproxy is running.  If I stop gssproxy, mount works.  If I change sec=krb5 to sec=sys, mount works.  It seems clear that gssproxy is preventing mount from working.  When I run mount -a, I get errors like this: > > > > > > > >       mount.nfs: access denied by server while mounting [...] > > > > > > > > When I add -vvv to rpc.gssd, this is what I see in syslog (anonymized): > > > > > > > >       rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com > > > >       rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' > > > >       rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' > > > >       rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' > > > >       rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' > > > >       rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 > > > >       rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 > > > >       rpc.gssd[6512]: creating tcp client for server nfs.example.com > > > >       rpc.gssd[6512]: DEBUG: port already set to 2049 > > > >       rpc.gssd[6512]: creating context with server nfs@nfs.example.com > > > >       rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com > > > >       rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com > > > >       rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com > > > >       rpc.gssd[6512]: doing error downcall > > > > > > > > I'm running version 0.8.0, as distributed with Debian Buster (I worked around the systemd ordering cycle bug in that version by using the upstream unit file).  The fileserver is run by a different group and kerberos is AD. > > > > > > > > Googling for answers, I found others describe similar problems, but no solutions that make sense to me.  Help! > > > > > > > > John > > > > _______________________________________________ > > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > > > > To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org > > > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... > > > > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure > > > > > _______________________________________________ > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > > To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... > > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure > _______________________________________________ gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

Sorry to revive this, but I noticed something odd. We have two kerberos realms, AD.EXAMPLE.COM and ZZ.EXAMPLE.COM. Only the AD realm is configured in /etc/krb5.conf. The ZZ realm is not intended to be used, but there is still a KDC for it on our network. There are no ZZ realm keys in /etc/krb5.keytab.

If you look back, earlier in this thread, to the gssproxy syslog output I posted, gssproxy seems to be using a principal for a realm it shouldn't know about:

Jun 24 11:54:08 client gssproxy[32163]: GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 )

Indeed, once I got it working after our last exchange, I noticed this:

# klist /tmp/krb5ccmachine_AD.EXAMPLE.COM Ticket cache: FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM Default principal: CLIENT$@AD.EXAMPLE.COM

Valid starting Expires Service principal 06/24/2021 17:32:20 06/25/2021 03:32:20 krbtgt/AD.EXAMPLE.COM@AD.EXAMPLE.COM renew until 07/01/2021 17:32:20 06/24/2021 17:32:21 06/25/2021 03:32:20 nfs/nfs.example.com@AD.EXAMPLE.COM renew until 07/01/2021 17:32:20

# klist /var/lib/gssproxy/clients/krb5cc_0 Ticket cache: FILE:/var/lib/gssproxy/clients/krb5cc_0 Default principal: host/client.zz.example.com@ZZ.EXAMPLE.COM

Valid starting Expires Service principal 06/24/2021 17:25:50 06/25/2021 03:25:50 krbtgt/ZZ.EXAMPLE.COM@ZZ.EXAMPLE.COM renew until 07/01/2021 17:25:50

And then I noticed a ZZ domain key was added to /etc/krb5.keytab:

4 08/17/2017 22:35:13 host/client.zz.example.com@ZZ.EXAMPLE.COM 4 08/17/2017 22:35:13 host/client.zz.example.com@ZZ.EXAMPLE.COM 4 08/17/2017 22:35:13 host/client.zz.example.com@ZZ.EXAMPLE.COM

So, it's working, but this doesn't seem right. Could gssproxy be searching for the domain, finding our ZZ KDC and using it, even though it should only use domains declared in /etc/krb5.conf? Perhaps that was my original problem?

John

On 6/24/21 6:08 PM, John Bazik wrote:

Simo, thank you!

Adding a symlink to /etc/krb5.keytab and adding krb5_principal to the config fixes my problem, it's all working.

Thanks again.

John

On 6/24/21 4:39 PM, Simo Sorce wrote:

...

Ok my theory is that even though rpc.gssd obtains a credential that is valid, for some reason it later acts as if that is not the case. This causes interposition by gss-proxy which can obtain some creds (but fot eh worng principal), threfore gss-proxy assumes to have valid creds and keep interposing the later operation to try to setup a session with the server, which fails (no ticket) and therefore an error is returned.

I see a way to work around this. symlink the correct keytab as .../clients/0.keytab so that hopefully the correct creds are used. Additionally migh set the principal name to be used in this case as krb5_principal = client$@REALM (hopefully this won't break regular user auth).

This is so that gss-proxy can successfully interpose root using the machine account. The longer term fix would be to figure out why gss-proxy is invoked after initial credentials are obtained via fallback, I assume an issue in making libgssapi_crb5 see the correct ccache (FILE:/tmp/whatever_machine)

HTH, Simo.

On Thu, 2021-06-24 at 15:51 -0400, John Bazik wrote:

...

What I meant was that user auth, using gssproxy and keytabs, works fine.  It's the root fallback that breaks.

John

On 6/24/21 3:21 PM, Simo Sorce wrote:

...

That's really odd given rpc.gssd seem to fall-back just fine and obtains a ticket. Sounds more like a bug in rpc.gssd, maybe the fallback path has been broken ...

What version of rpc.gssd is this ?

On Thu, 2021-06-24 at 15:07 -0400, John Bazik wrote:

...

I should add that user authentication with NFS works fine with my original config.  It is only root executing mount that stopped working.

John

On 6/24/21 3:03 PM, John Bazik wrote:

...

I wondered about euid, but gssproxy.conf(5) says:

The "euid" parameter is imperative, any section without it will be discarded.

And when I delete those two lines from the configuration, gssproxy does not run:

Jun 24 14:50:32 client gssproxy[596]: [2021/06/24 18:50:32]: Debug Enabled (level: 2)      Jun 24 14:50:32 client gssproxy[596]: [2021/06/24 18:50:32]: Config file(s) not found!      Jun 24 14:50:32 client gssproxy: Option 'euid' is missing from [service/nfs-client].      Jun 24 14:50:32 client gssproxy: Error reading configuration 22: Invalid argument      Jun 24 14:50:32 client systemd[1]: gssproxy.service: Control process exited, code=exited, status=1/FAILURE      Jun 24 14:50:32 client systemd[1]: gssproxy.service: Failed with result 'exit-code'.

This is how I test that it's working (when gssproxy is running):

# su -c 'ls /home/jbazik' jbazik

So, you are right, I'm not interested in having root impersonate users.  I just want users to get keytab-authenticated.

John

On 6/24/21 2:21 PM, Simo Sorce wrote: > So think more about your config,given what you want to achieve you > should probably removed euid = 0 and trusted = yes > > rpc.gssd changes the uid used for the process, and this tells gss-proxy > what user (and hence) what keytab to use), however euid = 0 is forcing > to accept only connections from root, and trusted is saying root can > impersonated anyone, but you do not want that. > > I do not know if this will help the actual problem, but it is worth > getting anything cleared so we do not chase ghosts. > > Simo. > > On Thu, 2021-06-24 at 13:58 -0400, John Bazik wrote: >> So, no, there is no 0.keytab, and there is no AD account for root, so it would make no sense to have one. >> >> We normally rely on root using machine credentials to do mounts.  I don't understand why that stops working when gssproxy is running. >> >> John >> >> On 6/24/21 12:41 PM, Simo Sorce wrote: >>> >>> Do you have a keytab named /var/local/keytabs/0.keytab ? >>> >>> It looks like gss-proxy attempts to acquire creds but uses >>> ost/client.zz.example.com@ZZ.EXAMPLE.COM to try to obtain a TGT, but AD >>> KDCs are picky and do not allow to use the SPN as the initiator they >>> want to see a request from client$@AD.EXAMPLE.COM instead. >>> >>> So gss-proxy returns an error and then rpc.gssd falls back and tries to >>> directly obtain a credential and succeeds (ie the creds are not >>> obtained through gss-proxy in this case). >>> >>> I do not know if this should be in any way a problem because you are >>> not trying to use impersonation, you are trying to use actual keytabs >>> for users. So you should try to walk in amount point as a user and post >>> the gss-proxy/rpc.gssd errors when that happen. >>> >>> Root is probably squashed and is generally not a good user for >>> debugging as rpc.gssd falls back trying to use machine credentials for >>> root. >>> >>> Simo. >>> >>> >>> On Thu, 2021-06-24 at 12:09 -0400, John Bazik wrote: >>>> Sure, here's the klist output: >>>> >>>> Ticket cache: FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM >>>> Default principal: CLIENT$@AD.EXAMPLE.COM >>>> >>>> Valid starting       Expires              Service principal >>>> 06/24/2021 11:53:49  06/24/2021 21:53:49  krbtgt/AD.EXAMPLE.COM@AD.EXAMPLE.COM >>>>              renew until 07/01/2021 11:53:49 >>>> 06/24/2021 11:53:49  06/24/2021 21:53:49  nfs/nfs.example.com@AD.EXAMPLE.COM >>>>              renew until 07/01/2021 11:53:49 >>>> >>>> And here is larger snippet from syslog with gssproxy debug = 2: >>>> >>>> Jun 24 11:54:08 client rpc.gssd[6512]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt2fe) >>>> Jun 24 11:54:08 client rpc.gssd[6512]: krb5_use_machine_creds: uid 0 tgtname (null) >>>> Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' >>>> Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' >>>> Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' >>>> Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' >>>> Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 >>>> Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 >>>> Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client >>>> Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) >>>> Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) >>>> Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [  ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) >>>> Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com >>>> Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 >>>> Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com >>>> Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client >>>> Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) >>>> Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [  ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) >>>> Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration >>>> Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure.  Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [  ] } context_handle: <Null> output_token: <Null> ) >>>> Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com >>>> Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com >>>> Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com >>>> Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' >>>> Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' >>>> Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' >>>> Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' >>>> Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 >>>> Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 >>>> Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client >>>> Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) >>>> Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) >>>> Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [  ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) >>>> Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com >>>> Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 >>>> Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com >>>> Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client >>>> Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) >>>> Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [  ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) >>>> Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration >>>> Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure.  Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [  ] } context_handle: <Null> output_token: <Null> ) >>>> Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com >>>> Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com >>>> Jun 24 11:54:08 client rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com >>>> Jun 24 11:54:08 client rpc.gssd[6512]: doing error downcall >>>> >>>> John >>>> >>>> On 6/24/21 10:19 AM, Simo Sorce wrote: >>>>> Ok two points, >>>>> can you raise the debug level of gssproy and see what it prints ? >>>>> >>>>> Also can you klist the contents of /tmp/krb5ccmachine_AD.EXAMPLE.COM ? >>>>> >>>>> Thanks, >>>>> Simo. >>>>> >>>>> On Wed, 2021-06-23 at 23:46 -0400, John Bazik wrote: >>>>>> I've recently switched from using k5start to gssproxy to allow my users to access NFSv4 mounts with sec=krb5, using keytabs I manage for them.  I have just one service configured in gssproxy: >>>>>> >>>>>> [service/nfs-client] >>>>>>         mechs = krb5 >>>>>>         cred_store = keytab:/etc/krb5.keytab >>>>>>         cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U >>>>>>         cred_store = client_keytab:/var/local/keytabs/%u.keytab >>>>>>         cred_usage = initiate >>>>>>         allow_any_uid = yes >>>>>>         trusted = yes >>>>>>         euid = 0 >>>>>> >>>>>> I thought everything was working great, but now I find that I can't mount remote filesystems when gssproxy is running.  If I stop gssproxy, mount works.  If I change sec=krb5 to sec=sys, mount works.  It seems clear that gssproxy is preventing mount from working.  When I run mount -a, I get errors like this: >>>>>> >>>>>>         mount.nfs: access denied by server while mounting [...] >>>>>> >>>>>> When I add -vvv to rpc.gssd, this is what I see in syslog (anonymized): >>>>>> >>>>>>         rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com >>>>>>         rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' >>>>>>         rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' >>>>>>         rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' >>>>>>         rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' >>>>>>         rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 >>>>>>         rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 >>>>>>         rpc.gssd[6512]: creating tcp client for server nfs.example.com >>>>>>         rpc.gssd[6512]: DEBUG: port already set to 2049 >>>>>>         rpc.gssd[6512]: creating context with server nfs@nfs.example.com >>>>>>         rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com >>>>>>         rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com >>>>>>         rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com >>>>>>         rpc.gssd[6512]: doing error downcall >>>>>> >>>>>> I'm running version 0.8.0, as distributed with Debian Buster (I worked around the systemd ordering cycle bug in that version by using the upstream unit file).  The fileserver is run by a different group and kerberos is AD. >>>>>> >>>>>> Googling for answers, I found others describe similar problems, but no solutions that make sense to me.  Help! >>>>>> >>>>>> John >>>>>> _______________________________________________ >>>>>> gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org >>>>>> To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org >>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>>> List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... >>>>>> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure >>>>> >>>> _______________________________________________ >>>> gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org >>>> To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org >>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... >>>> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure >>> >> _______________________________________________ >> gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org >> To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org >> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... >> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure >

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

Oh thank you for following up with the description of the issue you found as well. It will definitely help others that may have similar issues in the future.

Simo.

On Thu, 2021-06-24 at 23:55 -0400, John Bazik wrote:

Problem solved!

We had "dns_lookup_realm = true" in our krb5.conf defaults. When I set that to false, everything works with our original configuration. No need for a symlink or krb5_principal setting.

This one was pilot error. Thanks for your patience and suggestions, I really appreciate it.

John

On 6/24/21 10:30 PM, John Bazik wrote:

...

Sorry to revive this, but I noticed something odd.  We have two kerberos realms, AD.EXAMPLE.COM and ZZ.EXAMPLE.COM.  Only the AD realm is configured in /etc/krb5.conf.  The ZZ realm is not intended to be used, but there is still a KDC for it on our network.  There are no ZZ realm keys in /etc/krb5.keytab.

If you look back, earlier in this thread, to the gssproxy syslog output I posted, gssproxy seems to be using a principal for a realm it shouldn't know about:

Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 )

Indeed, once I got it working after our last exchange, I noticed this:

# klist /tmp/krb5ccmachine_AD.EXAMPLE.COM   Ticket cache: FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM   Default principal: CLIENT$@AD.EXAMPLE.COM

Valid starting       Expires              Service principal   06/24/2021 17:32:20  06/25/2021 03:32:20  krbtgt/AD.EXAMPLE.COM@AD.EXAMPLE.COM           renew until 07/01/2021 17:32:20   06/24/2021 17:32:21  06/25/2021 03:32:20  nfs/nfs.example.com@AD.EXAMPLE.COM           renew until 07/01/2021 17:32:20

# klist /var/lib/gssproxy/clients/krb5cc_0   Ticket cache: FILE:/var/lib/gssproxy/clients/krb5cc_0   Default principal: host/client.zz.example.com@ZZ.EXAMPLE.COM   Valid starting       Expires              Service principal   06/24/2021 17:25:50  06/25/2021 03:25:50  krbtgt/ZZ.EXAMPLE.COM@ZZ.EXAMPLE.COM           renew until 07/01/2021 17:25:50

And then I noticed a ZZ domain key was added to /etc/krb5.keytab:

4 08/17/2017 22:35:13 host/client.zz.example.com@ZZ.EXAMPLE.COM    4 08/17/2017 22:35:13 host/client.zz.example.com@ZZ.EXAMPLE.COM    4 08/17/2017 22:35:13 host/client.zz.example.com@ZZ.EXAMPLE.COM

So, it's working, but this doesn't seem right.  Could gssproxy be searching for the domain, finding our ZZ KDC and using it, even though it should only use domains declared in /etc/krb5.conf?  Perhaps that was my original problem?

John

On 6/24/21 6:08 PM, John Bazik wrote:

...

Simo, thank you!

Adding a symlink to /etc/krb5.keytab and adding krb5_principal to the config fixes my problem, it's all working.

Thanks again.

John

On 6/24/21 4:39 PM, Simo Sorce wrote:

...

Ok my theory is that even though rpc.gssd obtains a credential that is valid, for some reason it later acts as if that is not the case. This causes interposition by gss-proxy which can obtain some creds (but fot eh worng principal), threfore gss-proxy assumes to have valid creds and keep interposing the later operation to try to setup a session with the server, which fails (no ticket) and therefore an error is returned.

I see a way to work around this. symlink the correct keytab as .../clients/0.keytab so that hopefully the correct creds are used. Additionally migh set the principal name to be used in this case as krb5_principal = client$@REALM (hopefully this won't break regular user auth).

This is so that gss-proxy can successfully interpose root using the machine account. The longer term fix would be to figure out why gss-proxy is invoked after initial credentials are obtained via fallback, I assume an issue in making libgssapi_crb5 see the correct ccache (FILE:/tmp/whatever_machine)

HTH, Simo.

On Thu, 2021-06-24 at 15:51 -0400, John Bazik wrote:

...

What I meant was that user auth, using gssproxy and keytabs, works fine.  It's the root fallback that breaks.

John

On 6/24/21 3:21 PM, Simo Sorce wrote:

...

That's really odd given rpc.gssd seem to fall-back just fine and obtains a ticket. Sounds more like a bug in rpc.gssd, maybe the fallback path has been broken ...

What version of rpc.gssd is this ?

On Thu, 2021-06-24 at 15:07 -0400, John Bazik wrote: > I should add that user authentication with NFS works fine with my original config.  It is only root executing mount that stopped working. > > John > > On 6/24/21 3:03 PM, John Bazik wrote: > > I wondered about euid, but gssproxy.conf(5) says: > > > >      The "euid" parameter is imperative, any section without it will be discarded. > > > > And when I delete those two lines from the configuration, gssproxy does not run: > > > >      Jun 24 14:50:32 client gssproxy[596]: [2021/06/24 18:50:32]: Debug Enabled (level: 2) > >      Jun 24 14:50:32 client gssproxy[596]: [2021/06/24 18:50:32]: Config file(s) not found! > >      Jun 24 14:50:32 client gssproxy: Option 'euid' is missing from [service/nfs-client]. > >      Jun 24 14:50:32 client gssproxy: Error reading configuration 22: Invalid argument > >      Jun 24 14:50:32 client systemd[1]: gssproxy.service: Control process exited, code=exited, status=1/FAILURE > >      Jun 24 14:50:32 client systemd[1]: gssproxy.service: Failed with result 'exit-code'. > > > > This is how I test that it's working (when gssproxy is running): > > > >      # su -c 'ls /home/jbazik' jbazik > > > > So, you are right, I'm not interested in having root impersonate users.  I just want users to get keytab-authenticated. > > > > John > > > > On 6/24/21 2:21 PM, Simo Sorce wrote: > > > So think more about your config,given what you want to achieve you > > > should probably removed euid = 0 and trusted = yes > > > > > > rpc.gssd changes the uid used for the process, and this tells gss-proxy > > > what user (and hence) what keytab to use), however euid = 0 is forcing > > > to accept only connections from root, and trusted is saying root can > > > impersonated anyone, but you do not want that. > > > > > > I do not know if this will help the actual problem, but it is worth > > > getting anything cleared so we do not chase ghosts. > > > > > > Simo. > > > > > > On Thu, 2021-06-24 at 13:58 -0400, John Bazik wrote: > > > > So, no, there is no 0.keytab, and there is no AD account for root, so it would make no sense to have one. > > > > > > > > We normally rely on root using machine credentials to do mounts.  I don't understand why that stops working when gssproxy is running. > > > > > > > > John > > > > > > > > On 6/24/21 12:41 PM, Simo Sorce wrote: > > > > > > > > > > Do you have a keytab named /var/local/keytabs/0.keytab ? > > > > > > > > > > It looks like gss-proxy attempts to acquire creds but uses > > > > > ost/client.zz.example.com@ZZ.EXAMPLE.COM to try to obtain a TGT, but AD > > > > > KDCs are picky and do not allow to use the SPN as the initiator they > > > > > want to see a request from client$@AD.EXAMPLE.COM instead. > > > > > > > > > > So gss-proxy returns an error and then rpc.gssd falls back and tries to > > > > > directly obtain a credential and succeeds (ie the creds are not > > > > > obtained through gss-proxy in this case). > > > > > > > > > > I do not know if this should be in any way a problem because you are > > > > > not trying to use impersonation, you are trying to use actual keytabs > > > > > for users. So you should try to walk in amount point as a user and post > > > > > the gss-proxy/rpc.gssd errors when that happen. > > > > > > > > > > Root is probably squashed and is generally not a good user for > > > > > debugging as rpc.gssd falls back trying to use machine credentials for > > > > > root. > > > > > > > > > > Simo. > > > > > > > > > > > > > > > On Thu, 2021-06-24 at 12:09 -0400, John Bazik wrote: > > > > > > Sure, here's the klist output: > > > > > > > > > > > > Ticket cache: FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM > > > > > > Default principal: CLIENT$@AD.EXAMPLE.COM > > > > > > > > > > > > Valid starting       Expires              Service principal > > > > > > 06/24/2021 11:53:49  06/24/2021 21:53:49  krbtgt/AD.EXAMPLE.COM@AD.EXAMPLE.COM > > > > > >              renew until 07/01/2021 11:53:49 > > > > > > 06/24/2021 11:53:49  06/24/2021 21:53:49  nfs/nfs.example.com@AD.EXAMPLE.COM > > > > > >              renew until 07/01/2021 11:53:49 > > > > > > > > > > > > And here is larger snippet from syslog with gssproxy debug = 2: > > > > > > > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt2fe) > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: krb5_use_machine_creds: uid 0 tgtname (null) > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > > > > > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > > > > > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) > > > > > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) > > > > > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [  ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com > > > > > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > > > > > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) > > > > > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [  ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) > > > > > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration > > > > > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure.  Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [  ] } context_handle: <Null> output_token: <Null> ) > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624586029 > > > > > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > > > > > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) > > > > > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) > > > > > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_ACQUIRE_CRED( status: { 0 { 1 2 840 113554 1 2 2 } 0 "" "" [  ] } output_cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 } ] [ ....p..w.z....o.... ] 0 } ) > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: creating tcp client for server nfs.example.com > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: DEBUG: port already set to 2049 > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: creating context with server nfs@nfs.example.com > > > > > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Connection matched service nfs-client > > > > > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) > > > > > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_ARG_INIT_SEC_CONTEXT( call_ctx: { "" [  ] } context_handle: <Null> cred_handle: { "host/client.zz.example.com@ZZ.EXAMPLE.COM" [ { "host/client.zz.example.com@ZZ.EXAMPLE.COM" { 1 2 840 113554 1 2 2 } INITIATE 36000 0 [ { [ krb5.set.allowed... ] [ ................... ] } ] } ] [ ....p..w.z....o.... ] 0 } target_name: "nfs@nfs.example.com" mech_type: { 1 2 840 113554 1 2 2 } req_flags: 2 time_req: 0 input_cb: <Null> input_token: <Null> [ { [ sync.modified.cr... ] [ 64656661756c740 ] } ] ) > > > > > > Jun 24 11:54:08 client gssproxy[32163]: [CID 9][2021/06/24 15:54:08]: Credentials allowed by configuration > > > > > > Jun 24 11:54:08 client gssproxy[32163]:     GSSX_RES_INIT_SEC_CONTEXT( status: { 851968 { 1 2 840 113554 1 2 2 } 2529638972 "Unspecified GSS failure.  Minor code may provide more information" "KDC returned error string: FINDING_SERVER_KEY" [  ] } context_handle: <Null> output_token: <Null> ) > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com > > > > > > Jun 24 11:54:08 client rpc.gssd[6512]: doing error downcall > > > > > > > > > > > > John > > > > > > > > > > > > On 6/24/21 10:19 AM, Simo Sorce wrote: > > > > > > > Ok two points, > > > > > > > can you raise the debug level of gssproy and see what it prints ? > > > > > > > > > > > > > > Also can you klist the contents of /tmp/krb5ccmachine_AD.EXAMPLE.COM ? > > > > > > > > > > > > > > Thanks, > > > > > > > Simo. > > > > > > > > > > > > > > On Wed, 2021-06-23 at 23:46 -0400, John Bazik wrote: > > > > > > > > I've recently switched from using k5start to gssproxy to allow my users to access NFSv4 mounts with sec=krb5, using keytabs I manage for them.  I have just one service configured in gssproxy: > > > > > > > > > > > > > > > > [service/nfs-client] > > > > > > > >         mechs = krb5 > > > > > > > >         cred_store = keytab:/etc/krb5.keytab > > > > > > > >         cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U > > > > > > > >         cred_store = client_keytab:/var/local/keytabs/%u.keytab > > > > > > > >         cred_usage = initiate > > > > > > > >         allow_any_uid = yes > > > > > > > >         trusted = yes > > > > > > > >         euid = 0 > > > > > > > > > > > > > > > > I thought everything was working great, but now I find that I can't mount remote filesystems when gssproxy is running.  If I stop gssproxy, mount works.  If I change sec=krb5 to sec=sys, mount works.  It seems clear that gssproxy is preventing mount from working.  When I run mount -a, I get errors like this: > > > > > > > > > > > > > > > >         mount.nfs: access denied by server while mounting [...] > > > > > > > > > > > > > > > > When I add -vvv to rpc.gssd, this is what I see in syslog (anonymized): > > > > > > > > > > > > > > > >         rpc.gssd[6512]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com > > > > > > > >         rpc.gssd[6512]: Full hostname for 'nfs.example.com' is 'nfs.example.com' > > > > > > > >         rpc.gssd[6512]: Full hostname for 'client.zz.example.com' is 'client.zz.example.com' > > > > > > > >         rpc.gssd[6512]: No key table entry found for client$@AD.EXAMPLE.COM while getting keytab entry for 'client$@AD.EXAMPLE.COM' > > > > > > > >         rpc.gssd[6512]: Success getting keytab entry for 'CLIENT$@AD.EXAMPLE.COM' > > > > > > > >         rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 > > > > > > > >         rpc.gssd[6512]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM' are good until 1624541464 > > > > > > > >         rpc.gssd[6512]: creating tcp client for server nfs.example.com > > > > > > > >         rpc.gssd[6512]: DEBUG: port already set to 2049 > > > > > > > >         rpc.gssd[6512]: creating context with server nfs@nfs.example.com > > > > > > > >         rpc.gssd[6512]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com > > > > > > > >         rpc.gssd[6512]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_AD.EXAMPLE.COM for server nfs.example.com > > > > > > > >         rpc.gssd[6512]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com > > > > > > > >         rpc.gssd[6512]: doing error downcall > > > > > > > > > > > > > > > > I'm running version 0.8.0, as distributed with Debian Buster (I worked around the systemd ordering cycle bug in that version by using the upstream unit file).  The fileserver is run by a different group and kerberos is AD. > > > > > > > > > > > > > > > > Googling for answers, I found others describe similar problems, but no solutions that make sense to me.  Help! > > > > > > > > > > > > > > > > John > > > > > > > > _______________________________________________ > > > > > > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > > > > > > > > To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org > > > > > > > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > > > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > > > > > List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... > > > > > > > > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure > > > > > > > > > > > > > _______________________________________________ > > > > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > > > > > > To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org > > > > > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > > > List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... > > > > > > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure > > > > > > > > > _______________________________________________ > > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > > > > To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org > > > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... > > > > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure > > > > _______________________________________________ > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

---

gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/gss-proxy@lists.fedorahosted.or... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure