Hi Simo, Probably I made a mistake here: I got the keytab from the IPA server and named it /var/lib/gssproxy/clients/krb5cc_<httpd uid> instead of /var/lib/gssproxy/clients/<httpd_uid>.keytab. I fixed that now (by mv'ing it) and ran systemctl stop gssproxy to not have gssproxy as a daemon gssproxy -d -i -C /etc/gssproxy/ & to have gssproxy dump any debug msgs to my terminal After temporarily mounting the docroot to /mnt using mount -vv -t nfs4 nfsserver:/path/to/export /mnt I saw some debug messages flying by: mount.nfs4: timeout set for Tue Feb 13 08:20:21 2018 mount.nfs4: trying text-based options 'vers=4.1,addr=<nfsservip>,clientaddr=<nfsclientip>' [2018/02/13 08:18:21]: Client connected (fd = 9)[2018/02/13 08:18:21]: (pid = 623) (uid = 0) (gid = 0)[2018/02/13 08:18:21]: (context = system_u:system_r:gssd_t:s0)[2018/02/13 08:18:21]: [2018/02/13 08:18:21]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) [2018/02/13 08:18:21]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null) And when I then su to the apache user and try to access the directory, I see some more debug messages: # su -s /bin/bash <apache-user> bash-4.2$ cd /mnt [2018/02/13 08:19:33]: Client connected (fd = 9)[2018/02/13 08:19:33]: (pid = 623) (uid = <httpd uid>) (gid = <httpd uid>)[2018/02/13 08:19:33]: (context = system_u:system_r:gssd_t:s0)[2018/02/13 08:19:33]: [2018/02/13 08:19:33]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: <httpd uid>,socket: (null) [2018/02/13 08:19:33]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: <httpd uid>,socket: (null) [2018/02/13 08:19:34]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: <httpd uid>,socket: (null) bash: cd: /mnt: Permission denied Unfortunately still no success, as the last line indicates, however I get to see a new /var/lib/gssproxy/clients/krb5cc_<httpd uid> file tat was created by gssproxy, I presume: -rw-------. 1 root root 1819 Feb 13 08:19 krb5cc_<httpd uid> The keytab file /var/lib/gssproxy/clients/<httpd uid>.keytab looks like this: # klist -ke <httpd uid>.keytab Keytab name: FILE:<httpd uid>.keytab KVNO Principal ---- -------------------------------------------------------------------------- 1 nfs/<nfsserver-fqdn@REALM> (aes256-cts-hmac-sha1-96) 1 nfs/<nfsserver-fqdn@REALM> (aes128-cts-hmac-sha1-96) 1 host/<nfsclient-fqdn@REALM> (aes256-cts-hmac-sha1-96) 1 host/<nfsclient-fqdn@REALM> (aes128-cts-hmac-sha1-96) Ok, that would make sense. Will do that once I can access the docroot as apache user. Do the debug messages shown above indicate what might be wrong? The "socket: (null)" bit perhaps? The last two log lines looked rather promising to me: [2018/02/13 08:19:33]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: <httpd uid>,socket: (null) [2018/02/13 08:19:34]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: <httpd uid>,socket: (null) This is probably the confirmation about the credtials being written to krb5cc_<httpd uid>, right? Still getting "permission denied" would perhaps indicate that the apache user has no access to the NFS mount, even considering the credential store? Best, Ray

Hi Simo, Ok, right now I switched SELinunx off (setenforce 0) until I have somehting working... I removed the above keytab and made a new one with user credentials: # klist -ke <httpd-uid>.keytab Keytab name: FILE:<httpd-uid>.keytab KVNO Principal ---- -------------------------------------------------------------------------- 2 <httpd-user>@REALM (aes256-cts-hmac-sha1-96) 2 <httpd-user>@REALM (aes128-cts-hmac-sha1-96) This seems insufficient though, 'coz when I mount the NFS docroot temporarily to /mnt and then "ls -l /" as httpd-user, I get to see this line: d?????????? ? ? ? ? ? mnt Which other principals should I add to the keytab? I tried adding the nfs service principal to the keytab, but that doesn't change this behavior... Does the order in which I add principals to the keytab matter in this context? klist for httpd-user looks not good either, concering the validy timestamps (1970...1970), or is that normal?: $ klist Ticket cache: KEYRING:persistent:<httpd uid>:<httpd uid> Default principal: <httpd-user>@REALM Valid starting Expires Service principal 01/01/1970 01:00:00 01/01/1970 01:00:00 Encrypted/Credentials/v1@X-GSSPROXY: I think there's something not right with gssproxy: when I call it with --debug-level=3 it still says "level: 0": # gssproxy --debug --debug-level=3 -i -C /etc/gssproxy/ & [1] 17831 root@nfsclient:/var/lib/gssproxy/clients# [2018/02/13 09:56:20]: Debug Enabled (level: 0) Nevertheless the output becomes extremely verbose (plenty of hex). In the man-page --debug-level is not mentioned at all (ok, still better this way than having something promised in the manpage that does not exist in the binary...). Best, Ray

Hi Simo, you were right, the server side had a problem in that the NFS service principal's key version was outdated with respect to the KDC's key version. Not sure why the KDCs key version suddenly leapt ahead, but it had to be something along those lines: suddenly no NFS mount worked for anyone on any client. Kicked the old NFS service principal key from the NFS server's keytab and imported the new one et voilá: all working, including the http-user! ** MANY ** thanks for your support. I would not have figured this out without your help. Best, Ray >> Hi Simo, >> >> > > Hi Simo, >> > > >> > > > > Hi there, >> > > > > >> > > > > I'm trying to make Apache to access a kerberized document root on >> > > > > CentOS >> > > > > 7 using gssproxy. So far without success. On the web server machine >> > > > > (=NFS client) I configured a gss-proxy config file: >> > > > > >> > > > > # cat /etc/gssproxy/99-nfs-client.conf >> > > > > [service/nfs-client] >> > > > > mechs = krb5 >> > > > > cred_store = keytab:/etc/krb5.keytab >> > > > > cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U >> > > > > cred_store = client_keytab:/var/lib/gssproxy/clients/%U.keytab >> > > > > cred_usage = initiate >> > > > > allow_any_uid = yes >> > > > > trusted = yes >> > > > > euid = 0 >> > > > > >> > > > > In addition to this I set up a credentials cache >> > > > > /var/lib/gssproxy/clients/krb5cc_<httpd uid> >> > > > >> > > > What did you put in this file? >> > > >> > > Probably I made a mistake here: I got the keytab from the IPA server >> > > and >> > > named it /var/lib/gssproxy/clients/krb5cc_<httpd uid> instead of >> > > /var/lib/gssproxy/clients/<httpd_uid>.keytab. I fixed that now (by >> > > mv'ing it) and ran >> > > >> > > systemctl stop gssproxy to not have gssproxy as a >> > > daemon >> > > gssproxy -d -i -C /etc/gssproxy/ & to have gssproxy dump any debug >> > > msgs to my terminal >> > >> > Note that running gssproxy from a root shell makes it run in unconfined >> > mode, so later on the files it create may have SELinux contexts that >> > are wrong and gssproxy may fail in various ways. >> >> Ok, right now I switched SELinunx off (setenforce 0) until I have >> somehting working... >> >> > > # klist -ke <httpd uid>.keytab >> > > Keytab name: FILE:<httpd uid>.keytab >> > > KVNO Principal >> > > ---- >> > > -------------------------------------------------------------------------- >> > > 1 nfs/<nfsserver-fqdn@REALM> (aes256-cts-hmac-sha1-96) >> > > 1 nfs/<nfsserver-fqdn@REALM> (aes128-cts-hmac-sha1-96) >> > > 1 host/<nfsclient-fqdn@REALM> (aes256-cts-hmac-sha1-96) >> > > 1 host/<nfsclient-fqdn@REALM> (aes128-cts-hmac-sha1-96) >> > >> > This is not the keytab you want most probably. >> > You want a keytab with credentials that the server can map to a >> > UID/GID, usually that is done for user principals. However the server >> > can be configured to map a specific service principal name to a local >> > use as well, it's on the server side at this point. >> >> I removed the above keytab and made a new one with user credentials: >> >> # klist -ke <httpd-uid>.keytab >> Keytab name: FILE:<httpd-uid>.keytab >> KVNO Principal >> ---- >> -------------------------------------------------------------------------- >> 2 <httpd-user>@REALM (aes256-cts-hmac-sha1-96) >> 2 <httpd-user>@REALM (aes128-cts-hmac-sha1-96) >> >> This seems insufficient though, 'coz when I mount the NFS docroot >> temporarily to /mnt and then "ls -l /" as httpd-user, I get to see >> this >> line: >> >> d?????????? ? ? ? ? ? mnt >> >> Which other principals should I add to the keytab? I tried adding the >> nfs service principal to the keytab, but that doesn't change this >> behavior... Does the order in which I add principals to the keytab >> matter in this context? > > The first one is picked, and you have all you need there. > >> klist for httpd-user looks not good either, concering the validy >> timestamps (1970...1970), or is that normal?: >> >> $ klist >> Ticket cache: KEYRING:persistent:<httpd uid>:<httpd uid> >> Default principal: <httpd-user>@REALM >> >> Valid starting Expires Service principal >> 01/01/1970 01:00:00 01/01/1970 01:00:00 >> Encrypted/Credentials/v1@X-GSSPROXY: > > It is an artifact of how Gssproxy stores credentials for you (in > encrypted form), it is normal, not a problem. > >> > The (null) represents the default socket, not a bug. >> > >> > Unfortunately the default log level does not show the result of the >> > calls, but it is normal to see 2 init context calls, so it seem to be >> > working right on the gssproxy side. >> >> I think there's something not right with gssproxy: when I call it with >> --debug-level=3 it still says "level: 0": >> >> # gssproxy --debug --debug-level=3 -i -C /etc/gssproxy/ & >> [1] 17831 >> root@nfsclient:/var/lib/gssproxy/clients# [2018/02/13 09:56:20]: Debug >> Enabled (level: 0) >> >> Nevertheless the output becomes extremely verbose (plenty of hex). In >> the man-page --debug-level is not mentioned at all (ok, still better >> this way than having something promised in the manpage that does not >> exist in the binary...). > > Do you have any logs on the server side ? > As far as I can tell the client side (GSSAPI wise at least) is working > correctly. > > Simo.