Unable to open /etc/xrootd/sss.keytab

[gusbroo]

Hello,

I'm running a setup with a standalone xrootd server which interfaces to the outside world (mainly for data subscriptions transferred using http), which is itself connected to a clustered system. As of some time ago, I can read files from the outside (ie copy out), but cannot write anymore. It's hard to tell exactly when this started as usage has peaks and large gaps. In trying to debug this, I found I can reproduce the issue on the clustered headnode using xrdcp: copy out works, but in fails. Here's what happens:
[gusbroo@xenia2]~% xrdcp test1.txt root://xenia2.nevis.columbia.edu:1094//atlas/dq2/.
Secsss (getKeyTab): Unable to open /etc/xrootd/sss.keytab; permission denied

Unable to open keytab file.
[ERROR] Server responded with an error: [3010] Unable to locate /atlas/dq2/.; permission denied

However, on initialization xrootd has no problem opening the /etc/xrootd/sss.keytab:

++++++ xroot protocol initialization started.
=====> all.export /xrootd
=====> all.export /atlas
Config warning: ignoring unknown directive 'checksum'.
=====> xrootd.checksum
=====> xrootd.seclib /usr/lib64/libXrdSec.so
Config exporting /atlas
Config exporting /xrootd
Plugin loaded secprot v5.6.8 from seclib /usr/lib64/libXrdSec-5.so
++++++ Authentication system initialization started.
Plugin loaded secsss v5.6.8 from sec.protocol /usr/lib64/libXrdSecsss-5.so
=====> sec.protocol /usr/lib64 sss -s /etc/xrootd/sss.keytab -c /etc/xrootd/sss.keytab
Plugin loaded secunix v5.6.8 from sec.protocol /usr/lib64/libXrdSecunix-5.so
=====> sec.protocol /usr/lib64 unix
Config 2 authentication directives processed in /etc/xrootd/xrootd-clustered.cfg
------ Authentication system initialization completed.

I wonder if this "Unable to open keytab file" is a red herring given that I can copy out, but if so, what is the underlying problem? A few more test outputs below.

Any ideas?

Thx

Gustaaf

-as a normal user:
[gusbroo@xenia2]~% xrdfs xenia2 ls -l /atlas
Secsss (getKeyTab): Unable to open /etc/xrootd/sss.keytab; permission denied

Unable to open keytab file.
[ERROR] Server responded with an error: [3010] Unable to locate /atlas; permission denied

-but as root:
xenia2:[09:15:21] /etc/xrootd # xrdfs xenia2 ls -l /atlas
drwxr-xr-x xrootd atlas 51 2022-07-16 12:32:15 /atlas/dq2
drwxr-xr-x geoclue atlas 62 2022-10-31 20:44:13 /atlas/dq2
drwxr-xr-x geoclue atlas 74 2022-07-16 12:32:02 /atlas/dq2
drwxr-xr-x xrootd atlas 94 2022-09-30 11:16:53 /atlas/dq2
drwxr-xr-x xrootd atlas 119 2022-09-30 11:16:06 /atlas/dq2
drwxr-xr-x xrootd atlas 4096 2022-07-16 12:31:30 /atlas/dq2

however, as root:

xenia2:[09:16:05] /etc/xrootd # xrdcp ~gusbroo/test1.txt root://xenia2.nevis.columbia.edu:1094//atlas/dq2/.
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Server responded with an error: [3010] Unable to create /atlas/dq2/test1.txt; permission denied (destination)