Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3's bucket cannot mount into a pod in k8s. #250

Open
Trainbow opened this issue May 6, 2023 · 4 comments
Open

S3's bucket cannot mount into a pod in k8s. #250

Trainbow opened this issue May 6, 2023 · 4 comments

Comments

@Trainbow
Copy link

Trainbow commented May 6, 2023

Hello, I tried datashim in the local k8s which has a master node and it works well.
But when I apply the datashim in another kubernetes(1master+10nodes) with the same way, The pvc can create successfully but cannot mount into a nginx pod.
This is the log I found in a csi-s3 pod:

I0506 01:42:52.653413       1 s3-driver.go:81] Driver: ch.ctrox.csi.s3-driver 
I0506 01:42:52.653734       1 s3-driver.go:82] Version: v1.1.1 
I0506 01:42:52.653992       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0506 01:42:52.654008       1 driver.go:93] Enabling volume access mode: SINGLE_NODE_WRITER
I0506 01:42:52.655539       1 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I0506 01:42:53.358925       1 utils.go:97] GRPC call: /csi.v1.Identity/GetPluginInfo
I0506 01:42:53.358995       1 utils.go:98] GRPC request: {}
I0506 01:42:53.363472       1 identityserver-default.go:32] Using default GetPluginInfo
I0506 01:42:53.363499       1 utils.go:103] GRPC response: {"name":"ch.ctrox.csi.s3-driver","vendor_version":"v1.1.1"}
I0506 01:42:53.449295       1 utils.go:97] GRPC call: /csi.v1.Identity/Probe
I0506 01:42:53.449356       1 utils.go:98] GRPC request: {}
I0506 01:42:53.449481       1 utils.go:103] GRPC response: {}
I0506 01:42:53.452289       1 utils.go:97] GRPC call: /csi.v1.Identity/GetPluginInfo
I0506 01:42:53.452343       1 utils.go:98] GRPC request: {}
I0506 01:42:53.452504       1 identityserver-default.go:32] Using default GetPluginInfo
I0506 01:42:53.452519       1 utils.go:103] GRPC response: {"name":"ch.ctrox.csi.s3-driver","vendor_version":"v1.1.1"}
I0506 01:42:53.454342       1 utils.go:97] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0506 01:42:53.454391       1 utils.go:98] GRPC request: {}
I0506 01:42:53.454635       1 identityserver-default.go:53] Using default capabilities
I0506 01:42:53.454656       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}}]}
I0506 01:42:53.457727       1 utils.go:97] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0506 01:42:53.457778       1 utils.go:98] GRPC request: {}
I0506 01:42:53.457839       1 controllerserver-default.go:62] Using default ControllerGetCapabilities
I0506 01:42:53.457850       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 01:42:53.985399       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetInfo
I0506 01:42:53.985492       1 utils.go:98] GRPC request: {}
I0506 01:42:53.985640       1 nodeserver-default.go:40] Using default NodeGetInfo
I0506 01:42:53.985663       1 utils.go:103] GRPC response: {"node_id":"js-yangzhou-7.172.18.0.95"}
I0506 02:00:34.904110       1 utils.go:97] GRPC call: /csi.v1.Controller/CreateVolume
I0506 02:00:34.904591       1 utils.go:98] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead","parameters":{"mounter":"goofys"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
I0506 02:00:34.907130       1 controllerserver.go:72] Got a request to create volume pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead
I0506 02:00:34.926703       1 controllerserver.go:105] create volume pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead
I0506 02:00:34.926763       1 utils.go:103] GRPC response: {"volume":{"capacity_bytes":10000000000000,"volume_context":{"mounter":"goofys"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}}
I0506 02:51:37.884415       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:37.884653       1 utils.go:98] GRPC request: {}
I0506 02:51:37.885203       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:37.897327       1 utils.go:97] GRPC call: /csi.v1.Node/NodeStageVolume
I0506 02:51:37.897356       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
I0506 02:51:37.901213       1 utils.go:103] GRPC response: {}
I0506 02:51:37.907690       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:37.907744       1 utils.go:98] GRPC request: {}
I0506 02:51:37.907883       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:37.914580       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:37.914641       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
I0506 02:51:37.915453       1 nodeserver.go:75] target /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount
device 
readonly false
volumeId pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead
attributes map[mounter:goofys storage.kubernetes.io/csiProvisionerIdentity:1683337373458-8081-ch.ctrox.csi.s3-driver]
mountflags []
I0506 02:51:37.915757       1 mounter_goofys.go:26] Mounting using goofys volume pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead
I0506 02:51:37.915780       1 mounter_goofys.go:53] Mounting using goofys!
I0506 02:51:37.916581       1 mounter.go:56] Mounting fuse with command: goofys and args: [--endpoint=http://minio-service:9000 --type-cache-ttl 1s --stat-cache-ttl 1s --dir-mode 0777 --file-mode 0777 --http-timeout 5m -o allow_other --profile=pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead test /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount]
E0506 02:51:39.066557       1 utils.go:101] GRPC error: stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:51:39.592184       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:39.592245       1 utils.go:98] GRPC request: {}
I0506 02:51:39.592380       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:39.599875       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:39.599939       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:51:39.600537       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:51:40.696793       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:40.696860       1 utils.go:98] GRPC request: {}
I0506 02:51:40.696956       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:40.702772       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:40.702844       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:51:40.703246       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:51:42.707661       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:42.707728       1 utils.go:98] GRPC request: {}
I0506 02:51:42.707837       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:42.715232       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:42.715278       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:51:42.715697       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:51:46.731186       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:46.731374       1 utils.go:98] GRPC request: {}
I0506 02:51:46.731704       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:46.738983       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:46.739047       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:51:46.739843       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:51:54.775237       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:54.775298       1 utils.go:98] GRPC request: {}
I0506 02:51:54.775471       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:54.780254       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:54.780330       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:51:54.780787       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:52:10.863501       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:52:10.865249       1 utils.go:98] GRPC request: {}
I0506 02:52:10.865393       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:52:10.872299       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:52:10.872365       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:52:10.873118       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:52:42.934041       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:52:42.934107       1 utils.go:98] GRPC request: {}
I0506 02:52:42.934211       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:52:42.941844       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:52:42.941897       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:52:42.942286       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:53:46.969583       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:53:46.969950       1 utils.go:98] GRPC request: {}
I0506 02:53:46.970224       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:53:46.975533       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:53:46.975585       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:53:46.976374       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:55:48.997553       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:55:48.997620       1 utils.go:98] GRPC request: {}
I0506 02:55:48.997733       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:55:49.006357       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:55:49.006543       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:55:49.007343       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:57:51.068016       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:57:51.068091       1 utils.go:98] GRPC request: {}
I0506 02:57:51.068226       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:57:51.074388       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:57:51.074464       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:57:51.075257       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:59:53.120890       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:59:53.120953       1 utils.go:98] GRPC request: {}
I0506 02:59:53.121059       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:59:53.128886       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:59:53.128971       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:59:53.129648       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 03:01:55.177258       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 03:01:55.177323       1 utils.go:98] GRPC request: {}
I0506 03:01:55.177481       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 03:01:55.183865       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 03:01:55.183916       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 03:01:55.184759       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 03:03:57.236970       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 03:03:57.237205       1 utils.go:98] GRPC request: {}
I0506 03:03:57.237404       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 03:03:57.245216       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 03:03:57.245274       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 03:03:57.245934       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 03:05:59.299269       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 03:05:59.300758       1 utils.go:98] GRPC request: {}
I0506 03:05:59.300887       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 03:05:59.308847       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 03:05:59.308915       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 03:05:59.309778       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 03:08:01.355651       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 03:08:01.355717       1 utils.go:98] GRPC request: {}
I0506 03:08:01.355819       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 03:08:01.364002       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 03:08:01.364073       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 03:08:01.364949       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 03:10:03.412736       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 03:10:03.414149       1 utils.go:98] GRPC request: {}
I0506 03:10:03.414329       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 03:10:03.420006       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 03:10:03.420044       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 03:10:03.420593       1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused

What caused this? Thanks for help.
@srikumar003
Copy link
Collaborator

@Trainbow check if the namespace in which you are creating the datasets is being watched by Datashim. See this link for an explanation

@Trainbow
Copy link
Author

Trainbow commented May 9, 2023 •
edited
Loading

@srikumar003 Thank you for your prompt reply, I tried to see the label of "monitor-pods-datasets" and it was set to "enabled."

kubectl get ns dlf --show-labels | grep monitor-pods-datasets
dlf    Active   3d3h   app.kubernetes.io/name=dlf,monitor-pods-datasets=enabled

But the pod still cannot mount the pvc, and the pod displays the following error information:

Events:
  Type     Reason       Age                  From     Message
  ----     ------       ----                 ----     -------
  Warning  FailedMount  24m (x466 over 22h)  kubelet  Unable to attach or mount volumes: unmounted volumes=[source], unattached volumes=[source default-token-w8hn4]: timed out waiting for the condition
  Warning  FailedMount  15m (x121 over 22h)  kubelet  Unable to attach or mount volumes: unmounted volumes=[source], unattached volumes=[default-token-w8hn4 source]: timed out waiting for the condition
  Warning  FailedMount  3s (x671 over 22h)   kubelet  MountVolume.SetUp failed for volume "pvc-89f0da6c-cd62-4d8c-97eb-00254fa77294" : rpc error: code = Internal desc = stat /var/lib/kubelet/pods/175e4c0f-78f5-4e16-8719-0031237b0829/volumes/kubernetes.io~csi/pvc-89f0da6c-cd62-4d8c-97eb-00254fa77294/mount: connection refused

@srikumar003
Copy link
Collaborator

@Trainbow
Could you verify that you have the correct endpoint for the object storage service ? Protocol (http:// vs 'https://'), Port, etc.

Since this error appears in the mounter rather than the actual connector, it could be related to #160. Can you check if the kubelet path that you used for installing Datashim is the same as that in your cluster ?

@Trainbow
Copy link
Author

@srikumar003
I tried the same install steps in the local docker-desktop k8s, and it works, so I think the minio's endpoint is correct.

Do you mean that the datashim's plugin didn't install in the correct path in the k8s cluster?
I exec a pod which named "csi-s3-xxxxx", and found the globalmount path is empty.
image

It seems installed in the correct path?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@srikumar003 @Trainbow