Any Limitation on number of mounts of a storage container on different servers using blobfuse2 ?

Himanshu Singh 0 Reputation points
2024-10-04T22:53:13.2066667+00:00

I have 12 on-prem servers at different locations and they all run k3s on it. I have mounted the same Azure storage account container on all these servers using blobfuse2 (Documentation referred). Please note that all 12 servers use the same container SAS token to mount the same container.
All these servers act as individual clusters (makes it 12 clusters). Each cluster has 3 pods responsible for writing data on azure blob container. All three pods write data in their unique directory.

Problem:
The above setup was working fine on 10 servers/clusters which crashed after 2 weeks of live run with the below error. As of today, the above setup is running only on 5-7 servers at a time.

Logs from csi-blob-node pod and container named blob :

I1002 15:47:26.575930 13323 nodeserver.go:127] NodePublishVolume: volume mystorage_mycontainer1 mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount at /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount with mountOptions: [bind]

I1002 15:47:26.575943 13323 mount_linux.go:218] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount)

E1002 15:47:26.577082 13323 mount_linux.go:230] Mount failed: exit status 32

Mounting command: mount

Mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount

Output: mount: /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount: special device /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount does not exist. DRIVER INFORMATION:

Build Date: "2023-10-19T11:51:15Z"

Compiler: gc

Driver Name: blob.csi.azure.com

Driver Version: v1.23.1

Git Commit: d6e08f4b0bcddbe42db9169c1bf9461d25b91093

Go Version: go1.21.1

Platform: linux/amd64

Is there a limit on number of read-write mounts for a single container using blobfuse2 ? Can you please help me in identifying the root cause of the problem?

Any help would be much appreciated.

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,147 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,843 questions
0 comments No comments
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.