Backup of large volume fails

[miguelcostaUI]

I have a volume that has about 150GB and the backup of it fails.

What steps did you take and what happened:

Using this VolumeSnapshotLocation:

apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: openebs.io/cstor-blockstore
  config:
    bucket: velero
    prefix: cstor
    provider: aws
    region: minio
    s3ForcePathStyle: "true"
    s3Url: http://10.0.1.221:9000
    restoreAllIncrementalSnapshots: "true"
    autoSetTargetIP: "true"

velero create backup backup-test-cstor-2

Backup is created but fails to upload caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000).

velero backup logs backup-test-cstor-2

time="2023-05-19T05:02:27Z" level=warning msg="Failed to close file interface : blob (code=Unknown): MultipartUpload: upload multipart failed\n\tupload id: YjQ1ZWE0ODAtN2Q5MS00ZDkyLTg5NDgtMjU5MDZiY2YzMjE0LmJhMDkzODUxLWEzM2ItNDRjYi1hOTdjLWVlMDMxMGEyNTVhNQ\ncaused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit" backup=velero/backup-test-cstor-3 cmd=/plugins/velero-blockstore-openebs logSource="/go/src/github.com/openebs/velero-plugin/pkg/clouduploader/conn.go:322" pluginName=velero-blockstore-openebs
time="2023-05-19T05:02:37Z" level=error msg="Error backing up item" backup=velero/backup-test-cstor-3 error="error taking snapshot of volume: rpc error: code = Unknown desc = Failed to upload snapshot, status:{Failed}" logSource="pkg/backup/backup.go:435" name=influxdb-influxdb2-0

This is strange since I thought the multiPartChunkSize was calculated from the file size.

Then I tried defining the multiPartChunkSize.

apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: openebs.io/cstor-blockstore
  config:
    bucket: velero
    prefix: cstor
    provider: aws
    region: minio
    s3ForcePathStyle: "true"
    s3Url: http://10.0.1.221:9000
    multiPartChunkSize: 64Mi
    restoreAllIncrementalSnapshots: "true"
    autoSetTargetIP: "true"

velero create backup backup-test-cstor-3

But with this the backup just fails with another error that is not very informative.

velero backup logs backup-test-cstor-3

time="2023-05-18T09:20:03Z" level=info msg="1 errors encountered backup up item" backup=velero/backup-test-cstor-2 logSource="pkg/backup/backup.go:431" name=influxdb-influxdb2-0
time="2023-05-18T09:20:03Z" level=error msg="Error backing up item" backup=velero/backup-test-cstor-2 error="error taking snapshot of volume: rpc error: code = Unavailable desc = error reading from server: EOF" logSource="pkg/backup/backup.go:435" name=influxdb-influxdb2-0

Is there anything that I'm missing to make the backup of large volumes work?

What did you expect to happen:
Backup to succeed and upload successfully.

Anything else you would like to add:
I'm also receiving a lot of these warnings and I'm not sure what they are or how to fix them.

time="2023-05-18T09:20:03Z" level=warning msg="Epoll wait failed : interrupted system call" backup=velero/backup-test-cstor-2 cmd=/plugins/velero-blockstore-openebs logSource="/go/src/github.com/openebs/velero-plugin/pkg/clouduploader/server.go:302" pluginName=velero-blockstore-openebs

Environment:

• Velero version (use velero version):

Client:
	Version: v1.11.0
	Git commit: -
Server:
	Version: v1.11.0

• Velero features (use velero client config get features):

features: <NOT SET>

• Velero-plugin version

v3.4.0

• OpenEBS version

NAME   	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART        	APP VERSION
openebs	openebs  	3       	2023-03-01 13:21:27.563122 +0000 UTC	deployed	openebs-3.4.1	3.4.0 

• Kubernetes version (use kubectl version):

Client Version: v1.27.1
Kustomize Version: v5.0.1
Server Version: v1.26.4

• Kubernetes installer & version:

MicroK8s v1.26.4 revision 5219

• OS (e.g. from /etc/os-release):

PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy