[WIP] Make defer RecordOperationErrorMetrics always report the correct error code

[amacaskill]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change

/kind bug

/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:
Currently, we rely on gceCS.Metrics.RecordOperationErrorMetrics within a defer call to record error codes to the pdcsi operation error metric. We define this at the beginning of each CSI Call, but this only works if we reassign the error (err) before returning from the CSI call (CreateVolume).

Issue 1
In lots of places, we do not reassign createVolErr, and just return errors directly. This leads to a nil error being reported to RecordOperationErrorMetrics, which makes us report this error as a success (status OK) to the pdcsi operation error metric.

func (gceCS *GCEControllerServer) CreateVolume(ctx context.Context, req *csi.CreateVolumeRequest) (*csi.CreateVolumeResponse, error) {
	var err error 
	diskTypeForMetric := metrics.DefaultDiskTypeForMetric
	enableConfidentialCompute := metrics.DefaultEnableConfidentialCompute
	enableStoragePools := metrics.DefaultEnableStoragePools
	defer func() {
		gceCS.Metrics.RecordOperationErrorMetrics("CreateVolume", err, diskTypeForMetric, enableConfidentialCompute, enableStoragePools)
	}()
       ...
}

Issue 2
In some places we do something like the following. We call a function (validateVolumeCapabilities) that returns an error that doesn't specify an error code. Inline, we add on an InvalidArgument error code to the error, and return without reassigning the error. This leads to the error without the error code being reported to RecordOperationErrorMetrics, which will count as an Internal error code, when we clearly meant for it to be an InvalidArgument error code.

	err = validateVolumeCapabilities(volumeCapabilities)
	if err != nil {
		return nil, status.Errorf(codes.InvalidArgument, "VolumeCapabilities is invalid: %v", err.Error())
	}

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Make defer RecordOperationErrorMetrics always report the correct error code

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amacaskill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [amacaskill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@amacaskill: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gcp-compute-persistent-disk-csi-driver-unit	38031fe	link	true	/test pull-gcp-compute-persistent-disk-csi-driver-unit
pull-gcp-compute-persistent-disk-csi-driver-sanity	38031fe	link	true	/test pull-gcp-compute-persistent-disk-csi-driver-sanity
pull-gcp-compute-persistent-disk-csi-driver-verify	38031fe	link	true	/test pull-gcp-compute-persistent-disk-csi-driver-verify
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration	38031fe	link	true	/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration
pull-gcp-compute-persistent-disk-csi-driver-e2e	38031fe	link	true	/test pull-gcp-compute-persistent-disk-csi-driver-e2e

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten