Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export metric for validator_identity and include index #12348

Open
0xste opened this issue Apr 28, 2023 · 4 comments · May be fixed by #14473
Open

Export metric for validator_identity and include index #12348

0xste opened this issue Apr 28, 2023 · 4 comments · May be fixed by #14473
Labels
Enhancement New feature or request Good First Issue Good for newcomers Help Wanted Extra attention is needed

Comments

@0xste
Copy link

0xste commented Apr 28, 2023

🚀 Feature Request

Description

We don't currently have a way to retrieve validator index from prometheus, it's useful when joining to other metrics

Describe the solution you'd like

Introduce a metric like:
validator_identity_info{pubkey="0xa8e6154ac3be1246a5a229c8b3d9328dfc2d5da3e73bc1891da0ee9596f345b42e79f3878f879d0fa3df9ea72eb28725", index="469113"}

@0xste
Copy link
Author

0xste commented Apr 28, 2023

Code may look something like:

prysm/validator/client.go Global Var

	// ValidatorIdentityGaugeVec used to track validator identity
	ValidatorIdentityGaugeVec = promauto.NewGaugeVec(
		prometheus.GaugeOpts{
			Namespace: "validator",
			Name:      "identity_info",
			Help:      "Identifying information for the validator",
		},
		[]string{
			"pubkey", "index",
		},
	)

prysm/validator/client.go logForEachValidator(...)

ValidatorIdentityGaugeVec.WithLabelValues(pubKey, index).Set(1) 

I'm not sure we'd need to care too much de-registering exited validators too much given this is an info metric

@rauljordan
Copy link
Contributor

Hi @0xste we've talked about this before but adding the pubkey would increase the cardinality of our metrics significantly. We have tried this and it affects operators that are running a lot of keys

@0xste
Copy link
Author

0xste commented May 9, 2023

Alternatively @rauljordan, we could add validator index to an existing high-cardinality metric behind the "emitAccountMetrics" flag to reduce the need for additional active series, would this approach satisfy the cardinality non-functional requirement?

ValidatorStatusesGaugeVec = promauto.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: "validator",
Name: "statuses",
Help: "validator statuses: 0 UNKNOWN, 1 DEPOSITED, 2 PENDING, 3 ACTIVE, 4 EXITING, 5 SLASHING, 6 EXITED",
},
[]string{
"pubkey",
},
)

becomes ->

...
		[]string{
			"pubkey", "index"
		},
...

if v.emitAccountMetrics {
fmtKey := fmt.Sprintf("%#x", status.publicKey)
ValidatorStatusesGaugeVec.WithLabelValues(fmtKey).Set(float64(status.status.Status))
}

becomes ->

		if v.emitAccountMetrics {
			fmtKey := fmt.Sprintf("%#x", status.publicKey)
			ValidatorStatusesGaugeVec.WithLabelValues(fmtKey, status.index).Set(float64(status.status.Status))
		}


if v.emitAccountMetrics {
ValidatorStatusesGaugeVec.WithLabelValues(validatorNotTruncatedKey).Set(float64(duty.Status))
}

☝️getting index is a bit more involved in this instance

@0xste
Copy link
Author

0xste commented May 9, 2023

My understanding is that there's a 1:1 relationship between pubkey + index and that it's assigned on validator activation? I may be missing some context

The validator is then assigned an index number and placed into a queue for activation

https://docs.prylabs.network/docs/how-prysm-works/validator-lifecycle#pending-state

@nisdas nisdas added Enhancement New feature or request Good First Issue Good for newcomers Help Wanted Extra attention is needed labels Jul 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement New feature or request Good First Issue Good for newcomers Help Wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants