Investigate best way to do scaling of bedrock pods

[duallain]

It seems like cpu/memory may not be the best signal to know when to spin up new bedrock instances. If we choose a metric that is better fitted to bedrock's capabilities we can likely deploy fewer pods, and need less k8s infrastructure resources. We should investigate a few items:

1. How do we expose additional metrics to HPA objects. (There are quite a few choices here, what's best?)
2. What metric is most reflective of load on bedrock (this implies some perf testing)
3. (generate stories for) deploying new metrics for HPA
4. (generate stories for) deploying bedrock prod/stg/dev with the new HPA metrics
5. Test the hpa works, ensure latency between origin/cdn doesn't change.

[bookshelfdave]

see also: https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler

[duallain]

https://github.com/FairwindsOps/goldilocks can maybe help with tuning the VPA. It seems like you basically install goldilocks + vpa and then get changes recommended. Which is rad as all get out.

[bookshelfdave]

let's give Goldilocks a shot!