Add receiver downscale endpoint #88
Conversation
yuchen-db
requested review from
a team,
christopherzli,
jnyi,
hczhu-db and
yulong-db
and removed request for
a team
pkg/server/http/http.go
Outdated
| @@ -17,6 +15,11 @@ import ( | |||
| toolkit_web "github.com/prometheus/exporter-toolkit/web" | |||
| "golang.org/x/net/http2" | |||
| "golang.org/x/net/http2/h2c" | |||
| "net/http" | |||
There was a problem hiding this comment.
please fix the import convention for go: https://google.github.io/styleguide/go/best-practices.html#import-ordering
cmd/thanos/receive.go
Outdated
| @@ -311,6 +311,8 @@ func runReceive( | |||
| httpserver.WithGracePeriod(time.Duration(*conf.httpGracePeriod)), | |||
| httpserver.WithTLSConfig(*conf.httpTLSConfig), | |||
| ) | |||
| var lastDownscalePrepareTimestamp *int64 = nil | |||
There was a problem hiding this comment.
this implementation assumes the pod won't restart, to preserve the state, we might need to persist it on local PVC too, any thoughts?
There was a problem hiding this comment.
yes but it's safe to lose this state. rollout operator will only do downscale if the returned timestamp is more than 30 seconds ago. If we lose this state, it returns the current timestamp and downscale will not accidentally happen. In the worst case, a scheduled downscale will be delayed for 30 seconds.
pkg/receive/multitsdb.go
Outdated
| @@ -106,6 +106,14 @@ func (t *MultiTSDB) SkipMatchExternalLabels() { | |||
| t.skipMatchExternalLabels = true | |||
| } | |||
|
|
|||
| func (t *MultiTSDB) GetTenants() map[string]*tenant { | |||
| return t.tenants | |||
There was a problem hiding this comment.
This map is protected by mtx. A call site of GetTenants() can hold a pointer to the map and do reads/writes while the map is being updated by other goroutines. That'd be a data race.
It's safer to return a deep copy of the map if copying is not too expensive.
pkg/server/http/http.go
Outdated
| func RegisterDownscale[K comparable, V any](s *Server, m map[K]V, mtx *sync.RWMutex, t *int64) { | ||
| s.mux.Handle("/-/downscale", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { | ||
| mtx.RLock() | ||
| n := len(m) |
There was a problem hiding this comment.
Don't pass in a map, if only the size of map is needed. Pass in the size of the map.
There was a problem hiding this comment.
the size of the map changes dynamically during handler runtime and Golang passes map by reference
yuchen-db
force-pushed
the
yuchen-db/scaledown-with-operator
branch
from
cf878f5
to
ea0d891
Compare
Signed-off-by: Yuchen Wang <[email protected]>
pkg/receive/multitsdb.go
Outdated
| func (t *MultiTSDB) GetTenantsLen() int { | ||
| t.mtx.RLock() | ||
| defer t.mtx.RUnlock() | ||
| return len(t.tenants) |
There was a problem hiding this comment.
shall we log how many tenants left and what are they?
cmd/thanos/receive.go
Outdated
| @@ -7,8 +7,10 @@ import ( | |||
| "context" | |||
| "fmt" | |||
| "net" | |||
| "net/http" | |||
The endpoint expose number of TSDBs, rollout operator will patch the sts if n_tsdb=0
related PRs:
https://github.com/databricks/universe/pull/753653
databricks/rollout-operator#7