New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Component #1823

Merged

mattdurham merged 48 commits into dev.new-wal from wal_component

Oct 15, 2024

Collaborator

mattdurham commented Oct 3, 2024 •

edited

Loading

This is the last PR, aside from the merge to main. This ties everything together and exposes the component. Most of this are very expansive tests. I am likely to change the metrics in subsequent PRs so dont get to in detail about them.

Things to do that I will do after this large merge and create tickets for each.

Add support for TLS
Rethink the metrics and the approach.
Add metadata support, this requires exposing additional properties on the scrape itself.
Add hook to not use labelstore if not running any remote_write components.

mattdurham and others added 12 commits

September 19, 2024 13:25


          Checkin the networking items.

a5cf26d


          Fix for config updating and tests.

9e84aee


          Update internal/component/prometheus/remote/queue/network/loop.go

1cf1e7a

Co-authored-by: William Dumont <[email protected]>


          Update internal/component/prometheus/remote/queue/network/loop.go

5c49e9e

Co-authored-by: Piotr <[email protected]>


          pr feedback

02a41e0


          Merge branch 'wal_network' of github.com:grafana/alloy into wal_network

8ab6a26


          pr feedback

a638c1a


          simplify stats

5abe271


          simplify stats

2d0eb00


          Initial push.

46e1764


          dev.new-wal merge

1e12b5e


          docs and some renaming

254dc4c

mattdurham requested a review from clayton-cornell as a code owner

October 3, 2024 19:44

clayton-cornell reviewed

View reviewed changes

Contributor

clayton-cornell left a comment

First high level pass of the docs. I'll go through again once the ToDos are added

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

mattdurham marked this pull request as draft

October 4, 2024 03:10

mattdurham and others added 15 commits

October 4, 2024 08:55


          Update docs/sources/reference/components/prometheus/prometheus.remote…

e06efff

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Update docs/sources/reference/components/prometheus/prometheus.remote…

992c703

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Update docs/sources/reference/components/prometheus/prometheus.remote…

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Update docs/sources/reference/components/prometheus/prometheus.remote…

e75e34f

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Update docs/sources/reference/components/prometheus/prometheus.remote…

0ea1f71

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Update docs/sources/reference/components/prometheus/prometheus.remote…

9a90c2b

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Update docs/sources/reference/components/prometheus/prometheus.remote…

f04edb4

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Changes and testing.

0d99288


          Update docs.

ed64bc3


          Update docs.

98bc887


          Fix race conditions in unit tests.

2e32ce6


          Tweaking unit tests.

e1aaa9f


          lower threshold more.

38b15a1


          lower threshold more.

6f9a820


          Fix deadlock in manager tests.

c78ea1d

mattdurham and others added 5 commits

October 9, 2024 08:27


          Update docs/sources/reference/components/prometheus/prometheus.remote…

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Update docs/sources/reference/components/prometheus/prometheus.remote…

d5568d9

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Update docs/sources/reference/components/prometheus/prometheus.remote…

d8cd012

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Update docs/sources/reference/components/prometheus/prometheus.remote…

42fbdd9

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>


          Update docs/sources/reference/components/prometheus/prometheus.remote…

d5eb26e

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>

clayton-cornell reviewed

View reviewed changes

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

clayton-cornell approved these changes

View reviewed changes

Contributor

clayton-cornell left a comment

Approving for docs.


          Update docs/sources/reference/components/prometheus/prometheus.remote…

0c9e755

….queue.md

Co-authored-by: Clayton Cornell <[email protected]>

ptodev reviewed

View reviewed changes

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated

Contributor

ptodev Oct 10, 2024

It'd be nice to add a bit more info on how prometheus.remote.queue is different from prometheus.remote_write, and when to use what component.

internal/component/prometheus/remote/queue/component.go Outdated Show resolved Hide resolved

thampiotr reviewed

View reviewed changes

Contributor

thampiotr left a comment

I focused on the user-facing API as this will be harder to change without breaking things.

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated Show resolved Hide resolved

docs/sources/reference/components/prometheus/prometheus.remote.queue.md Outdated

+              Name | Type | Description | Default | Required
+              ---- | ---- | ----------- | ------- | --------
+              `max_signals_to_batch` | `uint` | The maximum number of signals before they are batched to disk. | `10,000` | no
+              `batch_frequency` | `duration` | How often to batch signals to disk if `max_signals_to_batch` is not reached. | no

Contributor

thampiotr Oct 11, 2024

Why do we use signals as a name? maybe it should be samples? or is this to encompass histograms samples too? I'm not sure if that's a right name. In current remote_write we call this 'metrics' which makes sense although it's not very precise: https://grafana.com/docs/alloy/next/reference/components/prometheus/prometheus.remote_write/#wal-block

Collaborator Author

mattdurham Oct 11, 2024

Subsequent PR will enable metadata storage, so I wanted it to not be just metrics. Sample is to narrow, open to better names here.

Contributor

thampiotr Oct 14, 2024

maybe 'entries'? that would be a generic name for things inside a queue.

internal/component/prometheus/remote/queue/component.go

Comment on lines +99 to +111

+              	if len(s.endpoints) > 0 {
+              		for _, ep := range s.endpoints {
+              			ep.Stop()
+              		}
+              		s.endpoints = map[string]*endpoint{}
+              	}
+              	err := s.createEndpoints()
+              	if err != nil {
+              		return err
+              	}
+              	for _, ep := range s.endpoints {
+              		ep.Start()
+              	}

Contributor

thampiotr Oct 11, 2024

Yeah, maybe Update can signal via channel to Run that will manage everything in one thread. I found this to be a decent pattern in other components.

internal/component/prometheus/remote/queue/endpoint.go Outdated Show resolved Hide resolved

internal/component/prometheus/remote/queue/endpoint.go Outdated Show resolved Hide resolved

internal/component/prometheus/remote/queue/network/manager.go Outdated Show resolved Hide resolved

internal/component/prometheus/remote/queue/network/manager.go Outdated Show resolved Hide resolved

mattdurham and others added 8 commits

October 11, 2024 10:30


          Update docs/sources/reference/components/prometheus/prometheus.remote…

ce0ecb0

….queue.md

Co-authored-by: Paulin Todev <[email protected]>


          Docs PR feedback

872de53


          Merge remote-tracking branch 'origin/wal_component' into wal_component

0506c37


          Update docs/sources/reference/components/prometheus/prometheus.remote…

db5bd6a

….queue.md

Co-authored-by: Piotr <[email protected]>


          PR feedback

3ee51b3


          Merge remote-tracking branch 'origin/wal_component' into wal_component

98872f8


          PR feedback

bd2d083


          PR feedback

b2d5cab

wildum reviewed

View reviewed changes

internal/component/prometheus/remote/queue/types.go Outdated Show resolved Hide resolved

mattdurham added 5 commits

October 15, 2024 08:48


          PR feedback

57d81b0


          Fix typo

21952d8


          Fix typo

5966b6d


          Fix bug.

561bfeb


          Fix docs

9dc25f1

Collaborator Author

mattdurham commented Oct 15, 2024

Most of the feedback has been responded so merging.

mattdurham merged commit 079c3a9 into dev.new-wal

16 of 17 checks passed

mattdurham deleted the wal_component branch

October 15, 2024 13:36

mattdurham added a commit that referenced this pull request


          Merge prometheus.write.queue into main. (#1564)

eb1c840

* readme

* fix readme

* Add filequeue functionality (#1601)

* Checkin for file queue

* add comment

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* naming and error handling feedback from PR

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/filequeue/filequeue.go

Co-authored-by: Piotr <[email protected]>

* drop benchmark

* rename get to pop

---------

Co-authored-by: Piotr <[email protected]>

* Adding the serialization features. (#1666)

* Adding the serialization features.

* Dont test this with race condition since we access vars directly.

* Fix test.

* Fix typo in file name and return early in DeserializeToSeriesGroup.

* Update internal/component/prometheus/remote/queue/serialization/appender.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/serialization/serializer.go

Co-authored-by: Piotr <[email protected]>

* Rename to indicate that TimeSeries are Put/Get from a pool.

* Remove func that was about the same number of lines as inlining.

* Update internal/component/prometheus/remote/queue/types/serialization.go

Co-authored-by: Piotr <[email protected]>

* Update internal/component/prometheus/remote/queue/serialization/serializer.go

Co-authored-by: Piotr <[email protected]>

* Change benchmark to be more specific.

---------

Co-authored-by: Piotr <[email protected]>

* Network wal pr (#1717)

* Checkin the networking items.

* Fix for config updating and tests.

* Update internal/component/prometheus/remote/queue/network/loop.go

Co-authored-by: William Dumont <[email protected]>

* Update internal/component/prometheus/remote/queue/network/loop.go

Co-authored-by: Piotr <[email protected]>

* pr feedback

* pr feedback

* simplify stats

* PR feedback

---------

Co-authored-by: William Dumont <[email protected]>
Co-authored-by: Piotr <[email protected]>

* Component (#1823)

* Checkin the networking items.

* Fix for config updating and tests.

* Update internal/component/prometheus/remote/queue/network/loop.go

Co-authored-by: William Dumont <[email protected]>

* Update internal/component/prometheus/remote/queue/network/loop.go

Co-authored-by: Piotr <[email protected]>

* pr feedback

* pr feedback

* simplify stats

* simplify stats

* Initial push.

* docs and some renaming

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Changes and testing.

* Update docs.

* Update docs.

* Fix race conditions in unit tests.

* Tweaking unit tests.

* lower threshold more.

* lower threshold more.

* Fix deadlock in manager tests.

* rollback to previous

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Clayton Cornell <[email protected]>

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Paulin Todev <[email protected]>

* Docs PR feedback

* Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md

Co-authored-by: Piotr <[email protected]>

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* Fix typo

* Fix typo

* Fix bug.

* Fix docs

---------

Co-authored-by: William Dumont <[email protected]>
Co-authored-by: Piotr <[email protected]>
Co-authored-by: Clayton Cornell <[email protected]>
Co-authored-by: Paulin Todev <[email protected]>

* Change name to write instead of remote.

* Fix issue.

* Fix issue.

* Dont depend on random sync.pool behavior.

* small clarification on changelog.

* PR feedback

---------

Co-authored-by: Piotr <[email protected]>
Co-authored-by: William Dumont <[email protected]>
Co-authored-by: Clayton Cornell <[email protected]>
Co-authored-by: Paulin Todev <[email protected]>

github-actions bot added the frozen-due-to-age label

github-actions bot locked as resolved and limited conversation to collaborators

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

frozen-due-to-age