feat(new sink): initial `websocket_server` sink #22213

esensar · 2025-01-15T14:17:35Z

Summary

This adds a new sink, similar to websocket which acts like a client, while this one acts as a server. It currently broadcasts messages to all currently connected clients.

It might make sense to add some kind of buffer and allow clients to catch up to missed messages, or something like that, but I wanted to leave such complications for further PRs.

This also adds customizable authentication using VRL, which would allow us to use custom VRL scripts to check auth, meaning we no longer use hardcoded basic auth and can use enrichment tables to look for credentials.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

How did you test this PR?

Started up vector with the following configuration:

sources:
  demo_logs_test:
    type: "demo_logs"
    format: "json"

sinks:
  websocket_sink:
    inputs: ["demo_logs_test"]
    type: "websocket_listener"
    address: "0.0.0.0:1234"
    encoding:
      codec: "json"

Connected to the server with websocat and observed that events are sent to the client. I have also tested auth in this way, by defining auth header using websocat.

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

Please read our Vector contributor resources.
- make check-all is a good command to run locally. This check is
  defined here. Some of these
  checks might not be relevant to your PR. For Rust changes, at the very least you should run:
  - cargo fmt --all
  - cargo clippy --workspace --all-targets -- -D warnings
  - cargo nextest run --workspace (alternatively, you can run cargo test --all)
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

This adds a new sink, similar to `websocket` which acts like a client, while this one acts as a server. It currently broadcasts messages to all currently connected clients.

esensar · 2025-01-15T14:19:05Z

I have added customizable auth handler in this PR, but I can take it out into a separate PR if needed.

This is also missing some tests and some metrics. I will probably also work on an optional event buffer to enable replays for some clients. That one might be more complex, so I wanted to put it off for now, to keep the initial PR simple.

pront · 2025-01-15T19:07:16Z

I have added customizable auth handler in this PR, but I can take it out into a separate PR if needed.
Are you referring to src/common/server_auth.rs? I also wonder if that code adds duplication with other http components.

If it's not too much effort, please do. It should make reviewing faster. It's just me reviewing PRs for this month 😅

esensar · 2025-01-15T19:18:15Z

I have added customizable auth handler in this PR, but I can take it out into a separate PR if needed.
Are you referring to src/common/server_auth.rs? I also wonder if that code adds duplication with other http components.

If it's not too much effort, please do. It should make reviewing faster. It's just me reviewing PRs for this month 😅

Oh 😄
Alright, I will do it, it should be pretty simple to move out.

esensar · 2025-01-15T19:22:53Z

Oh and sorry, I just saw the other part of the comment. Yeah, it causes duplication, my idea was to reuse it across these components, but didn't want to make too many changes in this PR. I guess if I move it to a separate PR, then changing these out should be okay.

This adds `custom` auth strategy for components with HTTP server (`http_server`, `datadog_agent`, `opentelemetry`, `prometheus`) besides the default basic auth. This is a breaking change because `strategy` is now required for auth - for existing configurations `strategy: "basic"` needs to be added. Related: vectordotdev#22213

esensar · 2025-01-21T11:48:12Z

Would it make more sense to name this websocket_server? Not sure why I named it listener to be honest. I think server would be more consistent with other components.

jszwedko · 2025-01-21T20:32:03Z

Would it make more sense to name this websocket_server? Not sure why I named it listener to be honest. I think server would be more consistent with other components.

👍 websocket_server would be consistent with http_server. It leaves the door open for a websocket_client source that, like the http_client source, would initiate requests rather than listen for them.

website/cue/reference/components/sinks/base/websocket_server.cue

Co-authored-by: Sandra (neko) <[email protected]>

esensar · 2025-01-28T13:23:22Z

@neko-dd sorry for the late reply. I have applied one of the suggestions, which was directly related to things added in this PR. The rest of it has been generated automatically from other things that I have used (which are standard for all sinks) and you can see that documentation already live on almost all of the sinks. Maybe these should be fixed in a separate PR? Because if I fixed the here, I would generate too many changes that are not really related to this (it would change docs for all sinks).

…er (#22236) * feat(sources)!: add custom auth strategy for components with HTTP server This adds `custom` auth strategy for components with HTTP server (`http_server`, `datadog_agent`, `opentelemetry`, `prometheus`) besides the default basic auth. This is a breaking change because `strategy` is now required for auth - for existing configurations `strategy: "basic"` needs to be added. Related: #22213 * Generate components docs * Add changelog entry * Rename `user` to `username` to match old config * Apply docs suggestions from code review Co-authored-by: Esther Kim <[email protected]> * Update all docs based on suggestions * Prevent breaking change by defaulting to `basic` strategy for `HttpServerAuthConfig` * Remove breaking change flag on changelog entry * Add missing docs to `common/http/error` * Make tests clearer by using `panic` instead of matching enum variant * Move out `VRL` auth handling into a separate function for readability * Clean up changelog text Co-authored-by: Pavlos Rontidis <[email protected]> * Move test only functions in `server_auth` to `tests` module * Return serde tag for correct docs generation --------- Co-authored-by: Esther Kim <[email protected]> Co-authored-by: Pavlos Rontidis <[email protected]>

esensar · 2025-02-18T08:43:40Z

@pront is this okay to merge? I have updated it with changes from #22236

pront · 2025-02-18T19:15:09Z

@pront is this okay to merge? I have updated it with changes from #22236

Hi @esensar, there are open comments around docs and also I need to do another review before merging.

esensar · 2025-02-18T20:47:46Z

@pront is this okay to merge? I have updated it with changes from #22236

Hi @esensar, there are open comments around docs and also I need to do another review before merging.

I have added the docs changes in a separate PR: #22317

pront · 2025-02-18T21:49:27Z

I have added the docs changes in a separate PR: #22317

Ah yes, this is coming back to me 👍. Did we merge the master with those changes in this PR? If yes, please resolve those comments.

src/internal_events/websocket_server.rs

esensar · 2025-02-19T09:11:16Z

I have added the docs changes in a separate PR: #22317

Ah yes, this is coming back to me 👍. Did we merge the master with those changes in this PR? If yes, please resolve those comments.

Yes, but now looking at them, it seems that I have missed some of them. I will open up another PR with these.

Co-authored-by: Pavlos Rontidis <[email protected]>

esensar · 2025-02-19T09:25:40Z

I have added the docs changes in a separate PR: #22317

Ah yes, this is coming back to me 👍. Did we merge the master with those changes in this PR? If yes, please resolve those comments.

Yes, but now looking at them, it seems that I have missed some of them. I will open up another PR with these.

Actually, it seems that I just forgot to regenerate documentation. It should all be alright now.

pront · 2025-02-19T17:59:58Z

Wonderful, happy to see this merged. Optional request @esensar: can we add a "How it works" section with the steps you outlined in the test plan? The community has been finding these useful.

esensar · 2025-02-19T18:04:00Z

Wonderful, happy to see this merged. Optional request @esensar: can we add a "How it works" section with the steps you outlined in the test plan? The community has been finding these useful.

Of course! I will add that in tomorrow. Do you think something like that would be useful for #22236 too? Or is the included documentation enough.

Also, I planned on opening more PRs related to this sink in the following days and I think these could benefit even more from that section (if they are accepted of course).

pront · 2025-02-19T18:32:02Z

Wonderful, happy to see this merged. Optional request @esensar: can we add a "How it works" section with the steps you outlined in the test plan? The community has been finding these useful.

Of course! I will add that in tomorrow. Do you think something like that would be useful for #22236 too? Or is the included documentation enough.

Thanks! Yes, more usage examples are important. They help reduce user frustration and also our KTLO cost.

…ver` This adds a message buffer to `websocket_server`, backed by a ring buffer, which hold a set maximum number of messages to replay to clients that connect late, or re-connect after losing connection. Messages are replayed based on `last_received` query parameter. Related: vectordotdev#22213

…_server` docs Related: vectordotdev#22213

…t_server` Related: vectordotdev#22213

…t_server` (#22482) * docs(websocket_server sink): add `how_it_works` section for `websocket_server` Related: #22213 * Fix docs formatting * Add `websocket_server sink` to allowed PR titles

…_server` docs (#22480) Related: #22213

feat(new_sink): add websocket_listener sink

603cd75

This adds a new sink, similar to `websocket` which acts like a client, while this one acts as a server. It currently broadcasts messages to all currently connected clients.

github-actions bot added domain: topology Anything related to Vector's topology code domain: sinks Anything related to the Vector's sinks labels Jan 15, 2025

esensar added 2 commits January 15, 2025 15:26

Generate component docs for websocket_listener

0941970

Add changelog entry

c124a83

github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Jan 15, 2025

esensar changed the title ~~feat(new_sink): initial websocket_listener sink~~ feat(new sink): initial websocket_listener sink Jan 15, 2025

Add basic metrics for websocket_listener

7475c6e

esensar mentioned this pull request Jan 17, 2025

feat(sources): add custom auth strategy for components with HTTP server #22236

Merged

10 tasks

Remove custom auth

183d845

github-actions bot removed the domain: topology Anything related to Vector's topology code label Jan 17, 2025

Rename websocket_listener to websocket_server

eef4bf8

esensar changed the title ~~feat(new sink): initial websocket_listener sink~~ feat(new sink): initial websocket_server sink Jan 23, 2025

Add tests for websocket_server

b8853a2

esensar marked this pull request as ready for review January 24, 2025 11:32

esensar requested review from a team as code owners January 24, 2025 11:32

pront self-assigned this Jan 24, 2025

neko-dd reviewed Jan 24, 2025

View reviewed changes

Apply suggestions from code review

351fa45

Co-authored-by: Sandra (neko) <[email protected]>

Update docs changes in the code

0d074c0

esensar added 2 commits February 12, 2025 17:32

Merge branch 'master' into feature/websocket-listener-sink

ef4e200

Use new http server auth configuration

e881559

pront approved these changes Feb 18, 2025

View reviewed changes

src/internal_events/websocket_server.rs Outdated Show resolved Hide resolved

src/internal_events/websocket_server.rs Outdated Show resolved Hide resolved

Merge branch 'master' into feature/websocket-listener-sink

74240a0

esensar and others added 3 commits February 19, 2025 10:16

Add client count to websocket events logs

79c4f33

Co-authored-by: Pavlos Rontidis <[email protected]>

Change disconnection log from warning to info

7232c85

Regenerate docs

ba67c07

pront enabled auto-merge February 19, 2025 15:30

pront added this pull request to the merge queue Feb 19, 2025

Merged via the queue into vectordotdev:master with commit 78adec7 Feb 19, 2025
56 checks passed

esensar deleted the feature/websocket-listener-sink branch February 19, 2025 17:38

esensar mentioned this pull request Feb 20, 2025

feat(websocket_server sink): add message buffering for websocket_server #22479

Open

10 tasks

esensar added a commit to esensar/vector that referenced this pull request Feb 20, 2025

docs(external docs): fix missing configuration section for `websocket…

58baf2e

…_server` docs Related: vectordotdev#22213

esensar mentioned this pull request Feb 20, 2025

docs(external docs): fix missing configuration section for websocket_server docs #22480

Merged

10 tasks

esensar added a commit to esensar/vector that referenced this pull request Feb 20, 2025

docs(websocket_server sink): add how_it_works section for `websocke…

39cb41a

…t_server` Related: vectordotdev#22213

esensar mentioned this pull request Feb 20, 2025

docs(websocket_server sink): add how_it_works section for websocket_server #22482

Merged

10 tasks

github-merge-queue bot pushed a commit that referenced this pull request Feb 20, 2025

docs(external docs): fix missing configuration section for `websocket…

ae84bcd

…_server` docs (#22480) Related: #22213

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(new sink): initial `websocket_server` sink #22213

feat(new sink): initial `websocket_server` sink #22213

esensar commented Jan 15, 2025

esensar commented Jan 15, 2025

pront commented Jan 15, 2025

esensar commented Jan 15, 2025

esensar commented Jan 15, 2025

esensar commented Jan 21, 2025

jszwedko commented Jan 21, 2025

esensar commented Jan 28, 2025 •

edited

Loading

esensar commented Feb 18, 2025

pront commented Feb 18, 2025

esensar commented Feb 18, 2025

pront commented Feb 18, 2025

esensar commented Feb 19, 2025

esensar commented Feb 19, 2025

pront commented Feb 19, 2025

esensar commented Feb 19, 2025 •

edited

Loading

pront commented Feb 19, 2025

feat(new sink): initial websocket_server sink #22213

feat(new sink): initial websocket_server sink #22213

Conversation

esensar commented Jan 15, 2025

Summary

Change Type

Is this a breaking change?

How did you test this PR?

Does this PR include user facing changes?

Checklist

esensar commented Jan 15, 2025

pront commented Jan 15, 2025

esensar commented Jan 15, 2025

esensar commented Jan 15, 2025

esensar commented Jan 21, 2025

jszwedko commented Jan 21, 2025

esensar commented Jan 28, 2025 • edited Loading

esensar commented Feb 18, 2025

pront commented Feb 18, 2025

esensar commented Feb 18, 2025

pront commented Feb 18, 2025

esensar commented Feb 19, 2025

esensar commented Feb 19, 2025

pront commented Feb 19, 2025

esensar commented Feb 19, 2025 • edited Loading

pront commented Feb 19, 2025

feat(new sink): initial `websocket_server` sink #22213

feat(new sink): initial `websocket_server` sink #22213

esensar commented Jan 28, 2025 •

edited

Loading

esensar commented Feb 19, 2025 •

edited

Loading