From ba092940d5fec4d354b415b561dad58549b539af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ramona=20=C5=81uczkiewicz?= Date: Fri, 7 Oct 2022 11:38:33 +0200 Subject: [PATCH 1/7] Create the command driven connectors RFC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Ramona Łuczkiewicz --- .../0021-command-driven-connectors.md | 88 +++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 rfc/accepted/0021-command-driven-connectors.md diff --git a/rfc/accepted/0021-command-driven-connectors.md b/rfc/accepted/0021-command-driven-connectors.md new file mode 100644 index 000000000..e1f0af533 --- /dev/null +++ b/rfc/accepted/0021-command-driven-connectors.md @@ -0,0 +1,88 @@ +# Command driven connectors + +- Feature Name: `command_driven_connectors` +- Start Date: 2021-10-07 +- Tremor Issue: [tremor-rs/tremor-runtime#0000](https://github.com/tremor-rs/tremor-runtime/issues/0000) +- RFC PR: [tremor-rs/tremor-rfcs#0000](https://github.com/tremor-rs/tremor-rfcs/pull/0000) + +## Summary +[summary]: #summary + +This RFC adds support for connectors that don't do anything by themselves, but instead are driven by commands sent to them. + +## Motivation +[motivation]: #motivation + +There are many connectors, where, especially on the read side, there's no obvious "default" way to read data. For example object stores usually do not provide an API to stream changes (and if they did, the users might still want to read files in a different fashion). With command driven connectors, users will be able to read data that they need based on the commands they send to the connector. + +## Guide-level Explanation +[guide-level-explanation]: #guide-level-explanation + +Explain the proposal as if it was already included in Tremor and you were teaching it to another Tremor user. That generally means: + +- Introducing new named concepts. +- Explaining the feature largely in terms of examples. +- Explaining how stakeholders should *think* about the feature, and how it should impact the way they use Tremor. It should explain the impact as concretely as possible. +- If applicable, provide sample error messages, deprecation warnings, or migration guidance. +- If applicable, describe the differences between teaching this to existing Tremor stakeholders and new Tremor programmers. + +For implementation-oriented RFCs (e.g. for language internals), this section should focus on how language contributors should think about the change, and give examples of its concrete impact. For policy RFCs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms. + +## Reference-level Explanation +[reference-level-explanation]: #reference-level-explanation + +This is the technical portion of the RFC. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- It is reasonably clear how the feature would be implemented. +- Corner cases are dissected by example. + +The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. + +## Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +## Rationale and Alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +- Why is this design the best in the space of possible designs? +- What other designs have been considered, and what is the rationale for not choosing them? +- What is the impact of not doing this? + +## Prior Art +[prior-art]: #prior-art + +Discuss prior art, both the good and the bad, in relation to this proposal. +A few examples of what this can include are: + +- For language, library, tools, and clustering proposals: Does this feature exist in other programming languages, and what experience have their community had? +- For community proposals: Is this done by some other community and what were their experiences with it? +- For other teams: What lessons can we learn from what other communities have done here? +- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. + +This section is intended to encourage you as an author to think about the lessons from other projects, provide readers of your RFC with a fuller picture. +If there is no prior art, that is fine- your ideas are interesting to us whether they are brand new or if it is an adaptation from other projects. + +Note that while precedent set by other projects is some motivation, it does not, on its own, motivate an RFC. +Please also take into consideration that Tremor sometimes intentionally diverges from similar projects. + +## Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the RFC process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +## Future Possibilities +[future-possibilities]: #future-possibilities + +Think about what the natural extension and evolution of your proposal would be and how it would affect Tremor as a whole in a holistic way. Try to use this section as a tool to more fully consider all possible interactions with the project in your proposal. Also, consider how this all fits into the roadmap for the project and of the relevant sub-team. + +This is also a good place to "dump ideas", if they are out of scope for the RFC you are writing but otherwise related. + +If you have tried and cannot think of any future possibilities, you may state that you cannot think of anything. + +Note that having something written down in the future-possibilities section is not a reason to accept the current or a future RFC; such notes should be in the section on motivation or rationale in this or subsequent RFCs. +The section merely provides additional information. From d0f1e24c033ab84d36323f7876ab9c995d44a1db Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ramona=20=C5=81uczkiewicz?= Date: Tue, 11 Oct 2022 11:23:28 +0200 Subject: [PATCH 2/7] Add guide-level explanation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Ramona Łuczkiewicz --- .../0021-command-driven-connectors.md | 52 +++++++++++++++---- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/rfc/accepted/0021-command-driven-connectors.md b/rfc/accepted/0021-command-driven-connectors.md index e1f0af533..76fe96175 100644 --- a/rfc/accepted/0021-command-driven-connectors.md +++ b/rfc/accepted/0021-command-driven-connectors.md @@ -18,15 +18,43 @@ There are many connectors, where, especially on the read side, there's no obviou ## Guide-level Explanation [guide-level-explanation]: #guide-level-explanation -Explain the proposal as if it was already included in Tremor and you were teaching it to another Tremor user. That generally means: - -- Introducing new named concepts. -- Explaining the feature largely in terms of examples. -- Explaining how stakeholders should *think* about the feature, and how it should impact the way they use Tremor. It should explain the impact as concretely as possible. -- If applicable, provide sample error messages, deprecation warnings, or migration guidance. -- If applicable, describe the differences between teaching this to existing Tremor stakeholders and new Tremor programmers. - -For implementation-oriented RFCs (e.g. for language internals), this section should focus on how language contributors should think about the change, and give examples of its concrete impact. For policy RFCs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms. +Command-driven connectors define at least two ports - one for data and one for commands. +As an example, let's look at a connector that reads files: + +```tremor +define flow main +flow + define connector file_connector from file + with + codec="string", + config = {"command_driven": true} + end; + + define connector file_list from file + with + codec = "json", + config = { + "path": "in.json", + "mode": "read", + }, + end; + + create pipeline main + pipeline + select { "command": "read", "path": event.path } from in into out; + end; + + create connector file_connector from file_connector; + create connector file_list from file_list; + + connect /connector/file_list/out to /pipeline/main/in; + + # This is the magic - we send the commands here, note the "commands" port + connect /pipeline/main/out to /connector/file_connector/commands; + + connect /connector/file_connector/data to /pipeline/main/out; +end; +``` ## Reference-level Explanation [reference-level-explanation]: #reference-level-explanation @@ -86,3 +114,9 @@ If you have tried and cannot think of any future possibilities, you may state th Note that having something written down in the future-possibilities section is not a reason to accept the current or a future RFC; such notes should be in the section on motivation or rationale in this or subsequent RFCs. The section merely provides additional information. + + +## notes +- separate channels - one for commands, one for data +- traits (not necessarily rust traits) for the behaviours that a connector can implement + - e.g. KV store - "read key", "stream read key", filesystem - "create directory", "delete directory" \ No newline at end of file From 14c167aa53ff6010adeeb6eb249827a50609e2fa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ramona=20=C5=81uczkiewicz?= Date: Tue, 11 Oct 2022 11:39:02 +0200 Subject: [PATCH 3/7] Rename commands -> data MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Ramona Łuczkiewicz --- rfc/accepted/0021-command-driven-connectors.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfc/accepted/0021-command-driven-connectors.md b/rfc/accepted/0021-command-driven-connectors.md index 76fe96175..1022f91e4 100644 --- a/rfc/accepted/0021-command-driven-connectors.md +++ b/rfc/accepted/0021-command-driven-connectors.md @@ -49,8 +49,8 @@ flow connect /connector/file_list/out to /pipeline/main/in; - # This is the magic - we send the commands here, note the "commands" port - connect /pipeline/main/out to /connector/file_connector/commands; + # This is the magic - we send the commands here, note the "data" port + connect /pipeline/main/out to /connector/file_connector/data; connect /connector/file_connector/data to /pipeline/main/out; end; From b586d5aaab349942486678672f6c0f7d4327f7fb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ramona=20=C5=81uczkiewicz?= Date: Tue, 11 Oct 2022 12:24:43 +0200 Subject: [PATCH 4/7] Rename data -> control MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Ramona Łuczkiewicz --- rfc/accepted/0021-command-driven-connectors.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfc/accepted/0021-command-driven-connectors.md b/rfc/accepted/0021-command-driven-connectors.md index 1022f91e4..00cc9f2db 100644 --- a/rfc/accepted/0021-command-driven-connectors.md +++ b/rfc/accepted/0021-command-driven-connectors.md @@ -49,8 +49,8 @@ flow connect /connector/file_list/out to /pipeline/main/in; - # This is the magic - we send the commands here, note the "data" port - connect /pipeline/main/out to /connector/file_connector/data; + # This is the magic - we send the commands here, note the "control" port + connect /pipeline/main/out to /connector/file_connector/control; connect /connector/file_connector/data to /pipeline/main/out; end; From 55c8955bce64b2e085737f44428135ebc8c279ab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ramona=20=C5=81uczkiewicz?= Date: Tue, 11 Oct 2022 13:59:31 +0200 Subject: [PATCH 5/7] More content MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Ramona Łuczkiewicz --- .../0021-command-driven-connectors.md | 25 ++++++++----------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/rfc/accepted/0021-command-driven-connectors.md b/rfc/accepted/0021-command-driven-connectors.md index 00cc9f2db..214872bc2 100644 --- a/rfc/accepted/0021-command-driven-connectors.md +++ b/rfc/accepted/0021-command-driven-connectors.md @@ -59,25 +59,22 @@ end; ## Reference-level Explanation [reference-level-explanation]: #reference-level-explanation -This is the technical portion of the RFC. Explain the design in sufficient detail that: - -- Its interaction with other features is clear. -- It is reasonably clear how the feature would be implemented. -- Corner cases are dissected by example. - -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. +Each command driven connector implements at least two ports - `data` and `control`. +`control` is an input port, through which the commands are sent. +Currently only reads are supported, so `data` is an output port. +Each message in the `data` port is a single event, with metadata containing the original command. +The commands are standardised as far as it is practical, so the connectors can be swapped without adjusting the rest of the system. ## Drawbacks [drawbacks]: #drawbacks -Why should we *not* do this? +This raises the complexity of Tremor. ## Rationale and Alternatives [rationale-and-alternatives]: #rationale-and-alternatives -- Why is this design the best in the space of possible designs? -- What other designs have been considered, and what is the rationale for not choosing them? -- What is the impact of not doing this? +There are no known alternatives that provide the same benefits. +Currently, for example the S3 reader connector, will read all files in the bucket, once, which has limited use. ## Prior Art [prior-art]: #prior-art @@ -99,9 +96,9 @@ Please also take into consideration that Tremor sometimes intentionally diverges ## Unresolved Questions [unresolved-questions]: #unresolved-questions -- What parts of the design do you expect to resolve through the RFC process before this gets merged? -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? -- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? +- How do we enforce uniformity of commands across connectors? +- How would writes work? +- Are multiple events per command allowed in the output? ## Future Possibilities [future-possibilities]: #future-possibilities From a1aecaca656f0e8c6705a16c94e7507949a72c45 Mon Sep 17 00:00:00 2001 From: agares Date: Tue, 11 Oct 2022 14:03:05 +0200 Subject: [PATCH 6/7] Update rfc/accepted/0021-command-driven-connectors.md Co-authored-by: Heinz N. Gies Signed-off-by: agares --- rfc/accepted/0021-command-driven-connectors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/accepted/0021-command-driven-connectors.md b/rfc/accepted/0021-command-driven-connectors.md index 214872bc2..5ef72cc75 100644 --- a/rfc/accepted/0021-command-driven-connectors.md +++ b/rfc/accepted/0021-command-driven-connectors.md @@ -8,7 +8,7 @@ ## Summary [summary]: #summary -This RFC adds support for connectors that don't do anything by themselves, but instead are driven by commands sent to them. +This RFC introduces a control plane for connectors to allow controlling their behavior and actions via events. ## Motivation [motivation]: #motivation From e6186d166449bf90cf93123e185600ea93007963 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ramona=20=C5=81uczkiewicz?= Date: Wed, 12 Oct 2022 11:22:28 +0200 Subject: [PATCH 7/7] Address PR comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Ramona Łuczkiewicz --- .../0021-command-driven-connectors.md | 33 +++++++++++-------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/rfc/accepted/0021-command-driven-connectors.md b/rfc/accepted/0021-command-driven-connectors.md index 5ef72cc75..4a1b6a37f 100644 --- a/rfc/accepted/0021-command-driven-connectors.md +++ b/rfc/accepted/0021-command-driven-connectors.md @@ -13,30 +13,34 @@ This RFC introduces a control plane for connectors to allow controlling their be ## Motivation [motivation]: #motivation -There are many connectors, where, especially on the read side, there's no obvious "default" way to read data. For example object stores usually do not provide an API to stream changes (and if they did, the users might still want to read files in a different fashion). With command driven connectors, users will be able to read data that they need based on the commands they send to the connector. +In some cases, connectors might need to be controlled in ways, which are not necessarily expressible as data (or metadata). Things like querying databases, controlling TCP/UDP servers/clients, etc. might require an interface driven by commands. ## Guide-level Explanation [guide-level-explanation]: #guide-level-explanation -Command-driven connectors define at least two ports - one for data and one for commands. +Please note that the `oneshot` connector does not exist yet. It should be added together with implementation for this RFC. Its function is to send an event once on startup. + +Command-driven connectors define at least one port - `control`, and optionally `in` and/or `out` for data. As an example, let's look at a connector that reads files: ```tremor define flow main flow + use std; + + define connector read_file_list from oneshot + with + event = std::commands::file::read("in.json") + end; + define connector file_connector from file with - codec="string", - config = {"command_driven": true} + codec="string" end; define connector file_list from file with - codec = "json", - config = { - "path": "in.json", - "mode": "read", - }, + codec = "json" end; create pipeline main @@ -44,26 +48,29 @@ flow select { "command": "read", "path": event.path } from in into out; end; + create connector read_file_list from read_file_list; create connector file_connector from file_connector; create connector file_list from file_list; + connect /connector/read_file_list/out to /connector/file_list/control; connect /connector/file_list/out to /pipeline/main/in; # This is the magic - we send the commands here, note the "control" port connect /pipeline/main/out to /connector/file_connector/control; - connect /connector/file_connector/data to /pipeline/main/out; + connect /connector/file_connector/out to /pipeline/main/out; end; ``` ## Reference-level Explanation [reference-level-explanation]: #reference-level-explanation -Each command driven connector implements at least two ports - `data` and `control`. +Each command driven connector implements at least one port - `control`. `control` is an input port, through which the commands are sent. -Currently only reads are supported, so `data` is an output port. -Each message in the `data` port is a single event, with metadata containing the original command. + The commands are standardised as far as it is practical, so the connectors can be swapped without adjusting the rest of the system. +Convenience functions are provided to generate the events. +Each event has a `correlation` field, which will be a unique identifier provided in the metadata for the response. This is automatically generated by the convenience functions. ## Drawbacks [drawbacks]: #drawbacks