Add workflow collections #2569
Merged
Add workflow collections #2569
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
These are named and shared key-value storages
stuartc
requested changes
Oct 9, 2024
| add :collection_id, | ||
| references(:collections, on_delete: :delete_all, type: :binary_id, null: false) | ||
|
|
||
| timestamps() |
There was a problem hiding this comment.
Could you change this to timestamps(type: :naive_datetime_usec) and the models as well.
Probably a good idea to start with sub-second resolution on these.
| timestamps() | ||
| end | ||
|
|
||
| create unique_index(:collections_items, [:collection_id, :key]) |
There was a problem hiding this comment.
Nice one, do you think we should put an index on updated_at and inserted_at as well?
There was a problem hiding this comment.
Hi @stuartc I would like to play a bit more with the BRIN indexes because I am trying to avoid adding more cost to the job execution. May we add it during the "query by filter" PR? Will have better stats.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2569 +/- ##
==========================================
+ Coverage 90.44% 90.47% +0.02%
==========================================
Files 316 319 +3
Lines 10926 10955 +29
==========================================
+ Hits 9882 9911 +29
Misses 1044 1044 ☔ View full report in Codecov by Sentry. |
There is no user update on collection name
jyeshe
force-pushed
the
collections
branch
3 times, most recently
from
d1cf44a
to
f3872e9
Compare
jyeshe
force-pushed
the
collections
branch
2 times, most recently
from
bf0f894
to
cf579c0
Compare
|
Thank you for the AI disclosure @jyeshe |
jyeshe
force-pushed
the
collections
branch
3 times, most recently
from
1c4c60f
to
d160c17
Compare
jyeshe
force-pushed
the
collections
branch
2 times, most recently
from
7c331ed
to
9d6bca4
Compare
stuartc
approved these changes
Oct 15, 2024
jyeshe
added a commit
that referenced
this pull request
Oct 22, 2024
* Add workflow collections These are named and shared key-value storages * Increase timestamp precision * Use single return patter on same get function * Optimize all ops once the names are stable There is no user update on collection name * Changelog and formatting * Add stream_all allowing equivalent queries to HGETALL and HSCAN * Add stream_match for wildcard prefix queries * Add pg_trigram and GIN index on the key to allow multi wildcard
jyeshe
added a commit
that referenced
this pull request
Oct 22, 2024
* Add workflow collections These are named and shared key-value storages * Increase timestamp precision * Use single return patter on same get function * Optimize all ops once the names are stable There is no user update on collection name * Changelog and formatting * Add stream_all allowing equivalent queries to HGETALL and HSCAN * Add stream_match for wildcard prefix queries * Add pg_trigram and GIN index on the key to allow multi wildcard
stuartc
pushed a commit
that referenced
this pull request
Oct 22, 2024
* Add workflow collections These are named and shared key-value storages * Increase timestamp precision * Use single return patter on same get function * Optimize all ops once the names are stable There is no user update on collection name * Changelog and formatting * Add stream_all allowing equivalent queries to HGETALL and HSCAN * Add stream_match for wildcard prefix queries * Add pg_trigram and GIN index on the key to allow multi wildcard
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds workflow collections which are scoped to projects and provide similar functionality to Redis HSET and HGET (named hashes operations). It will be used not only to share mutable data between jobs on downstream but also shared among multiple workflows and it will be the responsibility of the programmer to use it without conflicts between workflows.
EDIT: The PRs are "merged" and it also includes functionalities similar to HGETALL and HSCAN allowing a query by pattern using a cursor (and respecting a limit).
Closes #2550
Closes #2551
Closes #2552
Validation steps
There is not UI using it yet but a complete test coverage is provided.
AI Usage
Please disclose how you've used AI in this work (it's cool, we just want to know!):
You can read more details in our Responsible AI Policy
Pre-submission checklist
:owner,:admin,:editor,:viewer)