Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

List API for the state store building block #61

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open
Prev Previous commit
Next Next commit
fixes table
Signed-off-by: Elena Kolevska <elena-kolevska@users.noreply.github.com>
elena-kolevska authored Sep 24, 2024
commit ab04f50e996742e8bdd2843ce70888fe48d6cd74
30 changes: 15 additions & 15 deletions 20240627-BC-listapi.md
Original file line number Diff line number Diff line change
@@ -164,19 +164,19 @@ Here's a list of the relevant capabilities of all the stable state stores:

| Store | Cursor listing | Offset listing | Sorting | Number of Items per Page | Prefix Search | Comments |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From this list, we can clearly see that with cursor listing, page limit and prefix search, we will have plenty of coverage.

| --- | --- | --- | --- | --- | --- | --- |
| aws dynamodb | Yes | No | Yes, with a GSI | Yes | Yes, with an additional sortKey and a GSI | In order to be able to use prefix search, users will need to have a Global Search Index(GSI) where the partition key will be a single fixed string (for ex. the `_` character) and the sort key will be the key name. There are some drawbacks to this that can be discussed in detail elsewhere. |
| azure blob store | Yes (continuation token) | No | Always sorted in ASC order. Desc, or unsorted is not possible. | Yes | Yes | Results are always sorted by key name in ascending order. |
| azure cosmos db | Yes | Yes | Yes | Yes | Yes |   |
| azure table storage | Yes | No | Yes, just ASC | Yes, with $top | Yes, with range search | Partition key is the application id. |
| cassandraYes | No | No | No | No | Can’t prefix search and sort across all partitions. We could consider maintaining a new table containing all keys, and mirroring the original key’s ttl. |   |
| cockroachdbYes, if sorting is required | Yes | Yes | No | Yes | Need to create an index on the search column |   |
| gcp firestore | Yes |   |   |   |   |   |
| in-memory | No | No | No | No | No | We can implement all the features, but it’s not trivial to aggregate data across multiple instances |
| memcached | No | No | No | No | No |   |
| mongodbYes | Yes | Yes |   | Yes |   |   |
| mysqlYes | Yes |   |   | Yes | Need to create an index on the id column. MySql supports specialized prefix indices, but you would have to know the exact length of the prefix you’ll be searching on, also sorting will not use the index. |   |
| postgresqlYes | Yes | Yes | No | Yes | Need to create an index on the key column. We can use the varchar\_pattern\_ops operator class, optimised for prefix search. |   |
| redisYes | No | No | No | Yes | Number of record per page is not guaranteed, but best effort. |   |
| **aws dynamodb** | Yes | No | Yes, with a GSI | Yes | Yes, with an additional sortKey and a GSI | In order to be able to use prefix search, users will need to have a Global Search Index(GSI) where the partition key will be a single fixed string (for ex. the `_` character) and the sort key will be the key name. There are some drawbacks to this that can be discussed in detail elsewhere. |
| **azure blob store** | Yes (continuation token) | No | Always sorted in ASC order. Desc, or unsorted is not possible. | Yes | Yes | Results are always sorted by key name in ascending order. |
| **azure cosmos db** | Yes | Yes | Yes | Yes | Yes |   |
| **azure table storage** | Yes | No | Yes, just ASC | Yes, with $top | Yes, with range search | Partition key is the application id. |
| **cassandra** | Yes | No | No | Yes | No | Can’t prefix search and sort across all partitions. We could consider maintaining a new table containing all keys, and mirroring the original key’s ttl. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can cassandra to filtering within the same partition? It might be enough to begin with. It is a common thing in CosmosDb too for transaction API, for example.

| **cockroachdb** | Yes, if sorting is required | Yes | Yes | Yes | Yes | Need to create an index on the search column |
| **gcp firestore** | Yes |   |   |   |   |   |
| **in-memory** | No | No | No | No | No | We can implement all the features, but it’s not trivial to aggregate data across multiple instances |
| **memcached** | No | No | No | No | No |   |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not belong to state store IMO, so we should not dismiss list API just because of this one.

| **mongodb** | Yes | Yes | Yes | Yes | Yes |   |
| **mysql** | Yes | Yes |   | Yes | Yes | Need to create an index on the id column. MySql supports specialized prefix indices, but you would have to know the exact length of the prefix you’ll be searching on, also sorting will not use the index. |
| **postgresql** | Yes | Yes | Yes | Yes | Yes | Need to create an index on the key column. We can use the varchar\_pattern\_ops operator class, optimised for prefix search. |
| **redis** | Yes | No | No | Yes (Best effort) | Yes | Number of record per page is not guaranteed, but best effort. |
|   |   |   |   |   |   |   |
| sqliteYes, if sorting is required | Yes | Yes | No | Yes | Need to create an index on the key column. It’s a standard b-tree index.We could maintain an index of all keys in a hash |   |
| sqlserver | Yes, if sorting is required | Yes | Yes | Yes | Yes | need to create a non-clustered index on the “key” column |
| **sqlite** | Yes, if sorting is required | Yes | Yes | Yes | Yes | Need to create an index on the key column. It’s a standard b-tree index.We could maintain an index of all keys in a hash |
| **sqlserver** | Yes, if sorting is required | Yes | Yes | Yes | Yes | need to create a non-clustered index on the “key” column |