feat!: use ShardingParams on subscriptions, make Decoder/Encoder auto sharding friendly by default #1958

weboko · 2024-04-15T23:12:24Z

Problem

Seems latest fleet update is not allowing to subscribe to DefaultPubsubTopic anymore and we should be using shards.

Solution

Use pubsub topic for subscription creation from those determined from shards / contentTopics

Though this PR is huge it includes only following changes:

change to the logic of Decoders/Encoders that makes use of default cluster ID and maps content topic to pubsubTopic;
createSubscription on Filter accepts ShardingParams;
updates to the tests to check for the above;
subscribe checks pubsub topics from Decoders;
some improvements to the tests to remove duplication;

Notes

Addresses - #1961
Next PR - #1959

weboko · 2024-04-15T23:13:06Z

packages/interfaces/src/waku.ts

@@ -18,7 +18,7 @@ export interface Waku {

  connectionManager: IConnectionManager;

-  dial(peer: PeerId | Multiaddr, protocols?: Protocols[]): Promise<Stream>;
+  dial(peer: PeerId | MultiaddrInput, protocols?: Protocols[]): Promise<Stream>;


fix for discrepancy in types - this prevents us from doing .dial(multiAddrStr) in TS

packages/core/src/lib/filter/index.ts

github-actions · 2024-04-15T23:16:30Z

size-limit report 📦

Path	Size	Loading time (3g)	Running time (snapdragon)	Total time
Waku node	180.93 KB (+0.11% 🔺)	3.7 s (+0.11% 🔺)	13.8 s (-43.29% 🔽)	17.5 s
Waku Simple Light Node	180.85 KB (-0.01% 🔽)	3.7 s (-0.01% 🔽)	22.2 s (+30.04% 🔺)	25.8 s
ECIES encryption	23.08 KB (-0.14% 🔽)	462 ms (-0.14% 🔽)	6.5 s (+11.66% 🔺)	6.9 s
Symmetric encryption	22.55 KB (-0.15% 🔽)	452 ms (-0.15% 🔽)	5.1 s (-20.88% 🔽)	5.6 s
DNS discovery	72.42 KB (0%)	1.5 s (0%)	12.3 s (+28.64% 🔺)	13.7 s
Peer Exchange discovery	74.1 KB (+0.18% 🔺)	1.5 s (+0.18% 🔺)	15.6 s (+64.29% 🔺)	17.1 s
Local Peer Cache Discovery	67.64 KB (0%)	1.4 s (0%)	16.5 s (+14.35% 🔺)	17.9 s
Privacy preserving protocols	38.87 KB (0%)	778 ms (0%)	11.2 s (+2.99% 🔺)	12 s
Waku Filter	111.42 KB (+0.16% 🔺)	2.3 s (+0.16% 🔺)	18.2 s (+93.72% 🔺)	20.5 s
Waku LightPush	110.11 KB (-0.12% 🔽)	2.3 s (-0.12% 🔽)	12.1 s (+9.49% 🔺)	14.3 s
History retrieval protocols	110.72 KB (+0.01% 🔺)	2.3 s (+0.01% 🔺)	15 s (-29.78% 🔽)	17.2 s
Deterministic Message Hashing	4.83 KB (0%)	97 ms (0%)	701 ms (+64.37% 🔺)	798 ms

… mapping default for decoder / encoder

danisharora099 · 2024-04-22T13:57:39Z

I might be missing something but I don't fully understand what the PR is solving for. Can you please elaborate, or point to related issues/references?

weboko · 2024-04-22T14:19:54Z

I might be missing something but I don't fully understand what the PR is solving for. Can you please elaborate, or point to related issues/references?

This problem is addressed - #1961
Happens due to https://discord.com/channels/1110799176264056863/1111539372618166294/1231967635722342540

packages/sdk/src/protocols/filter.ts

…ocol-fixes

weboko · 2024-04-26T14:25:34Z

packages/interfaces/src/protocols.ts

@@ -29,7 +29,7 @@ export type IBaseProtocolSDK = {
 };

 export type ContentTopicInfo = {
-  clusterId: number;
+  clusterId?: number;


Important change - assumption is that default choice is TWN cluster ID.

weboko · 2024-04-26T14:26:58Z

packages/sdk/src/protocols/filter.ts

  ): Promise<SubscriptionManager> {
    const pubsubTopic =
      typeof pubsubTopicShardInfo == "string"
        ? pubsubTopicShardInfo
-        : singleShardInfoToPubsubTopic(pubsubTopicShardInfo);
+        : shardInfoToPubsubTopics(pubsubTopicShardInfo)?.[0];


ShardingParams can be mapped to many pubsubTopics - we provide subscription to one.

Any idea how to handle it better? @waku-org/js-waku-developers

@weboko we could forbid passing shard info where shards length is greater than 1

fbarbu15

LGTM, Awesome refactoring of the tests and utils!

packages/tests/tests/filter/single_node/subscribe.node.spec.ts

weboko added 2 commits April 16, 2024 00:58

fix: use pubsubTopic from current ones if not set

6acf183

fix: improve type on dial method

a7bf4fd

weboko requested a review from a team as a code owner April 15, 2024 23:12

weboko commented Apr 15, 2024

View reviewed changes

packages/core/src/lib/filter/index.ts Outdated Show resolved Hide resolved

weboko marked this pull request as draft April 15, 2024 23:32

weboko changed the title ~~fix: use pubsubTopic from current on filter subscribe~~ fix!: use pubsubTopic from current on filter subscribe Apr 18, 2024

weboko added 6 commits April 18, 2024 22:17

enforce same pubusb on filter.subscribe, make content topic to pubsub…

08397ff

… mapping default for decoder / encoder

fix mapping problem

c89c8f9

update tests

5248761

add error handling

f9b9bda

fix typo

fd60cc2

merge with master

f7849c3

weboko marked this pull request as ready for review April 22, 2024 11:39

up lock

22cfb50

weboko added 3 commits April 22, 2024 17:34

rm lock

8db5bf8

up lock

0f79e17

remove only

0c7f4a1

adklempner approved these changes Apr 22, 2024

View reviewed changes

weboko mentioned this pull request Apr 22, 2024

feat: add keep alive to Filter #1970

Merged

weboko changed the title ~~fix!: use pubsubTopic from current on filter subscribe~~ feat!: use pubsubTopic from current on filter subscribe Apr 22, 2024

fix content topic

a8922ba

weboko mentioned this pull request Apr 22, 2024

feat: filter.createSubscription accept ShardParams #1967

Closed

danisharora099 reviewed Apr 23, 2024

View reviewed changes

packages/sdk/src/protocols/filter.ts Outdated Show resolved Hide resolved

packages/sdk/src/protocols/filter.ts Outdated Show resolved Hide resolved

weboko added 3 commits April 24, 2024 21:32

Merge branch 'master' of github.com:waku-org/js-waku into weboko/prot…

eabe940

…ocol-fixes

fix ephemeral test

68f3619

fix filter unsubscribe test

364045b

weboko added 10 commits April 25, 2024 23:08

remove only, fix subscribe filter test

a929da7

remove only

6e29167

merge with master

1641074

fix cluster ID selection and named sharding subscription test

7d79bda

fix unsubscribe test

c574863

fix light push test

8873aa1

fix light push test

7d24612

fix push test

d8b1952

fix relay publish

600c02b

create runNode and fix relay tests

88ba615

weboko mentioned this pull request Apr 26, 2024

feat(filter)!: return error codes instead of throwing errors #1971

Merged

weboko added 5 commits April 26, 2024 13:17

generalize runNodes, fix some tests

90731d7

fix store tests

a4c3d50

fix toAsyncIterator tests

16ffc67

remove only

e47cc67

fix lightPush

7e1260b

weboko requested review from danisharora099 and fbarbu15 April 26, 2024 13:58

weboko changed the title ~~feat!: use pubsubTopic from current on filter subscribe~~ feat!: use ShardingParams on subscriptions, make Decoder/Encoder auto sharding friendly by default Apr 26, 2024

use generics

9d02f2b

weboko commented Apr 26, 2024

View reviewed changes

fbarbu15 approved these changes Apr 26, 2024

View reviewed changes

packages/tests/tests/filter/single_node/subscribe.node.spec.ts Show resolved Hide resolved

weboko added 4 commits April 26, 2024 23:54

try fix test

b0381b4

run failing tests

53d4db8

remove only

fd9cb86

address failed tests, remove DefaultPubsubTopic dependency in some tests

f7595c2

weboko merged commit f3627c4 into master Apr 28, 2024
9 of 10 checks passed

weboko deleted the weboko/protocol-fixes branch April 28, 2024 09:15

weboko mentioned this pull request Apr 28, 2024

chore: release master #1957

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: use ShardingParams on subscriptions, make Decoder/Encoder auto sharding friendly by default #1958

feat!: use ShardingParams on subscriptions, make Decoder/Encoder auto sharding friendly by default #1958

weboko commented Apr 15, 2024 •

edited

Loading

weboko Apr 15, 2024

github-actions bot commented Apr 15, 2024 •

edited

Loading

danisharora099 commented Apr 22, 2024

weboko commented Apr 22, 2024

weboko Apr 26, 2024

weboko Apr 26, 2024 •

edited

Loading

adklempner Apr 26, 2024

fbarbu15 left a comment

feat!: use ShardingParams on subscriptions, make Decoder/Encoder auto sharding friendly by default #1958

feat!: use ShardingParams on subscriptions, make Decoder/Encoder auto sharding friendly by default #1958

Conversation

weboko commented Apr 15, 2024 • edited Loading

Problem

Solution

Notes

weboko Apr 15, 2024

Choose a reason for hiding this comment

github-actions bot commented Apr 15, 2024 • edited Loading

size-limit report 📦

danisharora099 commented Apr 22, 2024

weboko commented Apr 22, 2024

weboko Apr 26, 2024

Choose a reason for hiding this comment

weboko Apr 26, 2024 • edited Loading

Choose a reason for hiding this comment

adklempner Apr 26, 2024

Choose a reason for hiding this comment

fbarbu15 left a comment

Choose a reason for hiding this comment

weboko commented Apr 15, 2024 •

edited

Loading

github-actions bot commented Apr 15, 2024 •

edited

Loading

weboko Apr 26, 2024 •

edited

Loading