Skip to content

Commit

Permalink
fix: Rename skip-to-newest to skip-sync-newest for consistency wi…
Browse files Browse the repository at this point in the history
…th `skip-oldest`

test: Added test coverage for `skip-to-newest`
  • Loading branch information
robdmoore committed Dec 4, 2023
1 parent f22c148 commit 9cd89fb
Show file tree
Hide file tree
Showing 11 changed files with 162 additions and 96 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ const subscriber = new AlgorandSubscriber(
],
frequencyInSeconds: 1,
maxRoundsToSync: 100,
syncBehaviour: 'skip-to-newest',
syncBehaviour: 'skip-sync-newest',
watermarkPersistence: {
get: async () => watermark,
set: async (newWatermark) => {
Expand Down
32 changes: 17 additions & 15 deletions docs/code/interfaces/types_subscription.SubscriptionConfig.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The set of events to subscribe to / emit

[types/subscription.ts:106](https://github.com/algorandfoundation/algokit-subscriber-ts/blob/main/src/types/subscription.ts#L106)

___
---

### frequencyInSeconds

Expand All @@ -40,7 +40,7 @@ The frequency to poll for new blocks in seconds

[types/subscription.ts:102](https://github.com/algorandfoundation/algokit-subscriber-ts/blob/main/src/types/subscription.ts#L102)

___
---

### maxRoundsToSync

Expand All @@ -52,27 +52,29 @@ The maximum number of rounds to sync at a time.

[types/subscription.ts:104](https://github.com/algorandfoundation/algokit-subscriber-ts/blob/main/src/types/subscription.ts#L104)

___
---

### syncBehaviour

**syncBehaviour**: ``"skip-to-newest"`` \| ``"sync-oldest"`` \| ``"sync-oldest-start-now"`` \| ``"catchup-with-indexer"``
**syncBehaviour**: `"skip-sync-newest"` \| `"sync-oldest"` \| `"sync-oldest-start-now"` \| `"catchup-with-indexer"`

The behaviour when the number of rounds to sync is greater than `maxRoundsToSync`:
* `skip-to-newest`: Discard old rounds
* `sync-oldest`: Sync from the oldest records up to `maxRoundsToSync` rounds.

**Note:** will be slow to catch up if sync is significantly behind the tip of the chain
* `sync-oldest-start-now`: Sync from the oldest records up to `maxRoundsToSync` rounds, unless
current watermark is `0` in which case it will start `maxRoundsToSync` back from the tip of the chain.
* `catchup-with-indexer`: Will catch up to `tipOfTheChain - maxRoundsToSync` using indexer (fast) and then
continue with algod.
- `skip-sync-newest`: Discard old rounds
- `sync-oldest`: Sync from the oldest records up to `maxRoundsToSync` rounds.

**Note:** will be slow to catch up if sync is significantly behind the tip of the chain

- `sync-oldest-start-now`: Sync from the oldest records up to `maxRoundsToSync` rounds, unless
current watermark is `0` in which case it will start `maxRoundsToSync` back from the tip of the chain.
- `catchup-with-indexer`: Will catch up to `tipOfTheChain - maxRoundsToSync` using indexer (fast) and then
continue with algod.

#### Defined in

[types/subscription.ts:117](https://github.com/algorandfoundation/algokit-subscriber-ts/blob/main/src/types/subscription.ts#L117)

___
---

### watermarkPersistence

Expand All @@ -83,9 +85,9 @@ its position in the chain.

#### Type declaration

| Name | Type |
| :------ | :------ |
| `get` | () => `Promise`\<`number`\> |
| Name | Type |
| :---- | :------------------------------------------------ |
| `get` | () => `Promise`\<`number`\> |
| `set` | (`newWatermark`: `number`) => `Promise`\<`void`\> |

#### Defined in
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The filter to apply to find transactions of interest.

[types/subscription.ts:7](https://github.com/algorandfoundation/algokit-subscriber-ts/blob/main/src/types/subscription.ts#L7)

___
---

### maxRoundsToSync

Expand All @@ -36,41 +36,42 @@ ___
The maximum number of rounds to sync for each subscription pull/poll.

This gives you control over how many rounds you wait for at a time,
your staleness tolerance when using `skip-to-newest` or `fail`, and
your staleness tolerance when using `skip-sync-newest` or `fail`, and
your catchup speed when using `sync-oldest`.

#### Defined in

[types/subscription.ts:25](https://github.com/algorandfoundation/algokit-subscriber-ts/blob/main/src/types/subscription.ts#L25)

___
---

### syncBehaviour

**syncBehaviour**: ``"skip-to-newest"`` \| ``"sync-oldest"`` \| ``"sync-oldest-start-now"`` \| ``"catchup-with-indexer"`` \| ``"fail"``
**syncBehaviour**: `"skip-sync-newest"` \| `"sync-oldest"` \| `"sync-oldest-start-now"` \| `"catchup-with-indexer"` \| `"fail"`

If the current tip of the configured Algorand blockchain is more than `maxRoundsToSync`
past `watermark` then how should that be handled:
* `skip-to-newest`: Discard old blocks/transactions and sync the newest; useful
for real-time notification scenarios where you don't care about history and
are happy to lose old transactions.
* `sync-oldest`: Sync from the oldest rounds forward `maxRoundsToSync` rounds
using algod; note: this will be slow if you are starting from 0 and requires
an archival node.
* `sync-oldest-start-now`: Same as `sync-oldest`, but if the `watermark` is `0`
then start at the current round i.e. don't sync historical records, but once
subscribing starts sync everything; note: if it falls behind it requires an
archival node.
* `catchup-with-indexer`: Sync to round `currentRound - maxRoundsToSync + 1`
using indexer (much faster than using algod for long time periods) and then
use algod from there.
* `fail`: Throw an error.

- `skip-sync-newest`: Discard old blocks/transactions and sync the newest; useful
for real-time notification scenarios where you don't care about history and
are happy to lose old transactions.
- `sync-oldest`: Sync from the oldest rounds forward `maxRoundsToSync` rounds
using algod; note: this will be slow if you are starting from 0 and requires
an archival node.
- `sync-oldest-start-now`: Same as `sync-oldest`, but if the `watermark` is `0`
then start at the current round i.e. don't sync historical records, but once
subscribing starts sync everything; note: if it falls behind it requires an
archival node.
- `catchup-with-indexer`: Sync to round `currentRound - maxRoundsToSync + 1`
using indexer (much faster than using algod for long time periods) and then
use algod from there.
- `fail`: Throw an error.

#### Defined in

[types/subscription.ts:43](https://github.com/algorandfoundation/algokit-subscriber-ts/blob/main/src/types/subscription.ts#L43)

___
---

### watermark

Expand Down
4 changes: 2 additions & 2 deletions docs/subscriber.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ export interface SubscriptionConfig {
/** The set of events to subscribe to / emit */
events: SubscriptionConfigEvent<unknown>[]
/** The behaviour when the number of rounds to sync is greater than `maxRoundsToSync`:
* * `skip-to-newest`: Discard old rounds
* * `skip-sync-newest`: Discard old rounds
* * `sync-oldest`: Sync from the oldest records up to `maxRoundsToSync` rounds.
*
* **Note:** will be slow to catch up if sync is significantly behind the tip of the chain
Expand All @@ -37,7 +37,7 @@ export interface SubscriptionConfig {
* * `catchup-with-indexer`: Will catch up to `tipOfTheChain - maxRoundsToSync` using indexer (fast) and then
* continue with algod.
*/
syncBehaviour: 'skip-to-newest' | 'sync-oldest' | 'sync-oldest-start-now' | 'catchup-with-indexer'
syncBehaviour: 'skip-sync-newest' | 'sync-oldest' | 'sync-oldest-start-now' | 'catchup-with-indexer'
/** Methods to retrieve and persist the current watermark so syncing is resilient and maintains
* its position in the chain. */
watermarkPersistence: {
Expand Down
8 changes: 4 additions & 4 deletions docs/subscriptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,13 @@ export interface TransactionSubscriptionParams {
/** The maximum number of rounds to sync for each subscription pull/poll.
*
* This gives you control over how many rounds you wait for at a time,
* your staleness tolerance when using `skip-to-newest` or `fail`, and
* your staleness tolerance when using `skip-sync-newest` or `fail`, and
* your catchup speed when using `sync-oldest`.
**/
maxRoundsToSync: number
/** If the current tip of the configured Algorand blockchain is more than `maxRoundsToSync`
* past `watermark` then how should that be handled:
* * `skip-to-newest`: Discard old blocks/transactions and sync the newest; useful
* * `skip-sync-newest`: Discard old blocks/transactions and sync the newest; useful
* for real-time notification scenarios where you don't care about history and
* are happy to lose old transactions.
* * `sync-oldest`: Sync from the oldest rounds forward `maxRoundsToSync` rounds
Expand All @@ -64,7 +64,7 @@ export interface TransactionSubscriptionParams {
* use algod from there.
* * `fail`: Throw an error.
**/
syncBehaviour: 'skip-to-newest' | 'sync-oldest' | 'sync-oldest-start-now' | 'catchup-with-indexer' | 'fail'
syncBehaviour: 'skip-sync-newest' | 'sync-oldest' | 'sync-oldest-start-now' | 'catchup-with-indexer' | 'fail'
}
/** Specify a filter to apply to find transactions of interest. */
Expand Down Expand Up @@ -145,7 +145,7 @@ const subscription = await getSubscribedTransactions(
},
watermark,
maxRoundsToSync: 100,
onMaxRounds: 'skip-to-newest',
onMaxRounds: 'skip-sync-newest',
},
algod,
)
Expand Down
2 changes: 1 addition & 1 deletion examples/usdc/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ process.on('uncaughtException', (e) => console.error(e))
],
frequencyInSeconds: 1,
maxRoundsToSync: 100,
syncBehaviour: 'skip-to-newest',
syncBehaviour: 'skip-sync-newest',
watermarkPersistence: {
get: async () => watermark,
set: async (newWatermark) => {
Expand Down
5 changes: 3 additions & 2 deletions src/subscriptions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,14 @@ export async function getSubscribedTransactions(
const catchupTransactions: TransactionResult[] = []
let start = +new Date()

if (currentRound - algodSyncFromRoundNumber > maxRoundsToSync) {
if (currentRound - watermark > maxRoundsToSync) {
switch (onMaxRounds) {
case 'fail':
throw new Error(`Invalid round number to subscribe from ${algodSyncFromRoundNumber}; current round number is ${currentRound}`)
case 'skip-to-newest':
case 'skip-sync-newest':
algodSyncFromRoundNumber = currentRound - maxRoundsToSync + 1
startRound = algodSyncFromRoundNumber
console.log('YO', algodSyncFromRoundNumber, startRound, endRound)

Check warning on line 53 in src/subscriptions.ts

View workflow job for this annotation

GitHub Actions / Continuous Integration / node-ci

Unexpected console statement
break
case 'sync-oldest':
endRound = algodSyncFromRoundNumber + maxRoundsToSync - 1
Expand Down
10 changes: 5 additions & 5 deletions src/types/subscription.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,13 @@ export interface TransactionSubscriptionParams {
/** The maximum number of rounds to sync for each subscription pull/poll.
*
* This gives you control over how many rounds you wait for at a time,
* your staleness tolerance when using `skip-to-newest` or `fail`, and
* your staleness tolerance when using `skip-sync-newest` or `fail`, and
* your catchup speed when using `sync-oldest`.
**/
maxRoundsToSync: number
/** If the current tip of the configured Algorand blockchain is more than `maxRoundsToSync`
* past `watermark` then how should that be handled:
* * `skip-to-newest`: Discard old blocks/transactions and sync the newest; useful
* * `skip-sync-newest`: Discard old blocks/transactions and sync the newest; useful
* for real-time notification scenarios where you don't care about history and
* are happy to lose old transactions.
* * `sync-oldest`: Sync from the oldest rounds forward `maxRoundsToSync` rounds
Expand All @@ -40,7 +40,7 @@ export interface TransactionSubscriptionParams {
* use algod from there.
* * `fail`: Throw an error.
**/
syncBehaviour: 'skip-to-newest' | 'sync-oldest' | 'sync-oldest-start-now' | 'catchup-with-indexer' | 'fail'
syncBehaviour: 'skip-sync-newest' | 'sync-oldest' | 'sync-oldest-start-now' | 'catchup-with-indexer' | 'fail'
}

/** Specify a filter to apply to find transactions of interest. */
Expand Down Expand Up @@ -105,7 +105,7 @@ export interface SubscriptionConfig {
/** The set of events to subscribe to / emit */
events: SubscriptionConfigEvent<unknown>[]
/** The behaviour when the number of rounds to sync is greater than `maxRoundsToSync`:
* * `skip-to-newest`: Discard old rounds
* * `skip-sync-newest`: Discard old rounds
* * `sync-oldest`: Sync from the oldest records up to `maxRoundsToSync` rounds.
*
* **Note:** will be slow to catch up if sync is significantly behind the tip of the chain
Expand All @@ -114,7 +114,7 @@ export interface SubscriptionConfig {
* * `catchup-with-indexer`: Will catch up to `tipOfTheChain - maxRoundsToSync` using indexer (fast) and then
* continue with algod.
*/
syncBehaviour: 'skip-to-newest' | 'sync-oldest' | 'sync-oldest-start-now' | 'catchup-with-indexer'
syncBehaviour: 'skip-sync-newest' | 'sync-oldest' | 'sync-oldest-start-now' | 'catchup-with-indexer'
/** Methods to retrieve and persist the current watermark so syncing is resilient and maintains
* its position in the chain. */
watermarkPersistence: {
Expand Down
62 changes: 62 additions & 0 deletions tests/scenarios/skip-sync-newest.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
import { algorandFixture } from '@algorandfoundation/algokit-utils/testing'
import { beforeEach, describe, test } from '@jest/globals'
import { GetSubscribedTransactionsFromSender, SendXTransactions } from '../transactions'

describe('skip-sync-newest', () => {
const localnet = algorandFixture()

beforeEach(localnet.beforeEach, 10e6)

test('Only processes the latest transaction when starting from beginning of chain', async () => {
const { algod, testAccount } = localnet.context
const { txns, lastTxnRound } = await SendXTransactions(2, testAccount, algod)

const subscribed = await GetSubscribedTransactionsFromSender(
{ roundsToSync: 1, syncBehaviour: 'skip-sync-newest', watermark: 0 },
testAccount,
algod,
)

expect(subscribed.currentRound).toBe(lastTxnRound)
expect(subscribed.newWatermark).toBe(lastTxnRound)
expect(subscribed.syncedRoundRange).toEqual([lastTxnRound, lastTxnRound])
expect(subscribed.subscribedTransactions.length).toBe(1)
expect(subscribed.subscribedTransactions[0].id).toBe(txns[1].transaction.txID())
})

test('Only processes the latest transaction when starting from an earlier roudn with other transactions', async () => {
const { algod, testAccount } = localnet.context
const { lastTxnRound: olderTxnRound } = await SendXTransactions(2, testAccount, algod)
const { txns, lastTxnRound: currentRound } = await SendXTransactions(1, testAccount, algod)

const subscribed = await GetSubscribedTransactionsFromSender(
{ roundsToSync: 1, syncBehaviour: 'skip-sync-newest', watermark: olderTxnRound - 1 },
testAccount,
algod,
)

expect(subscribed.currentRound).toBe(currentRound)
expect(subscribed.newWatermark).toBe(currentRound)
expect(subscribed.syncedRoundRange).toEqual([currentRound, currentRound])
expect(subscribed.subscribedTransactions.length).toBe(1)
expect(subscribed.subscribedTransactions[0].id).toBe(txns[0].transaction.txID())
})

test('Process multiple transactions', async () => {
const { algod, testAccount } = localnet.context
const { txns, lastTxnRound } = await SendXTransactions(3, testAccount, algod)

const subscribed = await GetSubscribedTransactionsFromSender(
{ roundsToSync: 2, syncBehaviour: 'skip-sync-newest', watermark: 0 },
testAccount,
algod,
)

expect(subscribed.currentRound).toBe(lastTxnRound)
expect(subscribed.newWatermark).toBe(lastTxnRound)
expect(subscribed.syncedRoundRange).toEqual([lastTxnRound - 1, lastTxnRound])
expect(subscribed.subscribedTransactions.length).toBe(2)
expect(subscribed.subscribedTransactions[0].id).toBe(txns[1].transaction.txID())
expect(subscribed.subscribedTransactions[1].id).toBe(txns[2].transaction.txID())
})
})
47 changes: 0 additions & 47 deletions tests/scenarios/skip-to-newest.spec.ts

This file was deleted.

Loading

0 comments on commit 9cd89fb

Please sign in to comment.