Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add timeboost troubleshooting guide #2113

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
.vscode
arbitrum-docs/sdk
arbitrum-docs/stylus-by-example
stylus-by-example
.vercel
.DS_Store
4 changes: 4 additions & 0 deletions arbitrum-docs/run-arbitrum-node/how-to-use-timeboost.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -496,3 +496,7 @@ timeboosted true
```

In the sequencer feed, the `BroadcastFeedMessage` struct now contains a `blockMetadata` field that represents whether a particular transaction in the block was timeboosted or not. The field blockMetadata is an array of bytes and it starts with a byte representing the version (`0`), followed by `ceil(N/8)` number of bytes where `N` is the number of transactions in the block. If a particular transaction was timeboosted, the bit representing its position in the block will be set to `1`, while the rest will be set to `0`. For example, if the `blockMetadata` of a particular message, viewed as bits is `00000000 01100000`, then the 2nd and 3rd transactions in that block were timeboosted.

## Troubleshooting and best practices

Our guide on [Troubleshooting Timeboost](../run-arbitrum-node/troubleshoot-timeboost.mdx) contains more information on how response times work and how express lane transactions are sequenced, alongside common errors and best practices for using Timeboost.
62 changes: 62 additions & 0 deletions arbitrum-docs/run-arbitrum-node/troubleshoot-timeboost.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
title: 'Troubleshoot Timeboost'
description: A guide on common errors & best practices when using Timeboost
user_story: As a developer, I want to understand how to troubleshoot common issues when using Timeboost
author: dlee
content_type: how-to
---

This is a short guide on how response times work and how <a data-quicklook-from="express-lane">Express Lane</a> transactions are sequenced, alongside common errors and best practices for using <a data-quicklook-from="timeboost">Timeboost</a>. This guide assumes you have reviewed our guide on [How to use Timeboost](../run-arbitrum-node/how-to-use-timeboost.mdx).

## How response times work

The response for a <a data-quicklook-from="transaction">Transaction</a> submission is returned only once the block containing the transaction is produced. For example, irrespective of the type of transaction (express lane or normal), if a transaction arrives at the <a data-quicklook-from="sequencer">Sequencer</a> at `t=0ms` and the block is produced at `t=250ms`, then the expected response time is always (`time_to_arrive`) + 250ms, where `time_to_arrive` is the time it took for the transaction to be received by the sequencer.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph seems a bit confusing, since it starts saying that the transaction arrives at t=0ms but later say that it takes time_to_arrive to arrive.

I have some extra comments in general about the sequencer behavior explained here, that I'll share on Slack.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this sound?

Suggested change
The response for a <a data-quicklook-from="transaction">Transaction</a> submission is returned only once the block containing the transaction is produced. For example, irrespective of the type of transaction (express lane or normal), if a transaction arrives at the <a data-quicklook-from="sequencer">Sequencer</a> at `t=0ms` and the block is produced at `t=250ms`, then the expected response time is always (`time_to_arrive`) + 250ms, where `time_to_arrive` is the time it took for the transaction to be received by the sequencer.
The response for a <a data-quicklook-from="transaction">Transaction</a> submission is returned only once the block containing the transaction is produced, plus some time for the message to get through the network (`time_to_arrive`). For example, irrespective of the type of transaction (express lane or normal), if a transaction arrives at the <a data-quicklook-from="sequencer">Sequencer</a> at `t=0ms` and the block is produced at `t=250ms`, then the expected response time is at least (`time_to_arrive`) + 250ms, where `time_to_arrive` is the time it took for the transaction to be received by the sequencer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The paragraph is still confusing, at the beginning it says that the transaction arrives at t=0ms (if a transaction arrives at the Sequencer at t=0ms) but later says that it takes time_to_arrive to arrive (where time_to_arrive is the time...) 😅

Maybe it would be clearer if we say something like "If transaction is sent at t=0ms, and the block is produced at t=250ms, the expected response time is 250ms - time_to_arrive, where time_to_arrive is ..."

wdyt?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see what you're saying, apologies. How's this? Also I think its always going to be 250ms plus time_to_arrive right?

Suggested change
The response for a <a data-quicklook-from="transaction">Transaction</a> submission is returned only once the block containing the transaction is produced. For example, irrespective of the type of transaction (express lane or normal), if a transaction arrives at the <a data-quicklook-from="sequencer">Sequencer</a> at `t=0ms` and the block is produced at `t=250ms`, then the expected response time is always (`time_to_arrive`) + 250ms, where `time_to_arrive` is the time it took for the transaction to be received by the sequencer.
The response for a <a data-quicklook-from="transaction">Transaction</a> submission is returned only once the block containing the transaction is produced. For example, irrespective of the type of transaction (express lane or normal), if a transaction is sent to the <a data-quicklook-from="sequencer">Sequencer</a> at `t=0ms` and the block is produced at `t=250ms`, then the expected response time is at least `time_to_arrive` + 250ms, where `time_to_arrive` is the time it took for the transaction to be received by the sequencer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, that's less confusing, yeah. But I think the timing is still a bit off. If I send my transaction at t=0ms, and the block is produced at t=250ms, it will be confirmed at t=250ms if time_to_arrive is less than 250ms.

For example, say that I send at t=0ms, and time_to_arrive is 50ms. The transaction will make it to the next block at the t=250ms mark, but the current formula suggests that the expected response time will be 50+250ms = 300ms.

If time_to_arrive is more than 250ms, then it will make it to the next block. Always assuming that the block is not filled up before.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good - will make a change.


As another example, assume that a transaction is submitted at `t=150ms`. If the transaction is an express lane transaction, the response would be returned at `t =(time_to_arrive)+ 100ms`. If, however, the submitted transaction is normal, then the transaction’s inclusion will be delayed by 200ms, resulting in a response time at `t=(time_to_arrive)+100ms+250ms` (250ms because this is the time it takes to create the next block).

## How express lane transactions are ordered into blocks

The express lane time advantage is currently set to 200ms, while the current block creation time is 250ms. This means that if an express lane transaction and a normal transaction both arrive at the sequencer at the same time, but _before 50ms have passed since the last block was produced_, then both the transactions may appear in the same block, though the express lane transaction would be sequenced ahead of the normal transaction.

Express lane transactions are processed in the order of their `sequenceNumber`, which is a field in every express lane transaction. The first expected sequence number for a new round is zero and increments for each accepted transaction. You can read more about how to submit express lane transactions in: [How to submit transactions to the express lane](../run-arbitrum-node/how-to-use-timeboost.mdx#how-to-submit-transactions-to-the-express-lane).

## Errors relating to the sequenceNumber

The `sequenceNumber` field when submitting an express lane transaction is important because transactions with `sequenceNumber=n` can only be sequenced after all the transactions from `sequenceNumber=0` to `sequenceNumber=n-1` have been sequenced.

When it comes to submitting express lane transactions, different types of error responses may arise relating to the `sequenceNumber`. Below is a breakdown of the various error messages related to sequence numbers and the recommendation for avoiding them. Error messages are subject to change.

To illustrate how sequence numbers are used to accept or reject express lane transactions, assume first that the sequence number for a given round is `n` (i.e `0` to `n-1` sequence numbers were accepted in this round) and there is some maximum allowed sequence number limit `m` that the sequencer can keep track of (for DoS protections).

If the sequence number for your express lane transaction is:

- Less than `n`, then your transaction will be rejected because it has already accepted a transaction with that sequence number (`error="SEQUENCE_NUMBER_TOO_LOW"`).
- Exactly `n`, then your transaction will be accepted as soon as the sequencer receives it.
- Greater than `n` but less than or equal to `m`, then your transaction will be buffered to be processed. Your transaction will only be accepted once transactions with the other sequence numbers arrive to fill in the “gap” between the sequencer number `n` and your transaction’s sequence number. You may receive a timeout error if the transactions within the gap arrive after the timeout (i.e., between n and your transaction’s sequence number). However, this doesn’t mean your transaction was rejected - it may eventually get processed. So we recommend checking the transaction receipts ([error #1 below](../run-arbitrum-node/troubleshoot-timeboost.mdx#error-1)).
- Greater than `m`, then your transaction will be rejected ([error #2 below](../run-arbitrum-node/troubleshoot-timeboost.mdx#error-2)).

Timeboost's implementation of the above-described sequence number reordering buffer is intended to handle out-of-order delivery of transactions in the case of network delays (en route to the sequencer). This implementation is subject to change as we gather feedback from users.

#### Error 1: Timeout error

`"Accepted timeboost tx failed: Sequence number: %d (consumed), Transaction hash: %v, Error: Transaction sequencing hit timeout, result for the submitted transaction is not yet available: %w"`

This means the transaction was received with a sequence number after the expected sequence number, and the transactions with prior sequence numbers were not received in time.

Using an example to illustrate this case, this error could arise where the sequencer has already sequenced an express lane transaction with `sequenceNumber=3`, but the express lane transaction that was submitted used `sequenceNumber=5`. Since `sequenceNumber=4` has not yet been sequenced, the sequencer will respond by saying the result for the submitted transaction (with `sequenceNumber=5`) is “not yet available”.

Despite this error message, the transaction may still have been processed if transactions with the missing prior sequence numbers are received. Check for the transaction receipt.

#### Error 2: Sequence number reaching the max allowed limit

`"message sequence number has reached max allowed limit. SequenceNumber: %d, Limit: %d",`
This means the `sequenceNumber` of the received transaction is ahead of the next expected sequence number by more than the limit. To resolve this, we simply re-submit your transactions in sequential `sequenceNumber` order.

#### Error 3: Already-consumed `sequenceNumber` related errors

`"Accepted timeboost tx failed: Sequence number: %d (consumed), Transaction hash: %v, Error: $some error unrelated to timeout$”`
This type of error means that the underlying transaction of the express lane submission has erred for another reason, and the corresponding `sequenceNumber` has been consumed. A description of these other types of errors can be found [in our guide about how to submit transactions to the express lane"](../run-arbitrum-node/how-to-use-timeboost.mdx#how-to-submit-transactions-to-the-express-lane).

#### Other Errors

Other errors should be self-explanatory - more details can be found on our page about [How to submit transactions to the express lane](../run-arbitrum-node/how-to-use-timeboost.mdx#how-to-submit-transactions-to-the-express-lane).
10 changes: 10 additions & 0 deletions node_modules/.yarn-integrity

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -697,6 +697,11 @@ const sidebars = {
label: 'Use Timeboost',
id: 'run-arbitrum-node/how-to-use-timeboost',
},
{
type: 'doc',
label: 'Troubleshoot Timeboost',
id: 'run-arbitrum-node/troubleshoot-timeboost',
},
{
type: 'html',
value:
Expand Down
4 changes: 4 additions & 0 deletions yarn.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1