Release 1.205.0

.github/workflows/build.yml

@@ -85,7 +85,7 @@ jobs:
           version: ${{ env.POETRY_VERSION }}
 
       - name: Install build dependencies
-        run: python -m pip install --upgrade pip setuptools wheel pre-commit msgspec
+        run: python -m pip install --upgrade pip setuptools wheel pre-commit msgspec pytest-codspeed
 
       # ta-lib Python install currently broken
       # https://github.com/TA-Lib/ta-lib-python/issues/655
@@ -164,6 +164,14 @@ jobs:
           make pytest
           make test-examples
 
+      # Run codspeed for latest python-version only
+      # - name: Run benchmarks (Linux)
+      #   if: ${{ matrix.python-version == '3.12' }}
+      #   uses: CodSpeedHQ/action@v3
+      #   with:
+      #     token: ${{ secrets.CODSPEED_TOKEN }}
+      #     run: make test-performance
+
   build-windows:
     strategy:
       fail-fast: false

.pre-commit-config.yaml

@@ -83,7 +83,7 @@ repos:
         exclude: "docs/_pygments/monokai.py"
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.7.0
+    rev: v0.7.2
     hooks:
       - id: ruff
         args: ["--fix"]
@@ -112,7 +112,7 @@ repos:
         ]
 
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.12.1
+    rev: v1.13.0
     hooks:
       - id: mypy
         args: [

Makefile

@@ -157,6 +157,10 @@ pytest:
 pytest-coverage:
 	bash scripts/test-coverage.sh
 
+.PHONY: test-performance
+test-performance:
+	bash scripts/test-performance.sh
+
 .PHONY: test-examples
 test-examples:
 	bash scripts/test-examples.sh

README.md

@@ -1,6 +1,7 @@
 # <img src="https://github.com/nautechsystems/nautilus_trader/blob/develop/docs/_images/nautilus-trader-logo.png" width="500">
 
 [![codecov](https://codecov.io/gh/nautechsystems/nautilus_trader/branch/master/graph/badge.svg?token=DXO9QQI40H)](https://codecov.io/gh/nautechsystems/nautilus_trader)
+[![codspeed](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://codspeed.io/nautechsystems/nautilus_trader)
 ![pythons](https://img.shields.io/pypi/pyversions/nautilus_trader)
 ![pypi-version](https://img.shields.io/pypi/v/nautilus_trader)
 ![pypi-format](https://img.shields.io/pypi/format/nautilus_trader?color=blue)
@@ -38,7 +39,7 @@ NautilusTraders design, architecture and implementation philosophy holds softwar
 highest level, with the aim of supporting Python native, mission-critical, trading system backtesting
 and live deployment workloads.
 
-The platform is also univ2rsal and asset class agnostic - with any REST, WebSocket or FIX API able to be integrated via modular
+The platform is also universal and asset class agnostic - with any REST, WebSocket or FIX API able to be integrated via modular
 adapters. Thus, it can handle high-frequency trading operations for any asset classes
 including FX, Equities, Futures, Options, CFDs, Crypto and Betting - across multiple venues simultaneously.
 
@@ -136,7 +137,7 @@ The following integrations are currently supported:
 | [Interactive Brokers](https://www.interactivebrokers.com) | `INTERACTIVE_BROKERS` | Brokerage (multi-venue) | ![status](https://img.shields.io/badge/stable-green)    | [Guide](https://nautilustrader.io/docs/latest/integrations/ib.html)         |
 | [OKX](https://okx.com)                                    | `OKX`                 | Crypto Exchange (CEX)   | ![status](https://img.shields.io/badge/building-orange) | [Guide](https://nautilustrader.io/docs/latest/integrations/okx.html)        |
 | [Polymarket](https://polymarket.com)                      | `POLYMARKET`          | Prediction Market (DEX) | ![status](https://img.shields.io/badge/beta-yellow)     | [Guide](https://nautilustrader.io/docs/latest/integrations/polymarket.html) |
-| [Tardis](https://tardis.dev)                              | `TARDIS`              | Data Provider           | ![status](https://img.shields.io/badge/building-orange) | [Guide](https://nautilustrader.io/docs/latest/integrations/tardis.html)     |
+| [Tardis](https://tardis.dev)                              | `TARDIS`              | Data Provider           | ![status](https://img.shields.io/badge/beta-yellow)     | [Guide](https://nautilustrader.io/docs/latest/integrations/tardis.html)     |
 
 - **ID**: The default client ID for the integrations adapter clients.
 - **Type**: The type of integration (often the venue type).

RELEASES.md

@@ -1,3 +1,51 @@
+# NautilusTrader 1.205.0 Beta
+
+Released on 3rd November 2024 (UTC).
+
+### Enhancements
+- Added Tardis Machine and HTTP API integration in Python and Rust
+- Added `LiveExecEngineConfig.open_check_interval_secs` config option to actively reconcile open orders with the venue
+- Added aggregation of bars from historical data (#2002), thanks @faysou
+- Added monthly and weekly bar aggregations (#2025), thanks @faysou
+- Added `raise_exception` optional parameter to `TradingNode.run` (#2021), thanks @faysou
+- Added `OrderBook.get_avg_px_qty_for_exposure` in Rust (#1893), thanks @elementace
+- Added timeouts to Interactive Brokers adapter configurations (#2026), thanks @rsmb7z
+- Added optional time origins for time bar aggregation (#2028), thanks @faysou
+- Added Polymarket position status reports and order status report generation based on fill reports
+- Added USDC.e (PoS) currency (used by Polymarket) to internal currency map
+- Upgraded Polymarket WebSocket API to new version
+
+### Internal Improvements
+- Ported analysis subpackage to Rust (#2016), thanks @Pushkarm029
+- Improved Postgres testing (#2018), thanks @filipmacek
+- Improved Redis version parsing to support truncated versions (improves compatibility with Redis-compliant databases)
+- Refined Arrow serialization (record batch functions now also available in Rust)
+- Refined core `Bar` API to remove unnecessary unwraps
+- Standardized network client logging
+- Fixed all pyo3 deprecations for API breaking changes
+- Fixed all clippy warning lints for PyO3 changes (#2030), thanks @Pushkarm029
+- PyO3 upgrade refactor and repair catalog tests (#2032), thanks @twitu
+- Upgraded `pyo3` crate to v0.22.5
+- Upgraded `pyo3-async-runtimes` crate to v0.22.0
+- Upgraded `tokio` crate to v1.41.0
+
+### Breaking Changes
+- Removed pyo3 `DataTransformer` (was being used for namespacing, so refactored to separate functions)
+- Moved `TEST_DATA_DIR` constant from `tests` to `nautilus_trader` package (#2020), thanks @faysou
+
+### Fixes
+- Fixed use of Redis `KEYS` command which, is unsupported in cluster environments (replaced with `SCAN` for compatibility)
+- Fixed decoding fill HTTP messages for dYdX (#2022), thanks @davidsblom
+- Fixed account balance report for dYdX (#2024), thanks @davidsblom
+- Fixed Interactive Brokers market data client subscription log message (#2012), thanks @marcodambros
+- Fixed Polymarket execution reconciliation (was not able to reconcile from closed orders)
+- Fixed catalog query mem leak test (#2031), thanks @Pushkarm029
+- Fixed `OrderInitialized.to_dict()` `tags` value type to `list[str]` (was a concatenated `str`)
+- Fixed `OrderInitialized.to_dict()` `linked_order_ids` value type to `list[str]` (was a concatenated `str`)
+- Fix Betfair clients shutdown (#2037), thanks @limx0
+
+---
+
 # NautilusTrader 1.204.0 Beta
 
 Released on 22nd October 2024 (UTC).
@@ -33,7 +81,7 @@ Released on 22nd October 2024 (UTC).
 - Removed legacy `TardisQuoteDataLoader` (now redundant with new Rust implemented loader)
 - Removed legacy `TardisTradeDataLoader` (now redundant with new Rust implemented loader)
 - Custom signals are now passed to `on_signal(signal)` instead of `on_data(data)`
-- Changed `Position.to_dict()` `commissions` value type to `list[str]` (rather than an optional `str` of a list of strings)
+- Changed `Position.to_dict()` `commissions` value type to `list[str]` (was an optional `str` of a list of strings)
 - Changed `Position.to_dict()` `avg_px_open` value type to `float`
 - Changed `Position.to_dict()` `avg_px_close` value type to `float | None`
 - Changed `Position.to_dict()` `realized_return` value type to `float | None`

docs/api_reference/adapters/index.md

@@ -19,4 +19,5 @@
    interactive_brokers.md
    okx.md
    polymarket.md
+   tardis.md
 ```

docs/api_reference/adapters/tardis.md

@@ -0,0 +1,19 @@
+# Tardis
+
+```{eval-rst}
+.. automodule:: nautilus_trader.adapters.tardis
+   :show-inheritance:
+   :inherited-members:
+   :members:
+   :member-order: bysource
+```
+
+## Loaders
+
+```{eval-rst}
+.. automodule:: nautilus_trader.adapters.tardis.loaders
+   :show-inheritance:
+   :inherited-members:
+   :members:
+   :member-order: bysource
+```

docs/concepts/backtesting.md

@@ -64,6 +64,14 @@ Data provided for backtesting drives the execution flow. Since a variety of data
 it's crucial that your venue configurations align with the data being provided for backtesting.
 Mismatches between data and configuration can lead to unexpected behavior during execution.
 
+:::info
+NautilusTrader is primarily designed and optimized for backtesting on order book or tick data, providing the highest
+execution granularity and realism. If order book or tick data is unavailable or unsuitable, backtests
+can also be run on bar data; however, users should note that this results in a loss of information and detail,
+reducing execution precision and realism.
+:::
+
+
 ## Venues
 
 When initializing a venue for backtesting, you must specify its internal order `book_type` for execution processing from the following options:

docs/concepts/data.md

@@ -9,8 +9,25 @@ These data types include:
 - `QuoteTick`: Represents the best bid and ask prices along with their sizes at the top-of-book.
 - `TradeTick`: A single trade/match event between counterparties.
 - `Bar`: OHLCV (Open, High, Low, Close, Volume) bar/candle, aggregated using a specified *aggregation method*.
-- `InstrumentStatus`: An instrument level status event.
-- `InstrumentClose`: An instrument closing price.
+- `InstrumentStatus`: An instrument-level status event.
+- `InstrumentClose`: The closing price of an instrument.
+
+NautilusTrader is designed primarily to operate on granular order book data, providing the highest realism
+for execution simulations in backtesting.
+However, backtests can also be conducted on any of the supported market data types, depending on the desired simulation fidelity.
+
+## Order books
+
+A high-performance order book implemented in Rust is available to maintain order book state based on provided data.
+
+`OrderBook` instances are maintained per instrument for both backtesting and live trading, with the following book types available:
+- `L3_MBO`: **Market by order (MBO)** or L3 data, uses every order book event at every price level, keyed by order ID.
+- `L2_MBP`: **Market by price (MBP)** or L2 data, aggregates order book events by price level.
+- `L1_MBP`: **Market by price (MBP)** or L1 data, also known as best bid and offer (BBO), captures only top-level updates.
+
+:::note
+Top-of-book data, such as `QuoteTick`, `TradeTick` and `Bar`, can also be used for backtesting, with markets operating on `L1_MBP` book types.
+:::
 
 ## Timestamps
 
@@ -43,6 +60,84 @@ The following instrument definitions are available:
 - `OptionsSpread`: Represents a generic options spread instrument.
 - `Synthetic`: Represents a synthetic instrument with prices derived from component instruments using a formula.
 
+## Bars and aggregation
+
+A *bar*—also known as a candle, candlestick, or kline—is a data structure that represents price and
+volume information over a specific period, including the opening price, highest price, lowest price, closing price,
+and traded volume (or ticks as a volume proxy). These values are generated using an *aggregation method*,
+which groups data based on specific criteria to create the bar.
+
+The implemented aggregation methods are:
+
+| Name               | Description                                                                | Category     |
+|:-------------------|:---------------------------------------------------------------------------|:-------------|
+| `TICK`             | Aggregation of a number of ticks.                                          | Threshold    |
+| `TICK_IMBALANCE`   | Aggregation of the buy/sell imbalance of ticks.                            | Threshold    |
+| `TICK_RUNS`        | Aggregation of sequential buy/sell runs of ticks.                          | Information  |
+| `VOLUME`           | Aggregation of traded volume.                                              | Threshold    |
+| `VOLUME_IMBALANCE` | Aggregation of the buy/sell imbalance of traded volume.                    | Threshold    |
+| `VOLUME_RUNS`      | Aggregation of sequential runs of buy/sell traded volume.                  | Information  |
+| `VALUE`            | Aggregation of the notional value of trades (also known as "Dollar bars"). | Threshold    |
+| `VALUE_IMBALANCE`  | Aggregation of the buy/sell imbalance of trading by notional value.        | Information  |
+| `VALUE_RUNS`       | Aggregation of sequential buy/sell runs of trading by notional value.      | Threshold    |
+| `MILLISECOND`      | Aggregation of time intervals with millisecond granularity.                | Time         |
+| `SECOND`           | Aggregation of time intervals with second granularity.                     | Time         |
+| `MINUTE`           | Aggregation of time intervals with minute granularity.                     | Time         |
+| `HOUR`             | Aggregation of time intervals with hour granularity.                       | Time         |
+| `DAY`              | Aggregation of time intervals with day granularity.                        | Time         |
+| `WEEK`             | Aggregation of time intervals with week granularity.                       | Time         |
+| `MONTH`            | Aggregation of time intervals with month granularity.                      | Time         |
+
+### Bar types
+
+NautilusTrader defines a unique *bar type* (`BarType`) based on the following components:
+
+- **Instrument ID** (`InstrumentId`): Specifies the particular instrument for the bar.
+- **Bar Specification** (`BarSpecification`):
+  - `step`: Defines the interval or frequency of each bar.
+  - `aggregation`: Specifies the method used for data aggregation (see the list above).
+  - `price_type`: Indicates the price basis of the bar (e.g., bid, ask, mid, last).
+- **Aggregation Source** (`AggregationSource`): Indicates whether the bar was aggregated internally (within Nautilus) or externally (by a trading venue or data provider).
+
+Bar data can be aggregated either internally or externally:
+
+- `INTERNAL`: The bar is aggregated inside the local Nautilus system boundary.
+- `EXTERNAL`: The bar is aggregated outside the local Nautilus system boundary (typically by a trading venue or data provider).
+
+Bar types can also be classified as either *standard* or *composite*:
+
+- **Standard**: Generated from granular market data, such as quotes or trade ticks.
+- **Composite**: Derived from a higher-granularity bar type through subsampling.
+
+### Defining standard bars
+
+You can define bar types from strings using the following convention:
+
+`{instrument_id}-{step}-{aggregation}-{price_type}-{INTERNAL | EXTERNAL}`
+
+For example, to define a `BarType` for AAPL trades (last price) on Nasdaq (XNAS) using a 5-minute interval, aggregated from trades locally by Nautilus:
+```python
+bar_type = BarType.from_str("AAPL.XNAS-5-MINUTE-LAST-INTERNAL")
+```
+
+### Defining composite bars
+
+Composite bars are derived by aggregating higher-granularity bars into the desired bar type.
+To define a composite bar, use a similar convention to standard bars:
+
+`{instrument_id}-{step}-{aggregation}-{price_type}-INTERNAL@{step}-{aggregation}-{INTERNAL | EXTERNAL}`
+
+**Notes**:
+- The derived bar type must use an `INTERNAL` aggregation source (since this is how the bar is aggregated).
+- The sampled bar type must have a higher granularity than the derived bar type.
+- The sampled instrument ID is inferred to match that of the derived bar type.
+- Composite bars can be aggregated *from* `INTERNAL` or `EXTERNAL` aggregation sources.
+
+For example, to define a `BarType` for AAPL trades (last price) on Nasdaq (XNAS) using a 5-minute interval, aggregated locally by Nautilus, from 1-minute interval bars aggregated externally:
+```python
+bar_type = BarType.from_str("AAPL.XNAS-5-MINUTE-LAST-INTERNAL@1-MINUTE-EXTERNAL")
+```
+
 ## Data flow
 
 The platform ensures consistency by flowing data through the same pathways across all system [environment contexts](/concepts/architecture.md#environment-contexts)

docs/concepts/message_bus.md

@@ -64,9 +64,9 @@ def register_serializable_type(
     ...
 ```
 
-- `cls`: The type to register
-- `to_dict`: The delegate to instantiate a dict of primitive types from the object
-- `from_dict`: The delegate to instantiate the object from a dict of primitive types
+- `cls`: The type to register.
+- `to_dict`: The delegate to instantiate a dict of primitive types from the object.
+- `from_dict`: The delegate to instantiate the object from a dict of primitive types.
 
 ## Configuration
 

docs/integrations/betfair.md

@@ -20,6 +20,10 @@ To install from source using poetry:
 poetry install --extras betfair
 ```
 
+## Examples
+
+You can find working live example scripts [here](https://github.com/nautechsystems/nautilus_trader/tree/develop/examples/live/betfair/).
+
 ## Betfair documentation
 
 Betfair provides extensive [documentation](https://developer.betfair.com/en/get-started/) for developers integrating with their exchange APIs.

docs/integrations/binance.md

@@ -19,6 +19,10 @@ To install from source using poetry:
 poetry install --extras binance
 ```
 
+## Examples
+
+You can find working live example scripts [here](https://github.com/nautechsystems/nautilus_trader/tree/develop/examples/live/binance/).
+
 ## Overview
 
 The following documentation assumes a trader is setting up for both live market

docs/integrations/bybit.md

@@ -23,6 +23,10 @@ To install from source using poetry:
 poetry install --extras bybit
 ```
 
+## Examples
+
+You can find working live example scripts [here](https://github.com/nautechsystems/nautilus_trader/tree/develop/examples/live/bybit/).
+
 ## Overview
 
 The following documentation assumes a trader is setting up for both live market