diff --git a/docs/developer-tools/runtime/services/marlowe-chain-indexer.md b/docs/developer-tools/runtime/services/marlowe-chain-indexer.md index 4cec3cb..117aaad 100644 --- a/docs/developer-tools/runtime/services/marlowe-chain-indexer.md +++ b/docs/developer-tools/runtime/services/marlowe-chain-indexer.md @@ -5,7 +5,7 @@ title: Marlowe chain indexer service The `marlowe-chain-indexer` executable follows a local blockchain node and writes the blocks and transactions to a database. ```console -marlowe-chain-indexer : a chain indexer for the Marlowe Runtime. +marlowe-chain-indexer: Chain indexer for the Marlowe Runtime. Usage: marlowe-chain-indexer [--version] (-s|--socket-path SOCKET_FILE) [-m|--testnet-magic INTEGER] @@ -16,7 +16,20 @@ Usage: marlowe-chain-indexer [--version] (-s|--socket-path SOCKET_FILE) [--block-cost COST_UNITS] [--tx-cost COST_UNITS] [--max-cost COST_UNITS] [--http-port PORT_NUMBER] - Chain indexer for Marlowe Runtime. + The chain indexer for the Marlowe Runtime. This component connects to a local + Cardano Node and follows the chain. It copies a subset of the information + contained in every block to a postgresql database. This database can be queried + by downstream components, such as marlowe-chain-sync. + + There should only be one instance of marlowe-chain-indexer writing data to a + given chain database. There is no need to run multiple indexers. If you would + like to scale runtime services, it is recommended to deploy a postgres replica + cluster, run one indexer to populate it, and as many marlowe-chain-sync + instances as required to read from it. + + Before running the indexer, the database must be created and migrated using + sqitch. The migration plan and SQL scripts are included in the source code + folder for marlowe-chain-indexer. Available options: -h,--help Show this help text @@ -45,7 +58,7 @@ Available options: when persisting blocks. If the cost of the current batch would exceed this value, the chain sync client will wait until the current batch is persisted before - requesting another block. (default: 100000) + requesting another block. (default: 1000000) --http-port PORT_NUMBER Port number to serve the http healthcheck API on (default: 8080) ``` diff --git a/docs/developer-tools/runtime/services/marlowe-chain-sync.md b/docs/developer-tools/runtime/services/marlowe-chain-sync.md index 47b6ca8..02b3223 100644 --- a/docs/developer-tools/runtime/services/marlowe-chain-sync.md +++ b/docs/developer-tools/runtime/services/marlowe-chain-sync.md @@ -5,15 +5,36 @@ title: Marlowe chain sync service The `marlowe-chain-sync` executable provides services for querying the blockchain for information that may relate to Marlowe contracts. ```console -marlowe-chain-sync : a chain sync server for the Marlowe Runtime. +marlowe-chain-sync: Chain query and sync server for the Marlowe Runtime. -Usage: marlowe-chain-sync [--version] (-s|--socket-path SOCKET_FILE) +Usage: marlowe-chain-sync [--version] (-s|--socket-path SOCKET_FILE) [-m|--testnet-magic INTEGER] (-d|--database-uri DATABASE_URI) [-h|--host HOST_NAME] - [--port PORT_NUMBER] [--query-port PORT_NUMBER] + [--port PORT_NUMBER] [--query-port PORT_NUMBER] [--job-port PORT_NUMBER] [--http-port PORT_NUMBER] - Chain sync server for Marlowe Runtime. + The chain query engine for the Marlowe Runtime. This component exposes three + protocols through which downstream components can interact with the blockchain. + These are: chain seek, chain query, and chain command. + + The chain seek protocol is a synchronization protocol which allows the follower + to jump directly ahead to blocks that match a particular query. + + The chain query protocol allows various network parameters and UTxO state to be + queried. + + The chain command protocol allows transactions to be submitted to the connected node. + + marlowe-chain-sync relies on the connected database being migrated and populated by + a marlowe-chain-indexer instance. While marlowe-chain-sync can operate without + marlowe-chain-indexer running, marlowe-chain-indexer must be run first to insert + the genesis UTxOs before marlowe-chain-sync can be used, and relies on + marlowe-chain-indexer to keep the database up-to-date. + + marlowe-chain-sync is designed to scale horizontally. That is to say, multiple + instances can run in parallel to scale with demand. A typical setup for this would + involve running multiple marlowe-chain-sync instances in front of a load balancer + against a scalable postgres replica cluster being populated by a single chain indexer. Available options: -h,--help Show this help text diff --git a/docs/developer-tools/runtime/services/marlowe-contract.md b/docs/developer-tools/runtime/services/marlowe-contract.md index 1120637..89538ce 100644 --- a/docs/developer-tools/runtime/services/marlowe-contract.md +++ b/docs/developer-tools/runtime/services/marlowe-contract.md @@ -5,19 +5,49 @@ title: Marlowe contract service The `marlowe-contract` executable provides services related to managing the Marlowe Runtime contract store. ```console -marlowe-contract : a contract storage service for the Marlowe Runtime. +marlowe-contract: Contract storage service for the Marlowe Runtime. -Usage: marlowe-contract [-h|--host HOST_NAME] [-p|--port PORT_NUMBER] - [--query-port PORT_NUMBER] [--transfer-port PORT_NUMBER] - [-b|--buffer-size INTEGER] [-s|--store-dir DIR] - [--store-staging-dir DIR] +Usage: marlowe-contract [--version] [-h|--host HOST_NAME] + [-p|--port PORT_NUMBER] [--query-port PORT_NUMBER] + [--transfer-port PORT_NUMBER] + [--chain-sync-host HOST_NAME] + [--chain-sync-query-port PORT_NUMBER] + [--marlowe-sync-host HOST_NAME] + [--marlowe-bulk-port PORT_NUMBER] + [-b|--buffer-size INTEGER] [-s|--store-dir DIR] + [--store-staging-dir DIR] [--store-lock-microseconds-between-retries MICRO_SECONDS] - [--http-port PORT_NUMBER] + [--http-port PORT_NUMBER] [--min-contract-age MINUTES] + [--max-store-size BYTES] - Contract storage service for Marlowe Runtime + The contract storage service for the Marlowe Runtime. It manages a crucial component of the + Marlowe runtime: the contract store. The contract store is a content-addressable store of + contracts indexed by their hashes. Contracts can refer to sub-contracts via their hashes via + the MerkleizedCase construct. The contract store is used to store the continuations of a contract + after it has been merkleized (a process which recursively replaces Case constructs with MerkleizedCase + constructs). It is also used to lookup continuations when applying inputs to a merkleized contract. + This component exposes three protocols: marlowe load, marlowe transfer, and contract store query. + + The marlowe load protocol is one way to import a contract incrementally into the store. It presents + a stack-based interface for pushing a contract depth-first into the store. + + The marlowe transfer protocol is the other way to import a contract incrementally into the store. + It leverages the Marlowe object model to allow bundles of user-defined marlowe objects to be streamed + into the store. marlowe-contract will link the contract on-the-fly, merkleize the intermediate contracts, + and build the final contract incrementally. This protocol is generally more efficient and flexible than + Marlowe load because it allows duplicate sub-contracts to be pre-abstracted by the user. It also supports + An export mode which will stream the closure of a contract from the store to the client as a Marlowe object + bundle. This can be used to export continuations from the store and share it with other contract stores. + + The contract store query protocol provides queries that allow clients to fetch contracts by their hash, + fetch the adjacency or closure sets of a contract, or merkleize an input. + + marlowe-contract depends on a marlowe-sync and marlowe-chain-sync instance to run automatic + Garbage collection. These must both be running in order for marlowe-contract to run. Available options: -h,--help Show this help text + --version Show version. -h,--host HOST_NAME The host name to run the server on. (default: "127.0.0.1") -p,--port PORT_NUMBER The port number to run the marlowe load server on. @@ -27,14 +57,36 @@ Available options: --transfer-port PORT_NUMBER The port number to run the transfer server on. (default: 3729) + --chain-sync-host HOST_NAME + The host name of the chain sync server. + (default: "127.0.0.1") + --chain-sync-query-port PORT_NUMBER + The chain sync server's query protocol port. + (default: 3716) + --marlowe-sync-host HOST_NAME + The hostname of the Marlowe Runtime marlowe-sync + server. Can be set as the environment variable + MARLOWE_RT_SYNC_HOST (default: "127.0.0.1") + --marlowe-bulk-port PORT_NUMBER + The port number of the marlowe-sync server's bulk + synchronization API. Can be set as the environment + variable MARLOWE_RT_SYNC_MARLOWE_BULK_PORT + (default: 3730) -b,--buffer-size INTEGER The number of contracts to accept from the client before flushing to disk. (default: 512) - -s,--store-dir DIR The root directory of the contract store. + -s,--store-dir DIR The root directory of the contract store + (default: "/home/jamie/.local/share/marlowe/runtime/marlowe-contract/store") --store-staging-dir DIR The root directory of the contract store staging - areas. + areas (default: "/tmp/nix-shell.er7Lmt") --store-lock-microseconds-between-retries MICRO_SECONDS The number of microseconds to wait between retries when acquiring the store lock (default: 500000) --http-port PORT_NUMBER Port number to serve the http healthcheck API on (default: 8080) + --min-contract-age MINUTES + The minimum age contracts in the store must reach + before they can be garbage collected. + (default: 86400s) + --max-store-size BYTES The maximum allowed size of the contract store, in + bytes. (default: 34359738368) ``` diff --git a/docs/developer-tools/runtime/services/marlowe-indexer.md b/docs/developer-tools/runtime/services/marlowe-indexer.md index 003564c..85ed64b 100644 --- a/docs/developer-tools/runtime/services/marlowe-indexer.md +++ b/docs/developer-tools/runtime/services/marlowe-indexer.md @@ -5,18 +5,31 @@ title: Marlowe indexer service The `marlowe-indexer` executable follows `marlowe-chain-sync` and writes Marlowe contract transactions to a database. ```console -marlowe-indexer : a contract indexing service for the Marlowe Runtime. +marlowe-indexer: Contract indexing service for the Marlowe Runtime. -Usage: marlowe-indexer [--chain-sync-port PORT_NUMBER] - [--chain-sync-query-port PORT_NUMBER] +Usage: marlowe-indexer [--version] [--chain-sync-port PORT_NUMBER] + [--chain-sync-query-port PORT_NUMBER] [--chain-sync-host HOST_NAME] - (-d|--database-uri DATABASE_URI) + (-d|--database-uri DATABASE_URI) [--http-port PORT_NUMBER] - Contract indexing service for Marlowe Runtime + The contract indexer for the Marlowe Runtime. It connects to a marlowe-chain-sync + instance using both the chain seek and chain query protocol. It scans the chain for + Marlowe contract transactions and saves them in a postgresql database. This database + can be queried by downstream components, such as marlowe-sync. + + There should only be one instance of marlowe-indexer writing data to a given marlowe + database. There is no need to run multiple indexers. If you would like to scale runtime + services, it is recommended to deploy a postgres replica cluster, run one indexer to + populate it, and as many marlowe-sync instances as required to read from it. + + Before running the indexer, the database must be created and migrated using + sqitch. The migration plan and SQL scripts are included in the source code + folder for marlowe-indexer. Available options: -h,--help Show this help text + --version Show version. --chain-sync-port PORT_NUMBER The port number of the chain sync server. (default: 3715) diff --git a/docs/developer-tools/runtime/services/marlowe-proxy.md b/docs/developer-tools/runtime/services/marlowe-proxy.md index 3d60e08..f2db1e1 100644 --- a/docs/developer-tools/runtime/services/marlowe-proxy.md +++ b/docs/developer-tools/runtime/services/marlowe-proxy.md @@ -5,23 +5,29 @@ title: Marlowe proxy service The `marlowe-proxy` executable provides a single unified public API for the whole runtime. ```console -marlowe-proxy : an API proxy service for the Marlowe Runtime. +marlowe-proxy: The API gateway server for the Marlowe Runtime. -Usage: marlowe-proxy [-h|--host HOST_NAME] [-p|--port PORT_NUMBER] +Usage: marlowe-proxy [--version] [-h|--host HOST_NAME] [-p|--port PORT_NUMBER] [--port-traced PORT_NUMBER] [--marlowe-sync-host HOST_NAME] - [--marlowe-sync-port PORT_NUMBER] - [--marlowe-header-port PORT_NUMBER] - [--marlowe-query-port PORT_NUMBER] - [--marlowe-contract-host HOST_NAME] - [--marlowe-load-port PORT_NUMBER] - [--marlowe-transfer-port PORT_NUMBER] - [--contract-query-port PORT_NUMBER] [--tx-host HOST_NAME] + [--marlowe-sync-port PORT_NUMBER] + [--marlowe-header-port PORT_NUMBER] + [--marlowe-bulk-port PORT_NUMBER] + [--marlowe-query-port PORT_NUMBER] + [--marlowe-contract-host HOST_NAME] + [--marlowe-load-port PORT_NUMBER] + [--marlowe-transfer-port PORT_NUMBER] + [--contract-query-port PORT_NUMBER] [--tx-host HOST_NAME] [--tx-command-port PORT_NUMBER] [--http-port PORT_NUMBER] - API proxy service for Marlowe Runtime + The API gateway server for the Marlowe Runtime. It exposes all the public protocols of the + Marlowe runtime as a single multiplexed protocol: marlowe sync, marlowe header sync, marlowe + bulk sync, marlowe query, marlowe transaction job, marlowe load, marlowe transfer, and contract + store query. Please consult the help text for the individual services for detailed descriptions + of these individual protocols. Available options: -h,--help Show this help text + --version Show version. -h,--host HOST_NAME The host name to run the server on. (default: "127.0.0.1") -p,--port PORT_NUMBER The port number to run the server on. (default: 3700) @@ -42,6 +48,11 @@ Available options: synchronization API. Can be set as the environment variable MARLOWE_RT_SYNC_MARLOWE_HEADER_PORT (default: 3725) + --marlowe-bulk-port PORT_NUMBER + The port number of the marlowe-sync server's bulk + synchronization API. Can be set as the environment + variable MARLOWE_RT_SYNC_MARLOWE_BULK_PORT + (default: 3730) --marlowe-query-port PORT_NUMBER The port number of the marlowe-sync server's query API. Can be set as the environment variable diff --git a/docs/developer-tools/runtime/services/marlowe-sync.md b/docs/developer-tools/runtime/services/marlowe-sync.md index 6c39b7f..955b2eb 100644 --- a/docs/developer-tools/runtime/services/marlowe-sync.md +++ b/docs/developer-tools/runtime/services/marlowe-sync.md @@ -5,19 +5,50 @@ title: Marlowe sync service The `marlowe-sync` executable provides protocols that access Marlowe contract information. ```console -marlowe-sync : a contract synchronization and query service for the Marlowe +marlowe-sync: Contract synchronization and query service for the Marlowe Runtime. -Usage: marlowe-sync (-d|--database-uri DATABASE_URI) [--sync-port PORT_NUMBER] - [--header-sync-port PORT_NUMBER] [--query-port PORT_NUMBER] - [-h|--host HOST_NAME] [--chain-sync-host HOST_NAME] - [--chain-sync-query-port PORT_NUMBER] +Usage: marlowe-sync [--version] (-d|--database-uri DATABASE_URI) + [--sync-port PORT_NUMBER] [--header-sync-port PORT_NUMBER] + [--bulk-sync-port PORT_NUMBER] [--query-port PORT_NUMBER] + [-h|--host HOST_NAME] [--chain-sync-host HOST_NAME] + [--chain-sync-query-port PORT_NUMBER] [--http-port PORT_NUMBER] - Contract synchronization and query service for Marlowe Runtime + The contract query engine for the Marlowe Runtime. This component exposes four + protocols through which downstream components can interact with the blockchain. + These are: marlowe sync, marlowe header sync, marlowe bulk sync, and marlowe query. + + The marlowe sync protocol is a synchronization protocol which follows the history + of a specific marlowe contract. + + The marlowe header sync protocol is a synchronization protocol which scans the chain + for new Marlowe contracts and presents them as a compact summary called a header. + + The marlowe bulk sync protocol is a synchronization protocol which combines the + capabilities of marlowe header sync and marlowe sync. It presents a stream of blocks + which contain a combination of all three contract transaction types: creation, input + application, and payout withdrawal. + + The marlowe query protocol supports multiple queries that allow clients to fetch + data about marlowe contracts as of the current blockchain tip. This means that it + cannot guarantee consistent results between different queries, because the chain could + update in between queries, changing the result of queries. If consistency is needed, + use one of the sync protocols. + + marlowe-sync relies on the connected database being migrated and populated by a + marlowe-indexer instance. While marlowe-sync can operate without marlowe-indexer running, + The sqitch migrations must first be performed in order to create the expected tables, and + marlowe-indexer must be running to keep the database up-to-date. + + marlowe-sync is designed to scale horizontally. That is to say, multiple instances can run + in parallel to scale with demand. A typical setup for this would involve running multiple + marlowe-sync instances in front of a load balancer against a scalable postgres replica + cluster being populated by a single marlowe-indexer. Available options: -h,--help Show this help text + --version Show version. -d,--database-uri DATABASE_URI URI of the database where the contract information is saved. @@ -26,6 +57,9 @@ Available options: --header-sync-port PORT_NUMBER The port number to run the header sync protocol on. (default: 3725) + --bulk-sync-port PORT_NUMBER + The port number to run the bulk sync protocol on. + (default: 3730) --query-port PORT_NUMBER The port number to run the query protocol on. (default: 3726) -h,--host HOST_NAME The host name to run the server on. diff --git a/docs/developer-tools/runtime/services/marlowe-tx.md b/docs/developer-tools/runtime/services/marlowe-tx.md index c471be6..f835445 100644 --- a/docs/developer-tools/runtime/services/marlowe-tx.md +++ b/docs/developer-tools/runtime/services/marlowe-tx.md @@ -5,22 +5,98 @@ title: Marlowe transaction service The `marlowe-tx` executable provides services related to building and submitting transactions for Marlowe contracts. ```console -marlowe-tx : the transaction creation server of the Marlowe Runtime - -Usage: marlowe-tx [--chain-sync-port PORT_NUMBER] - [--chain-sync-query-port PORT_NUMBER] - [--chain-sync-command-port PORT_NUMBER] - [--chain-sync-host HOST_NAME] - [--contract-query-port PORT_NUMBER] - [--contract-host HOST_NAME] [--command-port PORT_NUMBER] - [-h|--host HOST_NAME] [--submit-confirmation-blocks INTEGER] +marlowe-tx: Transaction creation server for the Marlowe Runtime. + +Usage: marlowe-tx [--version] [--chain-sync-port PORT_NUMBER] + [--chain-sync-query-port PORT_NUMBER] + [--chain-sync-command-port PORT_NUMBER] + [--chain-sync-host HOST_NAME] + [--contract-query-port PORT_NUMBER] + [--contract-host HOST_NAME] [--command-port PORT_NUMBER] + [-h|--host HOST_NAME] [--submit-confirmation-blocks INTEGER] [--analysis-timeout SECONDS] [--http-port PORT_NUMBER] --minting-policy-cmd CMD - Marlowe runtime transaction creation server + The transaction creation engine for the Marlowe Runtime. This component exposes + a job protocol with four commands: create, apply inputs, withdraw, and submit. + + Create Command + + The create command will create an unsigned transaction body that starts a new Marlowe + contract. The options that can be configured for this command are: + + • Adding a staking credential to the Marlowe script address for the contract. + • What addresses can be used for coin selection. + • Which address to send change to. + • Rules for role tokens in the contract. Includes rules for minting and distributing + role tokens in the creation transaction and rules for using existing role tokens. + • Adding metadata to the transaction + • The amount that should be deposited in the contract to cover min UTxO rules + (can be omitted and the runtime will compute a worst-case value automatically. + • The initial contract or the hash of a contract in the contract store to use as the + initial contract. + + Apply Inputs Command + + The apply inputs command will create an unsigned transaction body that advances a + Marlowe contract by applying transaction inputs to it. The options that can be + configured for this command are: + + • What addresses can be used for coin selection. + • Which address to send change to. + • Adding metadata to the transaction + • The transaction's validity interval. If omitted, the runtime will compute default + values from the current contract and the most recent block's slot number. + • The initial contract or the hash of a contract in the contract store to use as the + initial contract. + + Withdraw Command + + The withdraw command will create an unsigned transaction body that withdraws role payouts + from Marlowe contracts from the role payout validators that hold them. When a contract + makes a payment to a role, it is not sent directly to the holder of the role token. Instead, + it is sent to an auxiliary Plutus script called a payout validator. The holder of the matching + role token is then able to spend this script output to withdraw the payout, sending it to an + address of their choice. That is what withdrawal means in this context. The options that can be + configured for this command are: + + • What addresses can be used for coin selection. + • Which address to send change to. + • Which payout outputs to withdraw. + + Submit Command + + The submit command will submit a signed transaction to a cardano node via a marlowe-chain-sync + instance. It waits until the transaction is confirmed and found in a block by marlowe-chain-sync + The options that can be configured for this command are: + + • The era of the transaction (babbage or conway) + • The transaction to submit. + + Dependencies + + marlowe-tx depends on a marlowe-chain-sync instance at various points. First, it runs a + chain seek client for the lifetime of the service to keep track of what the current tip + of the blockchain is. Second, it connects via both chain seek and chain query to fetch + current information about the UTxO for wallets and marlowe contracts needed to create + transactions. + + marlowe-tx also depends on a marlowe-contract instance both to create contracts and + apply inputs to merkleized contracts. For creation, it queries the store for the initial + contract if a contract hash was provided in the creation command. It also fetches the + closure of contract continuations from the store to analyze the contract for safety issues. + Finally, it queries contract hashes found in merkleized cases to build merkleized inputs + automatically while executing an apply inputs command. + + Scaling + + marlowe-tx is designed to scale horizontally. That is to say, multiple instances can run + in parallel to scale with demand. A typical setup for this would involve running multiple + marlowe-tx instances in front of a load balancer. Available options: -h,--help Show this help text + --version Show version. --chain-sync-port PORT_NUMBER The port number of the chain sync server. (default: 3715) @@ -55,5 +131,7 @@ Available options: --http-port PORT_NUMBER Port number to serve the http healthcheck API on (default: 8080) --minting-policy-cmd CMD A command which creates the role token minting policy - for a contract. + for a contract. It should read the arguments via the + command line and output the serialized script binary + to stdout. ```