From f19b7348a1b5e0560b1a1b38f92fd1f9aaa17032 Mon Sep 17 00:00:00 2001 From: Zachary Brown Date: Tue, 22 Oct 2024 12:36:19 -0700 Subject: [PATCH 1/2] docs: catch up README with latest changes --- README.md | 174 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 131 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index d7a52338..912e2651 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,15 @@ This project was formerly known as the Climate Warehouse and you may see this te ## User Guide -The Core Registry CADT application is designed to run 24/7, much like any other API. While it is possible to run it on-demand only when API requests need to be made, this guide assumes a permanently running solution. +The CADT application is designed to run 24/7, much like any other API. While it is possible to run it on-demand only when API requests need to be made, this guide assumes a permanently running solution. -The simplest way to run the Core Registry CADT application is to use the same machine the Chia Wallet, Datalayer, and Datalayer HTTP services. Core Registry CADT communicates with the Chia services over an RPC interface. The RPC interface uses certificates to authenticate, which will work automatically when the Core Registry CADT application is run as the same user on the same machine as the Chia services. To run Core Registry CADT on a separate machine from Chia, a public certificate from the Chia node most be used to authenticate (not yet documented). +The simplest way to run the CADT application is to use the same machine the Chia Full Node, Wallet, Datalayer, and Datalayer-HTTP services reside on. CADT communicates with the Chia services over an RPC interface. The RPC interface uses certificates to authenticate, which will work automatically when the CADT application is run as the same user on the same machine as the Chia services. To run CADT on a separate machine from Chia, a public certificate from the Chia node must be used to authenticate (not yet documented). + +Basic Chia installation instructions are provided below, but further installation options, please see the [Chia docs site](https://docs.chia.net/installation/). For most CADT setups, we recommend the installing the headless `chia-blockchain-cli` package via the `apt` repo and using [systemd](https://docs.chia.net/installation/#systemd). + +After the initial installation, it will take anywhere from a few days (most likely for a cloud-hosted server) to a few weeks (possible on lower-powered systems or servers with slow connections) to sync the Chia full node. During this time, CADT can start syncing, but any writes to CADT are likely to fail. For best results, we recommend waiting for the Chia full node to finish syncing before using CADT. Check the status of the full node sync with `chia show -s`. + +*Those familiar with bittorrent and have a fast connection can speed up the full node syncing by [downloading the database](https://www.chia.net/downloads/#database-checkpoint) and copying it into place manually* ### How to use the API @@ -20,12 +26,22 @@ Please see the [Core Registry CADT RPC API Guide](docs/cadt_rpc_api.md). ## Installation -[Releases are tagged in Github](https://github.com/Chia-Network/climate-warehouse/tags) and binaries are built for Windows, MacOS, and Linux. ARM binaries are available for Debian versions of Linux only. +[Releases are tagged in Github](https://github.com/Chia-Network/Core-Registry-CADT/tags) and binaries are built for Windows, MacOS, and Linux. ARM binaries are available for Debian versions of Linux only. ### Linux A binary file that can run on all Linux distributions on x86 hardware can be found for each tagged release with the name `core-registry-cadt-linux-x64-.zip`. This zip file will extract to the `core-registry-cadt-linux-64` directory by default, where the `core-registry-cadt` file can be executed to run the API. +### System Requirements + +CADT and Chia system usage will depend on many factors, including how busy the blockchain is, how much data is being mirrored by DataLayer, and how much data CADT is ingesting and processing. The current minimum requirements for running CADT and Chia together on a system are: + +* 4 CPU cores +* 8 GB RAM +* 300 GB disk space + +ARM and x86 systems are supported. While Windows, MacOS, and all versions of Linux are supported, Ubuntu Linux is the recommended operating system as it is used most in testing and our internal hosting. + #### Debian-based Linux Distros (Ubuntu, Mint, etc) The Core Registry CADT API can be installed with `apt`. Both ARM and x86 versions can be installed this way. @@ -43,31 +59,64 @@ sudo apt-get install ca-certificates curl gnupg curl -sL https://repo.chia.net/FD39E6D3.pubkey.asc | sudo gpg --dearmor -o /usr/share/keyrings/chia.gpg ``` -3. Use the following command to setup the repository. +3. Use the following command to setup the Core-Registry-CADT repository. ``` echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/chia.gpg] https://repo.chia.net/climate-tokenization/debian/ stable main" | sudo tee /etc/apt/sources.list.d/climate-tokenization.list > /dev/null ``` -4. Install CADT +4. Add the Chia repository (via the [official Chia installation instructions](https://docs.chia.net/installation/#using-the-cli)) + +``` +echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/chia.gpg] https://repo.chia.net/debian/ stable main" | sudo tee /etc/apt/sources.list.d/chia.list > /dev/null +``` + +5. Install Chia Blockchain and Core-Registry-CADT ``` sudo apt-get update -sudo apt-get install core-registry-cadt +sudo apt-get install chia-blockchain-cli core-registry-cadt ``` -5. Start CADT with systemd +6. Start Chia Wallet and Datalayer with [systemd](https://docs.chia.net/installation/#systemd) + +``` +sudo systemctl start chia-wallet@ chia-data-layer@ chia-full-node@ +``` + +For ``, enter the user that Chia runs as (the user with the `.chia` directory in their home directory). For example, if the `ubuntu` is where Chia runs, start Chia with `systemctl start chia-wallet@ubuntu`. + +Optional: If using Chia's built-in HTTP server to share datalayer files, start the `chia-data-layer-http` service with + +``` +sudo systemctl start chia-data-layer-http@ +``` + +7. Start CADT with systemd ``` sudo systemctl start core-registry-cadt@ ``` For ``, enter the user that Chia runs as (the user with the `.chia` directory in their home directory). For example, if the `ubuntu` is where Chia runs, start CADT with `systemctl start core-registry-cadt@ubuntu`. -6. Set CADT to run at boot +8. Set Chia and Core-Registry-CADT to run at boot ``` -sudo systemctl enable core-registry-cadt@ +sudo systemctl enable chia-wallet@ chia-data-layer@ chia-full-node@ core-registry-cadt@ +``` + +If using the built-in HTTP server for datalayer, start it at boot with + +``` +sudo systemctl enable chia-data-layer-http@ +``` + +9. View CADT logs to validate + +``` +journalctl -u cadt@ -f +(ctrl+c to exit) ``` ### Installation from Source @@ -88,9 +137,74 @@ npm install npm run start ``` +### Datalayer HTTP File Serving + +CADT relies on all participants publicly sharing their data over Chia Datalayer, which includes sharing the Chia-generated `.dat` files over HTTP. The files are located in `~/.chia/mainnet/data_layer/db/server_files_location_/` (where `` is the Chia network, usually either "mainnet" or "testneta") and can be shared over any web-accessible HTTP endpoint, including + +* Using the built-in datalayer-http service (see [Installation](#installation) instructions below). Datalayer-http runs on port 8575 by default which may need to be opened in your firewall configuration or forwarded by your router. Additionally, a static IP address, or stable DNS record, will be required, which is not offered by default on some hosting providers. On AWS, assign an Elastic IP to the EC2 instance or use an Application Load Balancer to solve this. + +* Using Nginx, Apache, Caddy, or any other web server. This also requires a static IP address, or dynamically assigned DNS record. Another challenge is that the default location for the .dat files is in the user's home directory, which the web server software will not have read-access to. One simple solution is + * `mv ~/.chia/mainnet/data_layer/db/server_files_location_ /var/www/` - move the datalayer file directory outside of the home directory + * `chmod -R 744 /var/www/server_files_location_` - change permissions on all datalayer files to be read by any user + * `ln -s /var/www/server_files_location_ ~/.chia/mainnet/data_layer/db/server_files_location_` - create a shortcut from the old location to the new + * Use [Nginx](https://nginx.org/), [Apache](https://httpd.apache.org/), [Caddy](https://caddyserver.com/), or any web server to serve the files over HTTP. Here is a sample Nginx config: + + ``` + server { + listen 80; + + root /var/www/server_files_location_; + + server_name datalayer.example.com; + + index index.html; + + expires 30d; + add_header Pragma "public"; + add_header Cache-Control "public"; + + } + ``` + +* Use [S3](https://aws.amazon.com/s3/) or other object store. Datalayer .dat files can be synced to any cloud file storage solution that can serve them publicly over HTTP. One recommended solution using S3 is to [use this script and follow the installation and usage instructions in the README](https://github.com/TheLastCicada/Chia-Datalayer-S3-Sync). + +Once the .dat files are publicly available, update `DATALAYER_FILE_SERVER_URL` in the [CADT configuration file](#configuration) with the URL or IP address (always include http:// or https://) and the port, then restart CADT. CADT will begin to create mirrors at this URL for your data and all stores you are subscribed to. + +### Run CADT on a Testnet + +Chia has a few test networks called "[testnets](https://docs.chia.net/testnets/)". Testnets allow anyone to test applications using plentiful and low value TXCH instead of needing to purchace XCH. We recommend running a testnet version of CADT in order to test integrations, software updates, and experiment in a low-stakes environment. + +CADT runs on a testnet called "testnetA" which is different than the main Chia testnet, testnet11. TestnetA has a CADT governance node and an [observer](https://chia-cadt-demo.chiamanaged.com/). To configure your Chia and CADT environment to use testnetA, do the following: + +*Note - these instructions only work with Chia version 2.4.4 and above* + + 1. Follow the [instructions here and install chia-tools](https://github.com/chia-network/chia-tools?tab=readme-ov-file#apt-repo-installation). + + 2. Use chia-tools to switch to testneta in the Chia config + + `chia-tools network switch testneta` + + 3. Restart Chia + + `sudo systemctl restart chia-wallet@ chia-data-layer@ chia-full-node@` + + 4. Stop CADT + + `sudo systemctl stop cadt@` + + 4. Update the `GOVERNANCE_BODY_ID` in `~/.chia/mainnet/cadt/v1/config.yaml` to be `1019153f631bb82e7fc4984dc1f0f2af9e95a7c29df743f7b4dcc2b975857409` + + 5. If you already were running CADT on mainnet, delete the CADT database + + `rm ~/.chia/mainnet/cadt/v1/data.sqlite3*` + + 6. Start CADT + + `sudo systemctl start cadt@` + ### Ports, Networking, and Security -By default, the CADT API will listen on localhost only on port 31310. If running a node with `READ_ONLY` set to `false`, it is highly recommended that CADT is run on a private network or with access limited by IP address. To allow remote connections to CADT, set the `BIND_ADDRESS` (see the [Configuration](#configuration) section below) to the IP to listen on, or `0.0.0.0` to listen on all interfaces. The port for the CADT API can be set with the parameter `CW_PORT`. The default port is 31310. In many cases, users will need to access the API from their workstations for either the [CADT UI](https://github.com/Chia-Network/climate-warehouse-ui) or to integrate with existing tools and scripts. To add authentication to the API, use the `CADT_API_KEY` parameter. Alternatively, the API can be served behind an authentication proxy to restrict access and the `CADT_API_KEY` can be left blank. If running an observer node with `READ_ONLY` set to `true`, the CADT API will only share data from the public blockchain, and running without authentication is usually safe. If `READ_ONLY` is set to `false`, authentication must be used to prevent unauthorized writes to the blockchain. +By default, the CADT API will listen on localhost only on port 31310. If running a node with `READ_ONLY` set to `false`, it is highly recommended that CADT is run on a private network or with access limited by IP address. To allow remote connections to CADT, set the `BIND_ADDRESS` (see the [Configuration](#configuration) section below) to the IP to listen on, or `0.0.0.0` to listen on all interfaces. The port for the CADT API can be set with the parameter `CW_PORT`. The default port is 31310. In many cases, users will need to access the API from their workstations for either the [Core Registry CADT UI](https://github.com/Chia-Network/core-registry-cadt-ui) or to integrate with existing tools and scripts. To add authentication to the API, use the `CADT_API_KEY` parameter. Alternatively, the API can be served behind an authentication proxy to restrict access and the `CADT_API_KEY` can be left blank. If running an observer node with `READ_ONLY` set to `true`, the CADT API will only share data from the public blockchain, and running without authentication is usually safe. If `READ_ONLY` is set to `false`, authentication must be used to prevent unauthorized writes to the blockchain. ### Adding Encryption to the CADT API @@ -98,37 +212,11 @@ The CADT API uses HTTP and is unencrypted. To add encryption, use a reverse pro ### Configuration -In the `CHIA_ROOT` directory (usually `~/.chia/mainnet` on Linux), CADT will add a directory called `cadt/v1` when the application is first run (in fact, this directory could be deleted at any time and CADT will recreate it next time it is started). The main CADT configuration file is called `config.yaml` and can be found in this directory. The options in this file are as follows (the full list of available options can be seen in the [config template](src/utils/defaultConfig.json)): - -* **MIRROR_DB**: This section is for configuring the MySQL-compatible database that can be used for easy querying for report generation. This is optional and only provides a read-only mirror of the data CADT uses. - * **DB_USERNAME**: MySQL username - * **DB_PASSWORD**: MySQL password - * **DB_NAME**: MySQL database name - * **DB_HOST**: Hostname of the MySQL database -* **APP**: This section is for configuring the CADT application. - * **PORT**: CADT port where the API will be available. 31310 by default. - * **BIND_ADDRESS**: By default, CADT listens on localhost only. To enable remote connections to CADT, change this to `0.0.0.0` to listen on all network interfaces, or to an IP address to listen on a specific network interface. - * **DATALAYER_HOST**: URL and port to connect to the [Chia DataLayer RPC](https://docs.chia.net/datalayer-rpc). If Chia is installed locally with default settings, https://localhost:8562 will work. - * **WALLET_HOST**: URL and port to conned to the [Chia Wallet RPC](https://docs.chia.net/wallet-rpc). If Chia is installed on the same machine as CADT with default settings, https://localhost:9256 will work. - * **USE_SIMULATOR**: Developer setting to populate CADT from a governance file and enables some extra APIs. Should always be "false" under normal usage. - * **READ_ONLY**: When hosting an Observer node, set to "true" to prevent any data being written using the CADT APIs. This makes the application safe to run with public endpoints as it is just displaying publicly available data. When running a governance node, or a participant node, set to "false" to allow data to be written to the CADT APIs. When "false", additional authentication or access restrictions must be applied to prevent unauthorized alteration of the data. - * **API_KEY**: This key is used by the [CADT UI](https://github.com/Chia-Network/climate-warehouse-ui) to authenticate with the CADT API endpoints. This allows the API to power the UI only without allowing requests missing the API key in the header to access the API. This can be left blank to allow open access to the API, or if access is restricted by other means. The CADT_API_KEY can be set to any value, but we recommend at least a 32 character random string. The CADT_API_KEY can be passed in a request using the `x-api-key` header. See the [RPC documentation](docs/climate_warehouse_rpc_api.md) for examples. - * **USE_DEVELOPMENT_MODE**: Should be false in most use cases. If a developer writing code for the app, this can be changed to "true" which will bypass the need for a governance node. - * **IS_GOVERNANCE_BODY**: "True" or "false" toggle to enable/disable mode for this instance being a governing body. - * **DEFAULT_FEE**: [Fee](https://docs.chia.net/mempool/) for each transaction on the Chia blockchain in mojos. The default is 300000000 mojos (0.0003 XCH) and can be set higher or lower depending on how [busy](https://dashboard.chia.net/d/46EAA05E/mempool-transactions-and-fees?orgId=1) the Chia network is. If a fee is set very low, it may cause a delay in transaction processing. - * **DEFAULT_COIN_AMOUNT**: Units are mojo. Each DataLayer transaction needs a coin amount and the default is 300000000 mojo. - * **DATALAYER_FILE_SERVER_URL**: Publicly available Chia DataLayer HTTP URL and port, including schema (http:// or https://). If serving DataLayer files from S3, this would be the public URL of the S3 bucket. Port can be omitted if using standard ports for http or https requests. - * **TASKS**: Section for configuring sync intervals - * **AUDIT_SYNC_TASK_INTERVAL**: Default 30 - * **DATAMODEL_SYNC_TASK_INTERVAL**: Default 60 - * **GOVERNANCE_SYNC_TASK_INTERVAL**: Default 86400 - * **ORGANIZATION_META_SYNC_TASK_INTERVAL**: Default 86400 - * **PICKLIST_SYNC_TASK_INTERVAL**: Default 30 -* **GOVERNANCE**: Section on settings for the Governance body to connect to. - * **GOVERNANCE_BODY_ID**: This determines the governance body your CADT network will be connected to. While there could be multiple governance body IDs, the default of `23f6498e015ebcd7190c97df30c032de8deb5c8934fc1caa928bc310e2b8a57e` is the right ID for most people. - -​ -Note that the CADT application will need to be restarted after any changes to the config.yaml file. +This projects uses the unified Core Registry config file, located at `CHIA_ROOT/core-registry/config.yaml` (where `CHIA_ROOT` is configurable but set to `~/.chia/mainnet` on Linux by default). Each application installed will have a top-level YAML element of the project name with application-specific config options below. There is also a `GENERAL` configuration section and `CHIA` section whose config options apply to all applications. This config file will get created at application startup if it doesn't exist. Each application will add it's own section to this config file upon first startup. Options for the `LOG_LEVEL` are [documented here](https://github.com/Chia-Network/core-registry-logger). + +The configuration options of Core Registry CADT closely match the options of the CADT application, [which are well documented](https://github.com/Chia-Network/cadt?tab=readme-ov-file#configuration). Be careful to follow the YAML formatting of the generated [config file](https://github.com/Chia-Network/Core-Registry-CADT/blob/develop/src/utils/defaultConfig.js) for Core Registry CADT. + +Note that the Core Registry CADT application will need to be restarted after any changes to the config.yaml file. ## Developer Guide @@ -136,7 +224,7 @@ A development environment for CADT assumes a synced Chia wallet running locally. ### Contributing -All branches should be created from the `develop` branch and not from `main`. All pull requests should be made against the `develop` branch, unless it is a new release. The `develop` branch will be merged into the `main` branch to create a release. Automation in the CI will create the [release](https://github.com/Chia-Network/cadt/releases) and attach the installation files to it automatically whenever code is merged to `main`. Additionally, the changelog will automatically be updated in the `main` branch. Therefore, the `main` branch should always be a representation of the latest released code. +All branches should be created from the `develop` branch and not from `main`. All pull requests should be made against the `develop` branch, unless it is a new release. The `develop` branch will be merged into the `main` branch to create a release. Automation in the CI will create the [release](https://github.com/Chia-Network/Core-Registry-CADT/releases) and attach the installation files to it automatically whenever code is merged to `main`. Additionally, the changelog will automatically be updated in the `main` branch. Therefore, the `main` branch should always be a representation of the latest released code. ​This repo uses a [commit convention](https://www.conventionalcommits.org/en/v1.0.0/). A typical commit message might read: ​ @@ -187,7 +275,7 @@ chmod ug+x .git/hooks/* ### Build Binaries -After running the ["Installation from Source"](https://github.com/Chia-Network/cadt#installation-from-source) steps above, do the following: +After running the ["Installation from Source"](https://github.com/Chia-Network/Core-Registry-CADT#installation-from-source) steps above, do the following: ``` // transcompile project to es5 From 75f74ec6df5429c9b4b9148f9c1d6bf288b404a3 Mon Sep 17 00:00:00 2001 From: Zachary Brown Date: Tue, 22 Oct 2024 12:39:27 -0700 Subject: [PATCH 2/2] docs: naming fixes --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 912e2651..068dea7b 100644 --- a/README.md +++ b/README.md @@ -10,13 +10,13 @@ This project was formerly known as the Climate Warehouse and you may see this te ## User Guide -The CADT application is designed to run 24/7, much like any other API. While it is possible to run it on-demand only when API requests need to be made, this guide assumes a permanently running solution. +The Core Registry CADT application is designed to run 24/7, much like any other API. While it is possible to run it on-demand only when API requests need to be made, this guide assumes a permanently running solution. -The simplest way to run the CADT application is to use the same machine the Chia Full Node, Wallet, Datalayer, and Datalayer-HTTP services reside on. CADT communicates with the Chia services over an RPC interface. The RPC interface uses certificates to authenticate, which will work automatically when the CADT application is run as the same user on the same machine as the Chia services. To run CADT on a separate machine from Chia, a public certificate from the Chia node must be used to authenticate (not yet documented). +The simplest way to run the Core Registry CADT application is to use the same machine the Chia Full Node, Wallet, Datalayer, and Datalayer-HTTP services reside on. Core Registry CADT communicates with the Chia services over an RPC interface. The RPC interface uses certificates to authenticate, which will work automatically when the Core Registry CADT application is run as the same user on the same machine as the Chia services. To run Core Registry CADT on a separate machine from Chia, a public certificate from the Chia node must be used to authenticate (not yet documented). -Basic Chia installation instructions are provided below, but further installation options, please see the [Chia docs site](https://docs.chia.net/installation/). For most CADT setups, we recommend the installing the headless `chia-blockchain-cli` package via the `apt` repo and using [systemd](https://docs.chia.net/installation/#systemd). +Basic Chia installation instructions are provided below, but further installation options, please see the [Chia docs site](https://docs.chia.net/installation/). For most Core Registry CADT setups, we recommend the installing the headless `chia-blockchain-cli` package via the `apt` repo and using [systemd](https://docs.chia.net/installation/#systemd). -After the initial installation, it will take anywhere from a few days (most likely for a cloud-hosted server) to a few weeks (possible on lower-powered systems or servers with slow connections) to sync the Chia full node. During this time, CADT can start syncing, but any writes to CADT are likely to fail. For best results, we recommend waiting for the Chia full node to finish syncing before using CADT. Check the status of the full node sync with `chia show -s`. +After the initial installation, it will take anywhere from a few days (most likely for a cloud-hosted server) to a few weeks (possible on lower-powered systems or servers with slow connections) to sync the Chia full node. During this time, Core Registry CADT can start syncing, but any writes to Core Registry CADT are likely to fail. For best results, we recommend waiting for the Chia full node to finish syncing before using Core Registry CADT. Check the status of the full node sync with `chia show -s`. *Those familiar with bittorrent and have a fast connection can speed up the full node syncing by [downloading the database](https://www.chia.net/downloads/#database-checkpoint) and copying it into place manually*