From b02d996bf253b72894cd479a8cac78c20a8a76fd Mon Sep 17 00:00:00 2001 From: scott lee Date: Mon, 13 Jan 2025 12:10:10 +0900 Subject: [PATCH] Add node monitoring setup guide --- docs/misc/operation/monitoring-setup.md | 267 ++++++++++++++++++++++++ 1 file changed, 267 insertions(+) create mode 100644 docs/misc/operation/monitoring-setup.md diff --git a/docs/misc/operation/monitoring-setup.md b/docs/misc/operation/monitoring-setup.md new file mode 100644 index 00000000000..cb01448b09b --- /dev/null +++ b/docs/misc/operation/monitoring-setup.md @@ -0,0 +1,267 @@ +# Configure Node Monitoring + +This guide explains how to set up Prometheus and Grafana to monitor your Kaia node. + +## 1\. Metric Configuration in Kaia + +Kaia provides the following flags for metric export: + +* `--metric`: Enables metric recording. This flag is typically used in conjunction with the `--prometheus` flag. +* `--prometheus`: Enables exporting recorded metrics to a Prometheus server. This flag is typically used in conjunction with the `--metric` flag. +* `--prometheusport`: Specifies the port for Prometheus metrics. Defaults to `61001`. + +To enable metrics and Prometheus exporting, set both `METRICS` and `PROMETHEUS` to `1` in your `.conf` file: + +```conf +METRICS=1 +PROMETHEUS=1 +``` + +## 2\. Setting up Prometheus + +[Prometheus](https://prometheus.io/) acts as the central system for monitoring, providing robust data mining capabilities to extract real-time data from your nodes and archive it. + +:::note[Prometheus Hardware Requirements] + +Before setting up Prometheus, ensure your system meets the following hardware requirements: + +- **Processor:** At least 2 CPUs +- **Memory:** Minimum of 4 GB RAM +- **Storage:** At least 20 GB of free disk space + +::: + +### 2.1 Installing Prometheus + +The following steps outline the manual installation process for Prometheus. Choose your operating system for specific instructions. For more information about Prometheus installation, refer to the [official Prometheus documentation](https://prometheus.io/docs/prometheus/latest/getting_started/). + +1. Download the latest Prometheus release suitable for your architecture (e.g., darwin-amd64) from the official Prometheus download page. This guide uses version 2.53.3 as an example. + +```bash +curl -LO https://github.com/prometheus/prometheus/releases/download/v2.53.3/prometheus-2.53.3.darwin-arm64.tar.gz +``` + +```bash +wget https://github.com/prometheus/prometheus/releases/download/v2.53.3/prometheus-2.53.3.linux-amd64.tar.gz +``` + +2. Extract the downloaded archive and install binaries by moving them to `/usr/local/bin/`: + +```bash +tar xvfz prometheus-2.53.3.darwin-arm64.tar.gz +mv prometheus-2.53.3.darwin-arm64/prometheus /usr/local/bin/ +mv prometheus-2.53.3.darwin-arm64/promtool /usr/local/bin/ +``` + +```bash +wget https://github.com/prometheus/prometheus/releases/download/v2.53.3/prometheus-2.53.3.linux-amd64.tar.gz +tar xvfz prometheus-2.53.3.linux-amd64.tar.gz +mv prometheus-2.53.3.linux-amd64/prometheus /usr/local/bin/ +mv prometheus-2.53.3.linux-amd64/promtool /usr/local/bin/ +``` + +3. Remove the downloaded archive and extracted directory: + +```bash +rm -rf prometheus-2.53.3.darwin-arm64.tar.gz prometheus-2.53.3.darwin-amd64 +``` + +```bash +rm -rf prometheus-2.43.0.linux-amd64.tar.gz prometheus-2.43.0.linux-amd64 +``` + +4. Add Prometheus to your `PATH` environment variable to access Prometheus from any terminal session. + +```bash +echo "export PATH=\"\$HOME/monitoring/prometheus:\$PATH\"" >> ~/.bashrc +source ~/.bashrc +# This assumes that a prometheus directory already exists in the HOME directory. If not, make a new one (mkdir -p $HOME/monitoring/prometheus). +``` + +### 2.2 Configuring Prometheus + +Prometheus needs to be configured to scrape metrics from your Kaia nodes. + +:::info\[Prometheus Configuration\] + +The `prometheus.yml` file configures Prometheus. The key sections are: + +* **`global`**: Sets global configuration parameters like `evaluation_interval` (how often Prometheus evaluates rules) and `scrape_interval` (how often Prometheus scrapes targets). 15 seconds is a reasonable starting point for both, but adjust based on your needs and block time. + +* **`scrape_configs`**: Defines the targets Prometheus monitors. The `job_name` identifies the target group. `static_configs` lists the target addresses. Replace `` with the IP address of your Kaia node and ensure the port (`61001` by default) is correctly configured. + +For more advanced configurations, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/). + +::: + +1. Open the `prometheus.yml` file located at `prometheus/prometheus.yml` in a text editor. + +2. Ensure the `scrape_configs` section includes your Kaia nodes. Below is an example configuration: + +```yaml +global: + evaluation_interval: 15s + scrape_interval: 15s + +scrape_configs: +- job_name: klaytn + static_configs: + - targets: #Replace `192.168.1.100` and `192.168.1.101` with the actual IP addresses of your Kaia nodes. + - "192.168.1.100:61001" + - "192.168.1.101:61001" + ... +``` + +3. Use `promtool` to check the configuration file for any syntax errors: + +```bash +promtool check config prometheus/prometheus.yml +``` + +4. Start Prometheus with your configuration file. + +```bash +prometheus --config.file=prometheus/prometheus.yml +``` + +### 2.3 Setting up Prometheus Using Macro Script (macOS) + +This script automates the Prometheus installation and configuration process on macOS. Adapt it for other Prometheus versions and operating systems as needed. + +```sh +rm -rf prometheus + +echo "Installing Prometheus..." +curl -LO https://github.com/prometheus/prometheus/releases/download/v2.43.0/prometheus-2.43.0.darwin-arm64.tar.gz +tar xvfz prometheus-2.43.0.darwin-arm64.tar.gz > /dev/null 2>&1 && mv prometheus-2.43.0.darwin-arm64 prometheus && rm -rf prometheus-2.43.0.darwin-arm64 && rm -rf prometheus-2.43.0.darwin-arm64.tar.gz +echo "export PATH=\"$HOMEDIR/monitoring/prometheus:\$PATH\"" >> ~/.bashrc && source ~/.bashrc + +# Generate Prometheus config file (prometheus.yml) +printf "%s\n" "global:" \ + " evaluation_interval: 15s" \ + " scrape_interval: 15s" \ + "" \ + "scrape_configs:" \ + "- job_name: klaytn" \ + " static_configs:" \ + " - targets:" > prometheus/prometheus.yml + +# Append target configurations for multiple nodes +for (( i=0; i and with the actual IP address and port (61001) for each node + printf " - \":%d\"\n" >> prometheus/prometheus.yml +done +``` + +## 3\. Setting up Grafana + +Grafana allows you to visualize the metrics collected by Prometheus through customizable dashboards. + +:::note[Grafana System Requirements] + +Before setting up Grafana, ensure your system meets the minimum hardware and software requirements from the [official Grafana documentation](https://grafana.com/docs/grafana/latest/setup-grafana/installation/). + +::: + +### 3.1 Installing Grafana + +Download and install Grafana using the appropriate method for your operating system. For example, you can install Grafana [on macOS using Hombrew](https://grafana.com/docs/grafana/latest/setup-grafana/installation/mac/) (`brew install grafana`). See the [official Grafana installation guide](https://grafana.com/docs/grafana/latest/setup-grafana/installation/) for detailed instructions. + +### 3.2 Configuring Grafana + +Set up Grafana to visualize the metrics collected by Prometheus. + +1. Start Grafana Server. + +```bash +# macOS using Homebrew +brew services start grafana +``` + +For other operating systems, refer to the [official Grafana documentation](https://grafana.com/docs/grafana/latest/setup-grafana/start-restart-grafana/). + +2. Open a web browser and navigate to `http://localhost:3000`. Log in using the default credentials (admin/admin). + +3. Add Prometheus as a Data Source. + + - Navigate to **Cofiguration** -> **Data Sources**. + - Click on **Add data source**. + - Select **Prometheus** as the type. + - Set the **URL** to `http://localhost:9090` (modify if Prometheus is on a different server). + - Click **Save & Test** to verify the connection. + +4. Add a Kaia dashboard and add a panel to visualize Kaia block number. + - [Create a new dashboard](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/create-dashboard/) or navigate to an existing one. + - Click **Edit** in the top-right corner, click **Add** in the dashboard header, and select **Visualization** in the drop-down to add a panel. + - Under **Query**: + 1. Select your Prometheus as **Data source**. + 2. Enter `klaytn_blockchain_head_blocknumber` in the **Metric** field. + 3. In **Options**, select **Custom** from the **Legend** dropdown and enter `{{instance}}` as the custom legend format. + - Click **Apply** to save the panel to your dashboard. + + +:::note[Additional Kaia Dashboards] + +For a complete pre-configured dashboard and automated provisioning setup, refer to the [klaytn-deploy repository](https://github.com/klaytn/klaytn-deploy/tree/main/grafana). This repository contains JSON files for pre-built dashboards and configuration files for provisioning data sources. + +::: + +### 3.3 Setting up Grafana Using Macro Script (macOS) + +This script automates the Grafana installation process on macOS. Adapt it for other Grafana versions and operating systems as needed. + +```sh +# Remove any existing Grafana installation +rm -rf grafana + +# Install Grafana +echo "Installing Grafana..." +curl -O https://dl.grafana.com/enterprise/release/grafana-enterprise-8.4.5.darwin-arm64.tar.gz +tar -zxvf grafana-enterprise-8.4.5.darwin-arm64.tar.gz > /dev/null 2>&1 && mv grafana-8.4.5 grafana && rm -rf grafana-enterprise-8.4.5.darwin-arm64.tar.gz +echo "export PATH=\"$HOMEDIR/monitoring/grafana/bin:\$PATH\"" >> ~/.bashrc && source ~/.bashrc + +# Generate Grafana dashboard config file +printf "%s\n" "apiVersion: 1" \ + "providers:" \ + "- name: 'klaytn'" \ + " folder: ''" \ + " options:" \ + " path: conf/provisioning/dashboards" > grafana/conf/provisioning/dashboards/klaytn-dashboard.yml + +# Generate Grafana datasource config file +printf "%s\n" "datasources:" \ + "- is_default: true " \ + " name: 'klaytn'" \ + " type: 'prometheus'" \ + " url: 'http://localhost:9090'" > grafana/conf/provisioning/datasources/klaytn.yml +``` + +```sh +# Clone the klaytn-deploy repository if not already cloned +if [ ! -d "klaytn-deploy" ]; then + echo "Cloning klaytn-deploy repository..." + git clone https://github.com/klaytn/klaytn-deploy.git +fi + +# Copy Grafana configuration files from klaytn-deploy +cp klaytn-deploy/grafana/*.json grafana/conf/provisioning/dashboards/ +``` + +## 4\. Access Services + +After installation and configuration, access Prometheus and Grafana interfaces to verify that everything is set up correctly. + +- **Prometheus Interface** + + - **URL:** `http://localhost:9090` + - **Verification:** Navigate to this URL in your browser. You should see the Prometheus web interface. Use the **Graph** tab to execute sample queries and ensure metrics are being scraped. + + +- **Grafana Interface** + + - **URL:** `http://localhost:3000` + - **Default Credentials:** + - **Username:** `admin` + - **Password:** `admin` + - **Verification:** Upon first login, you'll be prompted to change the default password. After logging in, ensure that the Prometheus data source is correctly configured and that the Kaia dashboard displays metrics. \ No newline at end of file