Rated Indexer

Index logs and metrics from various monitoring solutions for ingestion to Rated slaOS.

Introduction

The rated-log-indexer is a powerful tool designed to collect and process logs and metrics from various monitoring solutions and ingest them into Rated slaOS. This indexer supports multiple data sources, enabling centralized monitoring data collection for efficient analysis and visualization.

Prerequisites

Before you begin, ensure you have:

• A Rated slaOS account (sign up at app.rated.co if needed)
• Ingestion URL, ID, and key from your Rated slaOS general settings
• Docker installed on your system (for container deployment)

Documentation & Resources

📚 Documentation Comprehensive guides and API references Visit Docs →

📝 Examples Sample configurations and use cases View Examples →

🐛 Issues Report bugs and request features Open Issue →

Supported Integrations

CloudWatch Documentation → Templates →

Datadog Documentation → Templates →

Prometheus Documentation → Templates →

Deployment Options

Docker Deployment (Recommended)

1. Pull the latest image:

docker pull ratedlabs/rated-log-indexer:latest

2. Create your configuration file following the examples in the templates directory. The configuration is validated on startup and the indexer will exit with an error if the config is invalid.

3. Run the container:

docker run \
  --name rated-indexer \
  --volume "$(pwd)"/config/rated-config.yaml:/indexer/config/rated-config.yaml \
  --restart unless-stopped \
  ratedlabs/rated-log-indexer

The indexer includes built-in retry mechanisms and is designed for minimal maintenance. However, we recommend:

• Using --restart unless-stopped for automatic container restarts
• Setting up basic monitoring for the container's health
• Implementing log collection for troubleshooting

Local Development Setup

1. Clone the repository:

git clone https://github.com/rated-network/rated-log-indexer.git
cd rated-log-indexer

3. Configure your environment using the examples in the templates directory.

4. Run the indexer:

Using Make:

make run

• make to get an overview of supported Make commands.
• make run to start the indexer using your rated-config.yaml configuration.
• make test to run the automated tests. This will automatically create a virtualenv.

Alternatively, using Python directly:

python -m bytewax.run src.main:main

Note: When running locally, you may need PostgreSQL and/or Redis available for the offset tracker functionality. Check the templates directory for configuration examples.

Networking Requirements

Self-Hosted Deployment

The rated-log-indexer requires outbound connectivity to:

• Rated slaOS Ingestion Endpoint and API: For sending processed metrics and logs
  • Endpoint: https://api.rated.co
  • Port: 443 (HTTPS)

No inbound connections are required for normal operation. Note: this may change in the future.

Firewall Configuration

Ensure your firewall allows outbound HTTPS (TCP/443) connections to:

*.rated.co

If you're running behind a corporate proxy or have strict firewall policies, you may need to explicitly whitelist these domains.

Configuration

The rated-config.yaml file located in the config directory is structured into four main sections:

1. inputs: Define your data sources:

   • CloudWatch
   • Datadog
   • Prometheus

2. output: Configure the Rated slaOS API connection

   • Ingestion endpoint
   • API credentials
   • Batch settings

3. offset: Configure data point tracking

   • Storage location
   • Update frequency

4. secrets: Set up secrets management (recommended for production)

   • Environment variables
   • Secret files
   • Vault integration

Refer to the templates directory for detailed configuration examples.

Troubleshooting

If you encounter connectivity issues:

• Verify network connectivity:

# Check API health
curl -v https://api.rated.co/v1/health
# Should return: {"status":"healthy"}

• Check container logs:

docker logs rated-indexer

Contributing

We welcome contributions! Please see our Contributing Guide for details.

License

This project is licensed under the Rated Labs Ltd. Prosperity Public License.