Skip to content

Commit

Permalink
feat: add subscriptions (#49)
Browse files Browse the repository at this point in the history 
* feat: add subscriptions

* feat: add subscription to report

* chore: fixed linting

* chore: use default query

* feat: skip subscription if chain is different

* chore: make readme lines shorter

* chore: add README for the new schema

* chore: add config.example for the new schema

* chore: fixed linting

* feat: refactor metrics, part 1

* feat: refactor metrics, part 2

* feat: refactor metrics, part 3
  • Loading branch information
@freak12techno
freak12techno authored Feb 24, 2024
1 parent 57a1f23 commit e7b3d37
Show file tree
Hide file tree
Showing 21 changed files with 730 additions and 296 deletions.
110 changes: 95 additions & 15 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,12 @@
![Latest release](https://img.shields.io/github/v/release/QuokkaStake/cosmos-transactions-bot)
[![Actions Status](https://github.com/QuokkaStake/cosmos-transactions-bot/workflows/test/badge.svg)](https://github.com/QuokkaStake/cosmos-transactions-bot/actions)

cosmos-transactions-bot is a tool that listens to transactions with a specific filter on multiple chains and reports them to a Telegram channel.
cosmos-transactions-bot is a tool that listens to transactions with a specific filter on multiple chains
and reports them to a Telegram channel.

Here's how it may look like:

![Telegram](https://raw.githubusercontent.com/QuokkaStake/cosmos-transactions-bot/master/images/telegram.png)
![Telegram](https://raw.githubusercontent.com/QuokkaStake/cosmos-transactions-bot/main/images/telegram.png)

## How can I set it up?

Expand All @@ -31,7 +32,8 @@ Then we need to create a systemd service for our app:
sudo nano /etc/systemd/system/cosmos-transactions-bot.service
```

You can use this template (change the user to whatever user you want this to be executed from. It's advised to create a separate user for that instead of running it from root):
You can use this template (change the user to whatever user you want this to be executed from. It's advised
to create a separate user for that instead of running it from root):

```
[Unit]
Expand Down Expand Up @@ -69,29 +71,100 @@ sudo journalctl -u cosmos-transactions-bot -f --output cat

## How does it work?

There are multiple nodes this app is connecting to via Websockets (see [this](https://docs.tendermint.com/master/rpc/#/Websocket/subscribe) for more details) and subscribing to the queries that are set in config. When a new transaction matching the filters is found, it's put through a deduplication filter first, to make sure we don't send the same transaction twice. Then each message in transaction is enriched (for example, if someone claims rewards, the app fetches Coingecko price and validator rewards are claimed from). Lastly, each of these transactions are sent to a reporter (currently Telegram only) to notify those who need it.
There are multiple nodes this app is connecting to via Websockets (see [this](https://docs.tendermint.com/master/rpc/#/Websocket/subscribe) for more details) and subscribing
to the queries that are set in config. When a new transaction matching the filters is found, it's put through
a deduplication filter first, to make sure we don't send the same transaction twice. Then each message
in transaction is enriched (for example, if someone claims rewards, the app fetches Coingecko price
and validator rewards are claimed from). Lastly, each of these transactions are sent to a reporter
(currently Telegram only) to notify those who need it.

## How can I configure it?

All configuration is done with a `.toml` file, which is passed to an app through a `--config` flag. See `config.example.toml` for reference.
All configuration is done with a `.toml` file, which is passed to an app through a `--config` flag.
See `config.example.toml` for reference.

### Chains, subscriptions, chain subscriptions and reporters

This app's design is quite complex to allow it to be as flexible as possible.
There are the main objects that this app has:

- reporter - something that acts as a destination point (e.g. Telegram bot) and maybe allows you as a user
to interact with it in a special way (like, setting aliases etc.)
- chain - info about chain itself, its denoms, queries (see below), nodes used to receive data from, etc.
- subscription - info about which set of chains and their events to send to which reporter,
has many chain subscriptions
- chain subscription - info about which chain to receive data from, filters on which events to match
(see below) and how to process errors/unparsed/unsupported messages, if any.

Each chain has many chain subscriptions, each subscription has one reporter, each chain subscription
has one chain and many filters.

Generally speaking, the workflow of the app looks something like this:

![Schema](https://raw.githubusercontent.com/QuokkaStake/cosmos-transactions-bot/main/images/schema.png)

This allows to build very flexible setups. Here's the example of the easy and the more difficult setup.

1) "I want to receive all transactions sent from my wallet on chain A, B and C to my Telegram channel"

You can do it the following way:
- have 1 reporter, a Telegram channel
- have 3 chains, A, B and C, and their configs
- have 1 subscription, with Telegram reporter and 3 chain subscriptions inside (one for chain A, B and C
with 1 filter each matching transfers from wallets on these chains)

2) "I want to receive all transactions sent from my wallet on chains A, B and C to one Telegram chat,
all transactions that are votes on chains A and B to another Telegram chat, and all transactions that are delegations
with amount more than 10M $TOKEN on chain C to another Telegram chat"

That's also manageable. You can do the following:
- reporter 1, "first", a bot that sends messages to Telegram channel 1
- reporter 2, "second", a bot that sends messages to Telegram channel 2
- reporter 3, "third", a bot that sends messages to Telegram channel 3
- chain A and its config
- chain B and its config
- chain C and its config
- subscription 1, let's call it "my-wallet-sends", with reporter "first" and the following chain subscriptions
- - chain subscription 1, chain A, 1 filter matching transfers from my wallet on chain A
- - chain subscription 2, chain B, 1 filter matching transfers from my wallet on chain B
- - chain subscription 3, chain C, 1 filter matching transfers from my wallet on chain C
- subscription 2, let's call it "all-wallet-votes", with reporter "second" and the following chain subscriptions
- - chain subscription 1, chain A, 1 filter matching any vote on chain A
- - chain subscription 2, chain B, 1 filter matching any vote on chain B
- subscription 3, let's call it "whale-votes", with reporter "third" and the following chain subscription
- - chain subscription 1, chain C, 1 filter matching any delegations with amount more than 10M $TOKEN on chain C

See config.example.toml for real-life examples.

### Queries and filters

This is quite complex and deserves a special explanation.
This is another quite complex topic and deserves a special explanation.

When a node starts, it connects to a Websocket of the fullnode and subscribes to queries (`queries` in `.toml` config).
If there's a transaction that does not match these filters, a fullnode won't emit the event for it
and this transaction won't reach the app.

When a node starts, it connects to a Websocket of the fullnode and subscribes to queries (`queries` in `.toml` config). If there's a transaction that does not match these filters, a fullnode won't emit the event for it and this transaction won't reach the app.
If using filters (`filters` in `.toml` config), when a transaction is received, all messages in the transaction
are checked whether they match these filters, and can be filtered out (and the transaction itself would be filtered out
if there are 0 non-filtered messages left).

If using filters (`filters` in `.toml` config), when a transaction is received, all messages in the transaction are checked whether they match these filters, and can be filtered out (and the transaction itself would be filtered out if there are 0 non filtered messages left).
Using filters can be useful is you have transactions with multiple messages, where you only need to know about one
(for example, someone claiming rewards from your validator and other ones, when you need to know only about claiming
from your validator).

Using filters can be useful is you have transactions with multiple messages, where you only need to know about one (for example, someone claiming rewards from your validator and other ones, when you need to know only about claiming from your validator).
Keep in mind that queries is set on the app level, while filters are set on a chain subscription level,
so you can have some generic query on a chain, and more granular filter on each of your chain subscriptions.

Filters should follow the same pattern as queries, but they can only match the following pattern (so no AND/OR support):
- `xxx = yyy` (which would filter the transaction if key doesn't match value)
- `xxx! = yyy` (which would filter the transaction if key does match value)

Please note that the message would not be filtered out if it matches at least one filter. Example: you have a message that has `xxx = yyy` as events, and if using `xxx != yyy` and `xxx != zzz` as filters, it won't get filtered out (as it would not match the first filter but would match the second one).
Please note that the message would not be filtered out if it matches at least one filter.
Example: you have a message that has `xxx = yyy` as events, and if using `xxx != yyy` and `xxx != zzz` as filters,
it won't get filtered out (as it would not match the first filter but would match the second one).

You can always use `tx.height > 0`, which will send you the information on all transactions in chain, or check out something we have:
You can always use `tx.height > 0`, which will send you the information on all transactions in chain,
or check out something we have:


```
Expand Down Expand Up @@ -135,7 +208,10 @@ filters = [

See [the documentation](https://docs.tendermint.com/master/rpc/#/Websocket/subscribe) for more information on queries.

One important thing to keep in mind: by default, Tendermint RPC now only allows 5 connections per client, so if you have more than 5 filters specified, this will fail when subscribing to 6th one. If you own the node you are subscribing to, o fix this, change this parameter to something that suits your needs in `<fullnode folder>/config/config.toml`:
One important thing to keep in mind: by default, Tendermint RPC now only allows 5 connections per client,
so if you have more than 5 filters specified, this will fail when subscribing to 6th one.
If you own the node you are subscribing to, o fix this, change this parameter to something that suits your needs
in `<fullnode folder>/config/config.toml`:

```
max_subscriptions_per_client = 5
Expand All @@ -144,12 +220,16 @@ max_subscriptions_per_client = 5
## Notifications channels

Go to [@BotFather](https://t.me/BotFather) in Telegram and create a bot. After that, there are two options:
- you want to send messages to a user. This user should write a message to [@getmyid_bot](https://t.me/getmyid_bot), then copy the `Your user ID` number. Also keep in mind that the bot won't be able to send messages unless you contact it first, so write a message to a bot before proceeding.
- you want to send messages to a channel. Write something to a channel, then forward it to [@getmyid_bot](https://t.me/getmyid_bot) and copy the `Forwarded from chat` number. Then add the bot as an admin.
- you want to send messages to a user. This user should write a message to [@getmyid_bot](https://t.me/getmyid_bot),
then copy the `Your user ID` number. Also keep in mind that the bot won't be able to send messages
unless you contact it first, so write a message to a bot before proceeding.
- you want to send messages to a channel. Write something to a channel, then forward it to [@getmyid_bot](https://t.me/getmyid_bot)
and copy the `Forwarded from chat` number. Then add the bot as an admin.

Then run a program with Telegram config (see `config.example.toml` as example).

You would likely want to also put only the IDs of trusted people to admins list in Telegram config, so the bot won't react to anyone writing messages to it except these users.
You would likely want to also put only the IDs of trusted people to admins list in Telegram config, so the bot
won't react to anyone writing messages to it except these users.

Additionally, for the ease of using commands, you can put the following list as bot commands in @BotFather settings:

Expand Down
129 changes: 95 additions & 34 deletions config.example.toml
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
# Path to where aliases in .toml will be stored. If omitted, no aliases setting/displaying would work.
aliases = "/home/monitoring/config/cosmos-transactions-bot-aliases.toml"
# Timezone in which time (like undelegation finish time) will be displayed. Defaults to "Etc/GMT", so UTC+0
# Path to where aliases in .toml will be stored.
# If omitted, no aliases setting/displaying would work.
aliases = "cosmos-transactions-bot-aliases.toml"
# Timezone in which time (like undelegation finish time) will be displayed.
# Defaults to "Etc/GMT", so UTC+0
timezone = "Europe/Moscow"

# Logging configuration
[log]
# Log level. Set to "debug" or "trace" to make it more verbose, or to "warn"/"error" to make it less verbose.
# Log level. Set to "debug" or "trace" to make it more verbose, or to "warn"/"error"
# to make it less verbose.
# Defaults to "info"
level = "info"
# If true, all logs would be displayed in JSON. Useful if you are using centralized logging
Expand All @@ -14,47 +17,42 @@ json = false

# Reporters configuration.
[[reporters]]
# Reporter name. Should be unique.
name = "telegram-1"
# Reporter type. Currently, the only supported type is "telegram", which is the default.
type = "telegram"
# Telegram config configuration. Required if the type is "telegram".
# See README.md for more details.
# Has 3 params:
# - token - bot token
# - chat - a chat/channel to post messages to
# - admins - a whitelist of user IDs allowed to send commands to the bot, optional but recommended.
telegram-config = { token = "xxx:yyy", chat = 12345, admins = [67890] }

# Per-chain configuration. There can be multiple chains.
[[chains]]
# Chain codename, required.
# There can be multiple reporters.
[[reporters]]
name = "telegram-2"
type = "telegram"
telegram-config = { token = "zzz:aaa", chat = 98765, admins = [43210] }

# Subscriptions config. See README.md on how this schema works.
[[subscriptions]]
# Reporter name to send events matching this subscription to.
# Should be one of the names of the reporters declared above, or the app won't start
# with the config validation error
reporter = "telegram-1"
# Subscription name, for metrics. Should be unique.
name = "subscription-1"

# Chain subscriptions for this subscription.
[[subscriptions.chains]]
# Chain name. Should be one of the names declared below in chains section,
# or the app won't start with the config validation error.
name = "cosmos"
# Chain pretty name, optional. If provided, would be used in reports, if not,
# codename would be used.
pretty-name = "Cosmos Hub"
# Tendermint RPC nodes to subscribe to. At least one is required, better to have multiple ones
# as a failover.
tendermint-nodes = [
"https://rpc.cosmos.quokkastake.io:443",
]
# API nodes to get blockchain data (validators, proposals etc.) from.
api-nodes = [
"https://api.cosmos.quokkastake.io",
]
# Queries, see README.md for details.
queries = [
"tx.height > 0"
]
# Filter, see README.md for details.
filters = [
"message.action = '/cosmos.gov.v1beta1.MsgVote'",
]
# Denoms list.
denoms = [
# Each denom inside must have "denom" and "display-denom" fields and additionaly
# denom-coefficient (set to 1000000 by default) and coingecko-currency.
# Example: if there's a transfer transaction for 10,000,000 uatom,
# and the coingecko price for $ATOM is 10$ and if all fields are set,
# instead of displaying amount as `10000000.000000uatom` it would be displayed
# as `10.000000atom ($100.00)`.
# If coingecko-currency is omitted, no price would be displayed.
{ denom = "uatom", display-denom = "atom", denom-coefficient = 1000000, coingecko-currency = "cosmos" }
]
# If set to true and there is a message not supported by this app,
# it would post a message about that, otherwise it would ignore such a message.
# Defaults to false.
Expand Down Expand Up @@ -84,6 +82,58 @@ filter-internal-messages = true
# - `Error: RPC error -32000 - Server error: subscription was cancelled (reason: Tendermint exited)`
# If this is set to true (default), such messages would be displayed, if not, they will be skipped.
log-node-errors = true

# There can be multiple chain subscriptions per subscription.
[[subscriptions.chains]]
name = "sentinel"
filters = ["message.action = '/cosmos.staking.v1beta1.MsgDelegate'"]

# There can also be multiple subscriptions. This one, for example,
# sends everything to a different reporter.
[[subscriptions]]
name = "subscription-2"
reporter = "telegram-2"
[[subscriptions.chains]]
name = "cosmos"
filters = ["message.action = '/cosmos.staking.v1beta1.MsgUndelegate'"]
[[subscriptions.chains]]
name = "sentinel"
filters = ["message.action = '/cosmos.staking.v1beta1.MsgBeginRedelegate'"]



# Per-chain configuration.
[[chains]]
# Chain codename, required.
name = "cosmos"
# Chain pretty name, optional. If provided, would be used in reports, if not,
# codename would be used.
pretty-name = "Cosmos Hub"
# Tendermint RPC nodes to subscribe to. At least one is required, better to have multiple ones
# as a failover.
tendermint-nodes = [
"https://rpc.cosmos.quokkastake.io:443",
]
# API nodes to get blockchain data (validators, proposals etc.) from.
api-nodes = [
"https://api.cosmos.quokkastake.io",
]
# Queries, see README.md for details.
# Defaults to ["tx.height > 0"], so basically all transactions on chain.
queries = [
"tx.height > 0"
]
# Denoms list.
denoms = [
# Each denom inside must have "denom" and "display-denom" fields and additionaly
# denom-coefficient (set to 1000000 by default) and coingecko-currency.
# Example: if there's a transfer transaction for 10,000,000 uatom,
# and the coingecko price for $ATOM is 10$ and if all fields are set,
# instead of displaying amount as `10000000.000000uatom` it would be displayed
# as `10.000000atom ($100.00)`.
# If coingecko-currency is omitted, no price would be displayed.
{ denom = "uatom", display-denom = "atom", denom-coefficient = 1000000, coingecko-currency = "cosmos" }
]
# Explorer configuration.
# Priorities:
# 1) ping.pub
Expand Down Expand Up @@ -111,3 +161,14 @@ block-link-pattern = "https://mintscan.io/cosmos/blocks/%s"
# A pattern for validator links for the explorer.
validator-link-pattern = "https://mintscan.io/cosmos/validators/%s"


# There can be multiple chains.
[[chains]]
name = "sentinel"
pretty-name = "Sentinel"
tendermint-nodes = ["https://rpc.sentinel.quokkastake.io:443"]
api-nodes = ["https://api.sentinel.quokkastake.io"]
denoms = [
{ denom = "udvpn", display-denom = "dvpn", coingecko-currency = "sentinel" }
]
mintscan-prefix = "sentinel"
Binary file added images/schema.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
12 changes: 6 additions & 6 deletions pkg/alias_manager/alias_manager.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"os"

"main/pkg/config"
"main/pkg/config/types"
configTypes "main/pkg/config/types"

"github.com/BurntSushi/toml"
"github.com/rs/zerolog"
Expand All @@ -14,15 +14,15 @@ type Aliases *map[string]string
type TomlAliases map[string]Aliases

type ChainAliases struct {
Chain *types.Chain
Chain *configTypes.Chain
Aliases Aliases
}
type AllChainAliases map[string]*ChainAliases

type AliasManager struct {
Logger zerolog.Logger
Path string
Chains config.Chains
Chains configTypes.Chains
Aliases AllChainAliases
}

Expand All @@ -36,15 +36,15 @@ func (a AllChainAliases) ToTomlAliases() TomlAliases {
}

type ChainAliasesLinks struct {
Chain *types.Chain
Links map[string]types.Link
Chain *configTypes.Chain
Links map[string]configTypes.Link
}

func (a AllChainAliases) ToAliasesLinks() []ChainAliasesLinks {
aliasesLinks := make([]ChainAliasesLinks, 0)

for _, chainAliases := range a {
links := make(map[string]types.Link)
links := make(map[string]configTypes.Link)

if chainAliases.Aliases == nil {
continue
Expand Down
Loading

0 comments on commit e7b3d37

Please sign in to comment.