feat: Migrate to NEAR Lake #326

morgsmccauley · 2022-12-09T03:46:11Z

This PR migrates the core indexing functionality to near-lake-framework by merging in the khorolets/near-indexer-for-explorer fork. Because the two branches have diverged, leading to many conflicts, it was easier to merge the fork in to a local branch, rather than creating a pull request from the fork directly.

We are in the process of phasing out the live nearcore based indexer-for-explorers, replacing them with this lake fork, which makes this a good time to merge this back in. I've pushed the current master to master-nearcore so that the previous state is still available to build in case we need it, can someone with admin access please make this branch protected?

The code is mostly identical to khorolets/near-indexer-for-explorer:master-lake, I've commented inline where things differ.

…ework

* refactor: Upgrade to NEAR Lake Framework 0.5.1 * remove .buildkite, add GH Actions workflows, drop redis requirement, fix clippy warnings * change the version to be relative to the original indexer-for-explorer

* Update test.yml * update rust version * update ubuntu version in the workflows * add libpq-dev to cross Dockerfile

* fix: Avoid recreating access key on transfer to implicit account * refactor: Remove `AccountDeletion` logic as its also covered by `AccessKeyDeletion` * chore: Update `CHANGELOG.md` * refactor: use `near_lake_framework` types rather than `near_primitives`

* build: Create Cargo workspace and empty `indexer` crate * build: Migrate crate root to workspaces

refactor: Extract database logic to library crate

…ng guide (#11)

… JSON RPC (#10) * feat: Add empty `circulating_supply` crate * feat: Add missing db adapaters/models to `database` lib * feat: Add core circulating supply files * refactor: Flatten/merge `circulating_supply` files * fix: Add missing dependencies and fix broken imports Excluding `near-indexer` and `near-client` as these will be replaced * refactor: Make in-use circulating_supply functions public * refactor: Replace `near_indexer::near_primitives` with `near_lake_framework::near_indexer_primitives` * refactor: Instantiate pg pool in `main` * refactor: Replace `ViewClient` calls with `JsonRpcClient` * refactor: `await` circulating_supply in `main` * chore: Rename `tracing` target `aggregated` -> `circulating_supply` * refactor: Extract json rpc url to environment variable * feat: Subscribe to tracing events via `RUST_LOG` env * feat: Improve logging * refactor: Replace `near-indexer-framework` with `near-indexer-primtives` * fix: Base64 decode lockup state before deserializing * feat: Print `anyhow` errors with cause * chore: Add note to ensure API use is updated when upgrading near-core * chore: Use kebab case for workspace crate

* chore: Update `indexer` to `10.2` * chore: Update `indexer` to `10.3`

* chore: Move `migrations` to same dir as `Diesel.toml` * chore: Remove unnecessary `.gitkeep` * feat: Add missing Diesel migrations * chore: Update `README.md` for running migrations

* feat: Enable JSON logging via environment variable * chore: Update CHANGELOG & Cargo version

* refactor: Move indexer stream to separate thread * feat: Standup HTTP server * fix: Ensure lake `ReceiverStream` finishes when process exits * feat: Serve block metrics via http * refactor: Rename `HTTP_PORT` -> `PORT` * chore: Update `CHANGELOG.md` and Cargo version * refactor: Rename block metrics for clarity

* refactor: Parse `port` via CLI and env rather than just env * refactor: Return `anyhow::Result` from `metrics::init_server`

.buildkite/pipeline.yml

.github/workflows/release.yml

Co-authored-by: Bohdan Khorolets <[email protected]>

khorolets

I am approving this PR to avoid blocking it, though I'd wait for @telezhnaya inputs before merging.

Also, I want to draw everybody's attention to questions:

Should we drop the Buildkite jobs since we can run checks via GitHub Actions and we're not going to build binaries with Buildkite anymore? (@ecp88 , @frol )
Is FOSSA still a thing and we should add its check to GH Actions?
( @ecp88, @pkudinov )
Is it worth switching to [tokio::main] instead of [actix::main]? (@telezhnaya , @frol )

frol · 2022-12-12T16:58:49Z

Should we drop the Buildkite jobs since we can run checks via GitHub Actions and we're not going to build binaries with Buildkite anymore?

I don't know about the plans regarding Buildkite, but I would personally prefer Github Actions. Make your own judgement here.

Is it worth switching to [tokio::main] instead of [actix::main]?

I would say it is a low-priority, and should only be done after the upgrade to near-lake-framework 0.6+ (after nearcore dependencies are upgraded to 0.15+ versions, which dropped actix as their leaf dependency), also, you will need to replace actix-diesel with tokio-diesel. Yet, after all of that the only observable benefits I can see are: (1) faster cold compilation, so expect some speed up on CI; (2) lower maintenance burden as actix used to cause some complications and we have to use a forked version fo actix-diesel now.

telezhnaya

Amazing work, thank you @morgsmccauley !

By the way, you've already migrated all the FT/NFT logic, as I see. We discussed that we need to keep the old indexer (node based) to collect FT/NFT stuff, but you've done everything we need, so we can kill the old instances, no blockers here from my side.
cc @khorolets

database/src/adapters/aggregated/circulating_supply.rs

database/src/models/serializers.rs

database/src/retriable.rs

indexer/src/configs.rs

indexer/src/main.rs

telezhnaya · 2022-12-13T12:53:29Z

indexer/src/main.rs

+    let config: near_lake_framework::LakeConfig = opts.to_lake_config().await;
+    let (_, stream) = near_lake_framework::streamer(config);
+
+    tokio::spawn(async move {


Am I right that we use actix only because of diesel?
Anyway, I'm with @khorolets , let's use [tokio::main]

indexer/src/main.rs

morgsmccauley · 2022-12-13T21:46:06Z

Thanks heaps @telezhnaya @khorolets !!

I've addressed all comments which I could, these have been resolved.

To make things easier, I've summarised the unresolved discussions below. From the comments that @khorolets left I think the majority can be addressed in future PRs, feel free to resolve these if you agree. There are still two discussions which require decisions.

Discussion	Outcome
Can we remove buildkite	Needs decision
Removing 'explorer' from names	Needs decision
Can we separate storing transactions/receipts	Separate PR
Migrate from actix::main -> tokio::main and tokio-diesel	Separate PR
Creating a single PG pool	Separate PR
Clearer parameters for final_block_height	Separate PR

indexer/src/metrics.rs

khorolets · 2022-12-14T09:33:43Z

@morgsmccauley @telezhnaya @ecp88

Discussion	Outcome
Can we remove buildkite	Drop buildkite once we shut down nodes, for now point it to `master-nearcore` branch
Removing 'explorer' from names	We keep the Explorer mention in names
Can we separate storing transactions/receipts	Separate issue
Migrate from actix::main -> tokio::main and tokio-diesel	Separate issue
Creating a single PG pool	Separate issue
Clearer parameters for final_block_height	Separate issue

morgsmccauley · 2022-12-14T18:51:29Z

Thanks for creating all the relevant issues @khorolets 🙏🏽

khorolets and others added 27 commits February 20, 2022 13:14

refactor: Switch from runing NEAR Indexer Framework to NEAR Lake Fram…

e1382b7

…ework

start streamer with actix instead of tokio (diesel-actix) is used

3f42ccc

fix find parent tx for receipt logic

e113dd2

refactor: upgraded to near-lake-framework 0.5.0 (#1)

55f17c6

refactor: Upgrade to NEAR Lake Framework 0.5.1 (#2)

0e89871

* refactor: Upgrade to NEAR Lake Framework 0.5.1 * remove .buildkite, add GH Actions workflows, drop redis requirement, fix clippy warnings * change the version to be relative to the original indexer-for-explorer

chore: Update test.yml (#3)

52d5046

* Update test.yml * update rust version * update ubuntu version in the workflows * add libpq-dev to cross Dockerfile

chore: Upgrade near-lake-framework to 0.5.2 (#4)

118e55e

build: Migrate to Cargo Workspaces (#6)

2473ddf

* build: Create Cargo workspace and empty `indexer` crate * build: Migrate crate root to workspaces

feat: Create empty database library crate

a2a1b0c

refactor: Extract db logic to database lib

0d97e3b

refactor: Change database logging target to explorer_database

fa2f47f

refactor: Rename db_adapters -> adapters

a8e27a9

refactor: Make current in-use DB funcions public

a4e3eea

refactor: Extract cache logic to database lib

73dfd6a

refactor: Replace near-lake-framework with near-indexer-primitives

d1cde8b

Merge pull request #9 from khorolets/refactor/extract-db-to-lib

59da9fa

refactor: Extract database logic to library crate

docs: Added a warning to the skipping blocks section in troubleshooti…

e9f5d9e

…ng guide (#11)

chore: Add missing Changelog/Cargo version entries (#16)

47d0210

* chore: Update `indexer` to `10.2` * chore: Update `indexer` to `10.3`

feat: Add missing diesel migrations (#12)

8db4780

* chore: Move `migrations` to same dir as `Diesel.toml` * chore: Remove unnecessary `.gitkeep` * feat: Add missing Diesel migrations * chore: Update `README.md` for running migrations

refactor: Use default AWS SDK credentials provider (#14)

faa882a

chore: Update Cargo.toml version to match previous code changes (#17)

34a6112

feat: Enable JSON logging via environment variable (#13)

720be80

* feat: Enable JSON logging via environment variable * chore: Update CHANGELOG & Cargo version

DPLT-632: Add Dockerfile files for circulating-supply and indexer (#18)

e454ea4

refactor: Metrics server config/errors (#20)

7161b77

* refactor: Parse `port` via CLI and env rather than just env * refactor: Return `anyhow::Result` from `metrics::init_server`

morgsmccauley commented Dec 9, 2022

View reviewed changes

.buildkite/pipeline.yml Show resolved Hide resolved

morgsmccauley commented Dec 9, 2022

View reviewed changes

.github/workflows/release.yml Outdated Show resolved Hide resolved

morgsmccauley force-pushed the feat/lake-migration branch from ecfae9d to 9cb7de8 Compare December 9, 2022 04:02

morgsmccauley force-pushed the feat/lake-migration branch from 5e59cc3 to 1112e68 Compare December 11, 2022 21:20

morgsmccauley and others added 4 commits December 12, 2022 16:40

chore: Remove Cross

826e84a

chore: Point to docs.near.org instead of near-indexers.io

4f68f87

Co-authored-by: Bohdan Khorolets <[email protected]>

chore: Add link to "store genesis" issue

9361c23

chore: Add missing chain-id in example command

2713da1

morgsmccauley force-pushed the feat/lake-migration branch from f45108a to 6b7edda Compare December 12, 2022 03:40

khorolets approved these changes Dec 12, 2022

View reviewed changes

morgsmccauley added 4 commits December 13, 2022 08:09

chore: Add breaking changes to CHANGELOG.md

800a64e

chore: Rephrase paragraph about non-strict-mode

9433d06

chore: Add subheadings for running indexer

758c271

docs: Add notes about docker deployments in Readme

5e79896

morgsmccauley force-pushed the feat/lake-migration branch from 6b7edda to 5e79896 Compare December 12, 2022 19:15

morgsmccauley mentioned this pull request Dec 12, 2022

chore: Upgrade nearcore to 1.30.0 #329

Merged

telezhnaya requested changes Dec 13, 2022

View reviewed changes

morgsmccauley added 4 commits December 14, 2022 07:36

refactor: Rename retriable.rs -> retryable.rs

4581db2

refactor: List duplicate expr out of match clause

58e7b48

refactor: Swap indexing/metrics threads

37825da

feat: Handle errors thrown during indexing

4942462

refactor: Drop -lake prefix in metrics

4519bb4

morgsmccauley commented Dec 14, 2022

View reviewed changes

indexer/src/metrics.rs Show resolved Hide resolved

telezhnaya approved these changes Dec 14, 2022

View reviewed changes

This was referenced Dec 14, 2022

Optimization of start options #331

Open

Update Buildkite job to build release binaries on master-nearcore branch for Indexer Framework-based version of indexer #332

Closed

morgsmccauley merged commit 5153374 into master Dec 14, 2022

morgsmccauley deleted the feat/lake-migration branch December 14, 2022 18:43

khorolets mentioned this pull request Mar 8, 2023

fix: Action's args serialization to base64 #365

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Migrate to NEAR Lake #326

feat: Migrate to NEAR Lake #326

morgsmccauley commented Dec 9, 2022 •

edited

Loading

khorolets left a comment

frol commented Dec 12, 2022

telezhnaya left a comment

telezhnaya Dec 13, 2022

morgsmccauley commented Dec 13, 2022 •

edited

Loading

khorolets commented Dec 14, 2022

morgsmccauley commented Dec 14, 2022

feat: Migrate to NEAR Lake #326

feat: Migrate to NEAR Lake #326

Conversation

morgsmccauley commented Dec 9, 2022 • edited Loading

khorolets left a comment

Choose a reason for hiding this comment

frol commented Dec 12, 2022

telezhnaya left a comment

Choose a reason for hiding this comment

telezhnaya Dec 13, 2022

Choose a reason for hiding this comment

morgsmccauley commented Dec 13, 2022 • edited Loading

khorolets commented Dec 14, 2022

morgsmccauley commented Dec 14, 2022

morgsmccauley commented Dec 9, 2022 •

edited

Loading

morgsmccauley commented Dec 13, 2022 •

edited

Loading