🧮 ABACUS - Rust Backend

🚧 WORK in PROGRESS 🚧

This server is written in Rust (using Warp), exposes GraphQL API (via Juniper) and works with ArangoDB database behind the scenes.

• Rust: https://www.rust-lang.org/learn
• Juniper: https://graphql-rust.github.io/juniper/current/
• ArangoDB:
  • https://www.arangodb.com/docs/stable/ (multi-model DB)
  • https://www.arangodb.com/docs/stable/data-modeling-documents-schema-validation.html
  • JSON Schema validator: https://www.jsonschemavalidator.net/

Install and run

🚧 WORK in PROGRESS 🚧

We use Telepresence for the local development:

Note: there is currently no K8S cluster for development. Production only. 💸

telepresence connect
telepresence list
telepresence intercept abacus-deployment --port=5000:80 --env-file=./src/abacus/.env

You can now start the service locally, rest of the application will run in the remote Kubernetes cluster:

(cd src/abacus && cargo run -- --arangodb-url=http://arangodb-single-server.default.svc.cluster.local:8529)

The server will be accessible on: http://abacus.mrtnzlml.com:32123/graphql (or http://0.0.0.0:5000/graphql). It's recommended to use Insomnia to send request to the GraphQL API: https://insomnia.rest/graphql/

The database will be available on: http://arangodb-single-server.default.svc.cluster.local:8529 (when connected via telepresence connect). User abacus, no password.

telepresence leave abacus-deployment
telepresence quit

Deploying a new Abacus version:

kubectl rollout restart deployment abacus-deployment
kubectl rollout status deployment abacus-deployment

Alternatively, you can run the application locally in Docker (instead of cargo run):

(cd src/abacus && docker build --progress=plain . --tag abacus)
docker run -p 5000:5000 abacus --arangodb-url=http://arangodb-single-server.default.svc.cluster.local:8529

Testing

(cd src/abacus && cargo clippy)
(cd src/abacus && cargo test)

There are some extra tests which are slow or require extra infrastructure (network access, ArangoDB). There tests are ignored by default but can be executed manually:

(cd src/abacus && cargo test -- --ignored)

Note: ignored tests are not being run on CI (at least not yet)!

Database migrations

Database migrations are currently being run automatically during the server start. It's not and ideal or final solution, but it's "good enough" for now.

ArangoDB

• https://www.arangodb.com/
• https://hub.docker.com/r/arangodb/arangodb/

Why ArangoDB? At the time of writing, it was essentially the most promising multi-model open-source DB (with graph support) out there. Source: https://db-engines.com/en/ranking/graph+dbms

• _id - document handle (uniquely identifies a document in the database)
• _key - document's primary key (uniquely identifies a document in the collection it is stored in)
• _rev - document revision

Resources:

• Best Practices for AQL Graph Queries

Arangodump & Arangorestore

Database backup with data (empty password):

arangodump \
    --server.password="" \
    --server.database=abacus \
    --output-directory="src/abacus/__dump" \
    --include-system-collections=true \
    --overwrite=true \
    --compress-output=false \
    --dump-data=true

Note: --include-system-collections=true + --dump-data=true is important because we are using named graphs and they are stored in a _graphs system collection. Eventually, we should probably split the dump into system exports with data and structural exports of the application. We are also not exporting _system DB at all.

Database restore:

arangorestore \
    --input-directory="src/abacus/__dump" \
    --server.database=abacus

Arangosh access:

arangosh \
    --server.password="" \
    --server.database=abacus

For example, to delete analyzers:

var analyzers = require('@arangodb/analyzers');
analyzers.remove('bigram');

GraphQL API design guide

🚧 WORK in PROGRESS 🚧

TKTK (anyhow)

One of the main parts of this server is GraphQL API. Generally speaking, we are not trying to have any restrictions when it comes to designing this API, and we encourage trying new things. However, there are some rules which should be always followed to create kind of contract/agreement between server and GraphQL client.

1. Every top-level GraphQL resolver should return Result<T, CustomModelError> where T can be a union of success/error payload or just a simple value. This allows us to return an application value (be it queryable error or the actual value) OR a critical server error when it happens (missing permissions, DB failure, …). See: Juniper error handling

TODOs

🚧 001 - DB strings and source-code translations
🚧 002 - DB schema validations (JSON schema)
🚧 003 - server monitoring and error reporting (?)
🚧 004 - integration tests for ArangoDB queries - auth package (https://youtu.be/muvU1DYrY0w, https://github.com/dropbox/dbx_build_tools)
✅ 005 - implement https://github.com/woltapp/blurhash
🚧 006 - use Bazel https://bazelbuild.github.io/rules_rust/
✅ 007 - DB migrations
🚧 008 - queries whitelisting (persistent queries)
🚧 009 - explore WASM on server instead of Docker (https://github.com/deislabs/krustlet)
🚧 010 - ArangoDB database backups and restores (k8s)
🚧 011 - development k8s cluster + Telepresence