refactor: sqlalchemy

[abulte]

This migrates the ORM from dataset to sqlalchemy.

Main advantages are:

• migrations support (Convention de nommage #8)
• better typing handling
• schema is now constrained, and thus more manageable
• foreign keys support in ORM

The strategy is to create an "initial migration" that provides iso-features after being applied, without the need for data migrations. I tried to create a SQLAlchemy schema that would generate a quasi-empty initial migration, but it became a game of whack-a-mole I couldn't win.

The initial migration should be applied once everywhere and then removed, it's meant to be ephemeral and does not handle the bootstrapping of an empty database (see comment in migration file).

Apart from unit tests, the migration has been validated in the following scenarios:

• load script works on an unmigrated local database (not needed in practice)
• migration works on a demo database restored from our deployment, and subsequent load script works
• python cli.py init-db works to bootstrap an empty database, and subsequent load script works

I've compared the output of metrics computation in every scenario (this should validate quite a few things in the process chain).

On local, you should remove the dataset dependency, reinstall requirements.txt and either:

• start from scratch (probably the best option, and made "easier" by the fact that the database name has changed)
• or run ALEMBIC_ENV=(demo|prod) alembic upgrade head to migrate an existing database

⚠️ The local databases are now dashboard_backend for demo and dashboard_backend_prod for prod. This allows easy restore from our deployment. Delete your docker container to have the new databases created (you will lose your existing databases) or migrate them yourself (the Postgres version has changed too, so this will be difficult). Also remember to change your local DATABASE_URL and DATABASE_URL_PROD env vars values.

[streino]

Do I need to run actual tests on this or you're confident enough to go with just the code review?

[abulte]

Do I need to run actual tests on this or you're confident enough to go with just the code review?

Good question! I'm not super confident.

[streino]

Do I need to run actual tests on this or you're confident enough to go with just the code review?

Good question! I'm not super confident.

I'll try to run some tests this week then

[streino]

I'll try to run some tests this week then

Change of plan, I won't have time to test... So let's go ahead with this making sure we have a backup plan 🤞