Refactor backups & add backups monitoring

[agoncharov-reef]

Total refactoring of backups.

• Instead of setting up backups manually on host, the backups container is automatically started with everything set up within. One deploys an app, he gets backups working automatically.
• The container does two things: runs cron and serves backup metrics on port 8000
• We check whether latest backup is not currupted by running pg_restore -l on it and searching for TABLE DATA public django_migrations in the output. It's dumb but better than nothing for now. For backups in b2, we download only first 10MiB of data, and run the above command on that chunk - so that we don't download the whole large backup. It seems to be enough to read the ToC of postgres archive (I checked manually).
• Backups are now auto-created during deployment (on poor man's deployment only though)
• Tested poor man's setup on Timas; haven't tested AWS setup
• Metrics are already set up in prometheus (with_backups=True) and Grafana:
  

[emnoor-reef]

opinion: checking if a backup is operational every time we update metrics is a bit too much I think, specially when using b2. We are creating new backups every 24 hours, but checking them every 10 minutes.

[olzhasar-reef]

Why not a separate package for this though?
We have a lot of projects that are way behind the current cookiecutter state, and we seem to consistently increase the upgrade cost by introducing large changes to the template.

[agoncharov-reef]

@olzhasar-reef It's a docker container, idk how to make a package from it.
I know that there is this problem of syncing old stuff with latest cookiecutter, but I think benefits are bigger than drawbacks, i.e. we are ready to pay the price of upgrading old projects.
Also, moving it to separate package doesn't mean that upgrade costs go away - now you'll need to upgrade that package in old projects, and track compatibility.
You may open a discussion on s3 and invite me, if we all decide some better design then i'll be able to implement it quickly now that everything is still fresh in my head.