ocp13 replacement

[RobHooper]

Resolves #494 and #504.

[jpmckinney]

I have noted that scrapyd needs a project configuring, on ocp13 this is configured under the datlab user (/home/datlab/.config/scrapy.cfg).

@RobHooper Is it that you observed that e.g. collect.data.open-contracting.org lists no projects? That's because we need to do something like scrapyd-deploy registry from our local computers. Not sure why Datlab has that config.

• We can document this on a new page for the registry.

[jpmckinney]

I will enable mod_md again when DNS is live (sending too many failed requests, as we are at the moment, causes Lets Encrypt to block us).

Yeah, I have a note about that in the docs https://ocdsdeploy.readthedocs.io/en/latest/develop/update/apache.html

Let’s Encrypt will reach a Failed Validation limit if DNS is not propagated.

I've now also added:

In the meantime, you can use Let’s Encrypt’s staging environment.

[jpmckinney]

• An additional manual step is to copy the exchange_rates table from the pelican_backend database. There are docs here, for the data support server.
• We also need to run the migrations for Kingfisher Process and Pelican frontend, like at https://ocdsdeploy.readthedocs.io/en/latest/deploy/servers/data-support.html#docker-apps
• Upgrade ocp13 (registry) to Ubuntu 24.04 LTS #494 mentions copying over /home/collect/scrapyd/logs

---

James when you have a moment, please can you confirm what data needs migrating to the new server from /data/.

332M	/data/storage/spoonbill

Yes, we also need to move /data/storage/spoonbill (looks like you already have).

It contains the Django media files for that project. It has a var subdirectory which is not referenced by anything and that has no changes since 2021, so I deleted it on both the old and new server (I assume it was from an old version).

---

Alongside this work, we can remove state ntp disabling the NTP service. This was only required for older systems (Ubuntu 20.04).
https://github.com/open-contracting/deploy/blob/main/salt/core/systemd/ntp.sls#L31-L34

• @RobHooper Did you want to do this, or you decided not to?

---

For the data support server, we have:

Adjust reserved disk space to 1% for large disks:

tune2fs -m 1 /dev/md2

• Do we want to do this for the registry server?

[jpmckinney]

In terms of migration process, is this the plan?

• Do we adjust the TTLs to speed up the DNS switchover?
• ocp13: Remove/comment out DATA_REGISTRY_CBOM so that new jobs aren't started
• ocp13: Delete /etc/cron.d/postgres_backups
• ocp27: We should re-copy any new files (data/storage/exporter) and databases from ocp13
• ocp27: Can we copy the mod_md files for data.open-contracting.org from ocp13? If so, we want to also restore the Apache conf files and reload Apache.
• GoDaddy: Change the DNS over for data.open-contracting.org
• Run scrapyd-deploy once DNS has propagated
• Deploy ocp27 with Salt, to install the deployer's crontab and the postgres_backups
• When satisfied, decommission ocp13

[jpmckinney]

Looks good! Just one comment.

[jpmckinney]

I added a documentation page that covers most of what I wrote above (except where I still have a question). Please add the steps to restore the data_registry and spoonbill databases from backups.

We should add a subheading on this other page, and explain in general how to restore from individual database backups versus the existing docs about pgBackrest backups: https://ocdsdeploy.readthedocs.io/en/latest/maintain/databases.html#restore-from-backup We can then just link to that from the registry page.

[RobHooper]

In the meantime, you can use Let’s Encrypt’s staging environment.

The Lets Encrypt staging environment won't generate SSL certificates without live DNS, so this doesn't help in this situation.

@RobHooper Did you want to [remove the disable NTP state], or you decided not to?

This was already done in a previous PR :)

Do we want [modify reserved disk bytes] for the registry server?

Yes, I have ran tune2fs -m 1 /dev/md2 now.
I think we should run this by default on any disk larger than ~50GBs.
Where would be best to document this?

[jpmckinney]

Where would be best to document this?

Probably at an appropriate step in create_server.rst, and maybe as a note at https://ocdsdeploy.readthedocs.io/en/latest/maintain/hosting.html#rescale-a-server

[jpmckinney]

The Lets Encrypt staging environment won't generate SSL certificates without live DNS, so this doesn't help in this situation.

It helps to not get temporarily blocked by Let's Encrypt :)

[RobHooper]

Detailed migration plan as follows:

Before go-live:

1. Inform parties (Dogsbody, OCP)

2. Reduce TTLs for:

• collect.data.open-contracting.org
• data.open-contracting.org
• flatten.open-contracting.org
• rabbitmq.data.open-contracting.org

On the day:

1. Merge GitHub Pull Request

2. Disable crons on ocp13:

• PELICAN_BACKEND_UPDATE_EXCHANGE_RATES
• DATA_REGISTRY_CBOM
• /etc/cron.d/postgres_backups

3. Disable sites on ocp13 (Ensure no new data is created and lost).

docker compose down everything except the data-registry web and static containers

4. Copy data from ocp13 and import on the new server

   4a. Rsync files

   rsync -avze "ssh -p 2223" [email protected]:/data/storage/spoonbill/ /data/storage/spoonbill/
   rsync -avze "ssh -p 2223" [email protected]:/data/storage/exporter_dumps/ /data/storage/exporter/
   rsync -avze "ssh -p 2223" [email protected]:/home/collect/scrapyd/logs /home/collect/scrapyd/logs

   4b. Migrate databases

   # Export databases on ocp13
   su - postgres -c "/usr/bin/pg_dump -Ft -f '~/spoonbill_web.tar' 'spoonbill_web'"
   su - postgres -c "/usr/bin/pg_dump -Ft -f '~/data_registry.tar' 'data_registry'"
   
   # Copy both files over
   
   # Import databases on ocp27
   sudo -u postgres pg_restore -cC --if-exists -v -U postgres -d spoonbill_web spoonbill_web.tar
   sudo -u postgres pg_restore -cC --if-exists -v -U postgres -d data_registry data_registry.tar

   4c. Migrate exchange rates table
   https://ocdsdeploy.readthedocs.io/en/latest/deploy/servers/data-support.html#pelican-backend

5. Run docker app migrations
   https://ocdsdeploy.readthedocs.io/en/latest/deploy/servers/data-support.html#docker-apps

6. Test sites are working - James and Bob

7. Update DNS to point to the ocp27 CNAME.

8. Run scrapyd-deploy once DNS has propagated - With James

9. Deploy ocp27 with Salt, to install the deployer's crontab, postgres_backups and enable Apache mod_md.

Post-Migration

1. Restore TTLs to 1 hour
2. Deploy Docs with salt updating Tinyproxy for new IP registry address
3. Deploy Prometheus with Salt
4. Block all access to ocp13, and then when we are happy decommission. (Usually a couple of weeks after the migration).

Changes in this PR to docker and rsyslog effect all docker servers so carefully deploy and restart the Docker daemon.

[RobHooper]

ocp27: Can we copy the mod_md files for data.open-contracting.org from ocp13? If so, we want to also restore the Apache conf files and reload Apache.

mod_md on the new server should instantly provision new SSL certificates as soon as DNS is live (and it is re-enabled).
I will only copy data from mod_md on ocp13 if there is a problem getting a new certificate.

[jpmckinney]

• Step 0: I need to message Dogsbody to say we're decommissioning ocp13, so there will be alerts about it, data.open-contracting.org, etc.

Reduce DNS TTLs for:

• Aren't those for ocp23? We're replacing ocp13. (data.open-contracting.org, etc.)

Disable sites on ocp13

• I think we can just (1) docker compose down everything except the data-registry web and static containers and (2) email [email protected] that work is starting (so that Yohanna doesn't edit any publications).

Test sites are working - James will you be online for this?

• At what time are you expecting?

Run scrapyd-deploy once DNS has propagated - With James?

• Yes, I can do this one. As I understand, DATA_REGISTRY_CBOM cron will be disabled until the next step.

Copy scrapyd logs

• Let's do this before step 9. Once cron starts I'm not sure how rsync'ing the directory works.

[jpmckinney]

Some other tasks I did:

• After disabling the CBOM cron, delete any jobs in Django admin, which cancels any jobs in Scrapyd: https://collect.data.open-contracting.org/jobs
• Check that RabbitMQ is quiet. Right now there's work being done, but it should finish by migration time: https://rabbitmq.data.open-contracting.org/#/queues

[jpmckinney]

Changes in this PR to docker and rsyslog effect all docker servers so carefully deploy and restart the Docker daemon

Is there anything extra to do when deploying other servers, or should Docker (and containers) restart normally?

[RobHooper]

Changes in this PR to docker and rsyslog effect all docker servers so carefully deploy and restart the Docker daemon

Is there anything extra to do when deploying other servers, or should Docker (and containers) restart normally?

Docker should restart as part of the salt deployment causing a small outage.
I will be re-deploying on servers effected tomorrow morning while traffic is quiet.