Update documentation around migrating data #6214

gherceg · 2024-01-29T16:10:02Z

After going through this process a number of times, this is where I've landed. I've tried to simplify the steps (take out what seemed unnecessary), and added/updated anything that caused issues but was needed.

Environments Affected

None

Most importantly, make it clear that CommCare needs to be stopped prior to wiping/rebuilding data, and that the correct revision of CommCare needs to be used when rebuilding/loading in data.

kaapstorm · 2024-01-29T16:20:32Z

docs/source/reference/howto/wipe_persistent_data.rst


   .. code-block::

-      $ cchq <env_name> service commcare stop
+      $ cchq <env_name> downtime start


(What's the difference between commcare stop and downtime start? When would one use one versus the other?)

Good question. This may not actually be necessary to change. Both commands operate on COMMCARE_INVENTORY_GROUPS, and stop those services using supervisorctl. The downtime command also involves creating a record of downtime in datadog which is obviously unnecessary in this case, so I can switch it back.

the downtime command does give the user the option to wait for all processes to be killed whereas service stop might take a bit of time, but by the time they run through the steps to wipe blobdb and es, Celery should be complete.

Yeah on second thought, I like downtime for that reason in particular, and updated the instructions to select the kill option when prompted in b7b46e6

kaapstorm · 2024-01-29T16:22:33Z

docs/source/installation/migration/1-migrating-project.rst

@@ -147,6 +147,8 @@ before importing data from the old environment.
 4. Import the data to the new environment
 -----------------------------------------

+* Ensure you are running the following steps from a release created using the CommCare version that you should
+  have been provided.


Maybe tell the reader that Celery needs to be stopped before running load_domain_data.

Totally up for continuing to make it clear that celery needs to be stopped, but it's tricky since it isn't as simple as stopping celery prior to running load_domain_data. If celery was running at any point after resetting postgres, it is likely the case that the db has been "corrupted" and needs to be reset again.

That is tricky. So a deploy will restart Celery, but we need to run a deploy after the environment is reset. Hmmm.

Well I think the framing should be we need to run a deploy before resetting the environment (I think I made that change here too), since the "current" release also impacts what state the database in migrated to when rebuilding the environment.

This is a critical point, so may be good to rephrase to share why

Ensure you are running the following steps from the same version of CommCareHQ as used to create the data dump being used to import data into your environment. Request for the CommCareHQ version/commit hash, if not shared.

Additionally, i think we need to do this during the initial setup. CommCareHQ is already deployed by this point and by default we deploy the latest code. So, at this point we are actually asking them to revert to an older version which isn't feasible due to migrations. Should we need a note about this when the setup is happening.

mkangia · 2024-01-30T10:09:35Z

docs/source/installation/migration/1-migrating-project.rst

@@ -147,6 +147,8 @@ before importing data from the old environment.
 4. Import the data to the new environment
 -----------------------------------------

+* Ensure you are running the following steps from a release created using the CommCare version that you should
+  have been provided.


This is a critical point, so may be good to rephrase to share why

Ensure you are running the following steps from the same version of CommCareHQ as used to create the data dump being used to import data into your environment. Request for the CommCareHQ version/commit hash, if not shared.

Additionally, i think we need to do this during the initial setup. CommCareHQ is already deployed by this point and by default we deploy the latest code. So, at this point we are actually asking them to revert to an older version which isn't feasible due to migrations. Should we need a note about this when the setup is happening.

mkangia · 2024-01-30T10:10:54Z

docs/source/reference/howto/wipe_persistent_data.rst


-#. Stop CommCare
+#. Stop CommCare services to prevent background processes from writing to databases.


mkangia · 2024-01-30T10:14:12Z

docs/source/reference/howto/wipe_persistent_data.rst


-#. Add "wipe_environment_enabled: True" to `public.yml` file.
+   .. note::
+        This is especially important if you are performing a migration of your data to a new instance. You should have


Isn't this only needed if performing a migration.
Otherwise there should not be a need to deploy?

Yeah this is true. I'll make that distinction 👍🏻

mkangia · 2024-01-30T10:17:04Z

docs/source/reference/howto/wipe_persistent_data.rst


-#. Reset PostgreSQL and PgBouncer
+#. Wipe BlobDB, Elasticsearch, and Couch using management commands.


Good to check no services are running before doing this. This could happen if

someone jumps straight to this step in the documentation without reading the block above

the services are restarted by monit or someone/something else unknowingly.

mkangia · 2024-01-30T11:37:56Z

docs/source/reference/howto/wipe_persistent_data.rst

-
-   .. code-block::
-
-      $  kafka-topics.sh --bootstrap-server localhost:9092 --list


we don't need this anymore?

My interpretation was that we never needed this but it was just a nice additional check. However given that we don't do any similar checks for the other steps that are run, I see no reason why we should treat kafka any differently and had no issues when running wipe_kafka myself. So I removed it because I prioritized simplifying these steps.

mkangia · 2024-01-30T11:39:01Z

docs/source/reference/howto/wipe_persistent_data.rst

@@ -56,41 +72,37 @@ in the sequence given below, so you shouldn't proceed to next steps until the pr

      $ cchq <env_name> ap wipe_kafka.yml

+#. Remove the "wipe_environment_enabled: True" line in your `public.yml` file.


is this a new option that has been added? "wipe_environment_enabled"?

No it is used in ansible playbooks that wipe data as an extra safety feature to prevent people from accidentally running those playbooks. I just moved it to "wrap" all of the steps in the wiping data section even though it doesn't apply to django management commands because I thought that made more logical sense.

This option was added when we added the Ansible tasks for nuking environments' data, to ensure that the people with their fingers on the button know what they're doing.

Edit: I should have refreshed my browser before responding. I see gherceg replied hours ago. ... I'll leave this here though, cos the meme is funny. 😏

mkangia · 2024-01-30T11:39:32Z

thanks for these updates @gherceg , much needed improvement to our documentation

mkangia · 2024-01-31T09:09:02Z

docs/source/installation/migration/1-migrating-project.rst



 3. Prepare the new environment to be populated
 ----------------------------------------------

+* Ensure you are running the following steps from a release created using the CommCare version/commit hash that you
+  should have been provided in Step 1. This ensures the database will be migrated to the same state it was in when
+  the data was dumped.


thanks for adding this @gherceg

There is one challenge though. The documented for deploy does not say how it is to be done. So, we would need to update there as well.

Thanks Manish I'll make a new PR as there are some other changes I'd like to make too. Feel free to add any additional comments you may still have and I'll address them there! I admittedly rushed this out just to get something closer to up to date instructions out there.

minhaminha

everything else looks good!

docs/source/installation/migration/1-migrating-project.rst

Update documentation around migrating data

1e0532d

Most importantly, make it clear that CommCare needs to be stopped prior to wiping/rebuilding data, and that the correct revision of CommCare needs to be used when rebuilding/loading in data.

gherceg requested review from kaapstorm, mkangia, dannyroberts, AmitPhulera and minhaminha January 29, 2024 16:10

kaapstorm reviewed Jan 29, 2024

View reviewed changes

kaapstorm approved these changes Jan 29, 2024

View reviewed changes

Add comment on how to answer prompt for downtime start command

b7b46e6

mkangia reviewed Jan 30, 2024

View reviewed changes

Address feedback

b344784

kaapstorm approved these changes Jan 30, 2024

View reviewed changes

gherceg merged commit f7b9777 into master Jan 30, 2024
2 checks passed

gherceg deleted the gh/data-dump/update-wipe-data-docs branch January 30, 2024 19:06

mkangia reviewed Jan 31, 2024

View reviewed changes

gherceg mentioned this pull request Jan 31, 2024

More doc updates applicable to migrating data to a new instance #6216

Merged

minhaminha reviewed Jan 31, 2024

View reviewed changes

docs/source/installation/migration/1-migrating-project.rst Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update documentation around migrating data #6214

Update documentation around migrating data #6214

gherceg commented Jan 29, 2024

kaapstorm Jan 29, 2024

gherceg Jan 29, 2024

gherceg Jan 29, 2024

gherceg Jan 29, 2024

kaapstorm Jan 29, 2024

gherceg Jan 29, 2024

kaapstorm Jan 29, 2024

gherceg Jan 29, 2024

mkangia Jan 30, 2024

mkangia Jan 30, 2024

mkangia Jan 30, 2024

mkangia Jan 30, 2024

gherceg Jan 30, 2024

mkangia Jan 30, 2024

mkangia Jan 30, 2024

gherceg Jan 30, 2024

mkangia Jan 30, 2024

gherceg Jan 30, 2024

kaapstorm Jan 30, 2024 •

edited

Loading

mkangia commented Jan 30, 2024

mkangia Jan 31, 2024

gherceg Jan 31, 2024

minhaminha left a comment


		#. Stop CommCare
		#. Stop CommCare services to prevent background processes from writing to databases.


		#. Reset PostgreSQL and PgBouncer
		#. Wipe BlobDB, Elasticsearch, and Couch using management commands.


		.. code-block::

		$ kafka-topics.sh --bootstrap-server localhost:9092 --list

		@@ -56,41 +72,37 @@ in the sequence given below, so you shouldn't proceed to next steps until the pr

		$ cchq <env_name> ap wipe_kafka.yml

		#. Remove the "wipe_environment_enabled: True" line in your `public.yml` file.

Update documentation around migrating data #6214

Update documentation around migrating data #6214

Conversation

gherceg commented Jan 29, 2024

Environments Affected

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaapstorm Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

mkangia commented Jan 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

minhaminha left a comment

Choose a reason for hiding this comment

kaapstorm Jan 30, 2024 •

edited

Loading