Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update documentation around migrating data #6214

Merged
merged 3 commits into from
Jan 30, 2024
Merged

Update documentation around migrating data #6214

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1e0532d
Update documentation around migrating data
gherceg Jan 26, 2024
b7b46e6
Add comment on how to answer prompt for downtime start command
gherceg Jan 29, 2024
b344784
Address feedback
gherceg Jan 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 23 additions & 14 deletions docs/source/installation/migration/1-migrating-project.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,10 +102,9 @@ Only after all the devices are updated to use a new/mobile URL, you can proceed
2. Pull the domain data from the old environment
------------------------------------------------

The migration will require you to block data access to prevent loss of data
created during the migration. In spite of this, a practice run may also be done
without this data freeze, though the data will need to be cleared before the
real run.
The migration will require you to block data access to prevent loss of data created during the migration. If you
would like to do a practice run, you will still need to block data access to ensure the exported data is in a
clean state, and the data will need to be cleared before the real run.

During the downtime, mobile users will still be able to collect data, but they
will be unable to submit forms or sync with the server.
Expand All @@ -125,30 +124,40 @@ will be unable to submit forms or sync with the server.
* ``./manage.py dump_domain_data <domain_name>``
* ``./manage.py run_blob_export --all <domain_name>``

* Transfer these two files to the new environment.
.. note::
It is important to have the commit hash that ``dump_domain_data`` and ``run_blob_export`` were run from. If
Dimagi does not provide you with this commit hash, please followup to ensure you are able to reference this
hash in future steps.

* Transfer these files to the new environment.

.. note::
If you are not able to use your own domain for a test run and would like dump data for a test domain for practising or testing, please contact us via https://forum.dimagi.com/c/developers/ with subject "Request for test domain dump data for migration testing" and mention this page. A dimagi developer will provide you above data for any test/QA domains (casesearch, ccqa, dataregistry, qateam, ben-test, qa-erm-v1-downstream1) from https://staging.commcarehq.org.
If you are not able to use your own domain for a test run and would like dump data for a test domain for
practising or testing, please contact us via https://forum.dimagi.com/c/developers/ with subject
"Request for test domain dump data for migration testing" and mention this page. A dimagi developer will
provide you above data for any test/QA domains (casesearch, ccqa, dataregistry, qateam, ben-test, qa-erm-v1-downstream1)
from https://staging.commcarehq.org.


3. Prepare the new environment to be populated
----------------------------------------------

* Ensure you are running the following steps from a release created using the CommCare version/commit hash that you
should have been provided in Step 1. This ensures the database will be migrated to the same state it was in when
the data was dumped.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for adding this @gherceg

There is one challenge though. The documented for deploy does not say how it is to be done. So, we would need to update there as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Manish I'll make a new PR as there are some other changes I'd like to make too. Feel free to add any additional comments you may still have and I'll address them there! I admittedly rushed this out just to get something closer to up to date instructions out there.

* Setup a new environment by following :ref:`deploy-commcarehq`
* Do a commcare-hq deploy using :ref:`operations/2-deploys:Deploying CommCare HQ code changes`
* Follow steps in
:ref:`reference/howto/wipe_persistent_data:How To Rebuild a CommCare HQ environment`
to ensure your environment is in a clean state before attempting to import data.
* Proceed to step 4.

If you have performed any tests on your new environment that has created test data, to delete
the data you can rebuild your environment using
:ref:`reference/howto/wipe_persistent_data:How To Rebuild a CommCare HQ environment`
before importing data from the old environment.


4. Import the data to the new environment
-----------------------------------------

* Ensure you are running the following steps from a release created using the CommCare version that you should
have been provided.
* Ensure you are running the following steps from a release created using the CommCare version/commit hash that you
should have been provided in Step 1. This ensures the database will be migrated to the same state it was in when
the data was dumped.

* Import the dump files (each blob file will need to be imported individually)

Expand Down
13 changes: 9 additions & 4 deletions docs/source/reference/howto/wipe_persistent_data.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,16 @@ Prior to Wiping Data
$ cchq <env_name> django-manage check_services


#. Deploy CommCare from a specific revision
#. If planning to migrate data, deploy CommCare from a specific revision

.. code-block::

$ cchq <env_name> deploy commcare --commcare-rev=<commit-hash>

.. note::
This is especially important if you are performing a migration of your data to a new instance. You should have
been given a commit hash that matches the revision of CommCare used to generate your exported data, and it is
critical that this same CommCare revision is used to rebuild the new environment, and load data in.
You should have been given a commit hash that matches the revision of CommCare used to generate your
exported data, and it is critical that this same CommCare revision is used to rebuild the new environment,
and load data in. Please request a commit hash if you were not provided one.

#. Stop CommCare services to prevent background processes from writing to databases.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍


Expand All @@ -41,6 +41,11 @@ How To Wipe Persistent Data
These steps are intended to be run in the sequence given below, so you shouldn't proceed to next step until
the prior step is completed.

#. Ensure CommCare services are stopped to prevent background processes from writing to databases.

.. code-block::

$ cchq <env_name> service commcare status

#. Add "wipe_environment_enabled: True" to `public.yml` file.

Expand Down
Loading