fix(exports): tasks stuck in processing TASK-1243

[rgraber]

🗒️ Checklist

1. run linter locally
2. update all related docs (API, README, inline, etc.), if any
3. draft PR with a title <type>(<scope>)<!>: <title> TASK-1234
4. tag PR: at least frontend or backend unless it's global
5. fill in the template below and delete template comments
6. review thyself: read the diff and repro the preview as written
7. open PR & confirm that CI passes
8. request reviewers, if needed
9. delete this section before merging

📣 Summary

Fixes occasional problem where tasks end up showing as 'Processing' forever.

💭 Notes

Tasks were failing when the celery task started too quickly and tried to fetch the task before it had actually hit the database. This PR implements 2 guards against this:

1. Call the tasks from within an on_commit block to ensure they only start after the task object has made it to the database
2. Automatically catch ObjectDoesNotExist exceptions and retry the task a hardcoded number of times. task.retry takes care of putting some time in between retries

👀 Preview steps

This bug tends to only appear in low traffic when the celery queue is empty. Make sure CELERY_TASK_ALWAYS_EAGER is false.

1. ℹ️ have an account and a project
2. Export the project submissions
3. 🔴 [on main] The job will show as 'Processing' indefinitely. In the worker logs, you'll see kpi.models.import_export_task.SubmissionExportTask.DoesNotExist: SubmissionExportTask matching query does not exist
4. 🟢 [on PR] The job will only show as 'Processing' for a few seconds and then will complete.

[noliveleger]

is this used anywhere?

[noliveleger]

LGTM. I left some tiny comments which are not blockers to merge this PR.
It would be great to address them, though ;-)

[noliveleger]

I think, I would add this to django settings instead having the constant in the file.