Allow for having a reassign backoff.

[NikeNano]

Description

We have seen on our hatchet deployment that reassignment of hatchet steps is super fast. This seems to exhaust the nbr of max retries(SERVER_MAX_INTERNAL_RETRY_COUNT) during a redeploy of the k8s pods and the workflow end up failing. We have long running workflows( hours) but redeploy the workers ~20 times a day.

Made this pr as a potential fix, but mostly hope to just learn how we can run hatchet better. Please also let med know how to test this the best.

Fixes # (issue)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Documentation change (pure documentation change)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactor (non-breaking changes to code which doesn't change any behaviour)
• CI (any automation pipeline changes)
• Chore (changes which are not directly related to any business logic)
• Test changes (add, refactor, improve or change a test)
• This change requires a documentation update

What's Changed

• Add a list of tasks or features here...

[vercel]

@NikeNano is attempting to deploy a commit to the Hatchet Team on Vercel.

A member of the Team first needs to authorize it.