Implement centralized scheduling for recurring tasks

[nscuro]

Recurring tasks are currently scheduled in every instance of the API server, in TaskScheduler, using simple Timers behind the scenes.

We need to scale the API server to multiple instances (#375), but this will be problematic when every instance schedules redundant tasks. As per #375 (comment), as a short- to mid-term solution, tasks should be triggerable via REST API endpoint.

Once the endpoints are available, tasks can be scheduled via CronJob. The CronJobs merely curl an API server endpoint to trigger tasks. Requests are made against the k8s service of the API server.

CronJob supports various settings for retry, backoff and timeouts. To leverage these functionalities, API endpoints must block as long as the respective task is running.

To prevent certain heavy tasks (e.g. portfolio metrics update) from being executed concurrently, a locking mechanism must be used. Usage of https://github.com/lukas-krecan/ShedLock should be investigated.

[nscuro]

There's a working implementation for locking the portfolio metrics update task. We'll get that merged and then create new issues for the remaining work. In the meantime the new implementation can be thoroughly tested on test environments.

[mehab]

Remaining tasks

• Use cron expression instead of fixed delay scheduling until kubernetes cron jobs are used.
• Make sure end to end test are unaffected even with multiple instances of apiserver
• Should not lead to performance degradation overall

[mehab]

Currently under test and needs to be observed for a few days before confirming and closing the issue

[nscuro]

Implemented in:

• Create table for shedlock hyades-apiserver#187
• Lock singleton tasks hyades-apiserver#222
• Substituting fixed delays with cron expression hyades-apiserver#234

And tested for multiple days in a test environment. Singleton tasks are only ever executed by one instance.

Thanks @VithikaS! 🙌