Combine all schedulers for all organisations

[jpbruinsslot]

Proposal

The current situation is that we create individual 'schedulers' for every organisation, this amounts to 3 schedulers for every organisation.

The addition of queues in the scheduler being persisted as a database table allows us to explore the possibility of running a dedicated schedulers for all organisations.

Create one BoefjeScheduler, NormalizerScheduler and ReportScheduler for all organisations instead of individual schedulers for every organisation. One message queue for all scan profile mutations to which scan profile mutations are posted of all organisations for the BoefjeScheduler, and one message queue for raw file creation of all organisations for the NormalizerScheduler, the ReportScheduler will reference its internal database table.

Advantages

• Less overhead of creating and removing schedulers for every organisation
• Stops continuous checking of available organisations
• Allows to replicate scheduler application instances when used with a blocking messaging queue
• Additionally we need to support popping multiple tasks from the endpoint

Disadvantages

• the allow_updates, allow_replace, allow_priority_updates for individual organisations can't be used, however this function doesn't seem to be used and is for every type scheduler the same
• we should investigate how we solve when organisations wan to disable a scheduler, and even if this has a use-case

Impact

• now we have several different rabbitmq messaging queues for every organisation, this needs to be reduced to one
• organisation ids need to be passed in the messages on the queue
• changes to task runner to pop off tasks
• rocky likely needs to be update to interface with changes from the scheduler API

Next steps and impact

• Pop endpoint has changed from /queues to /schedulers/{id}/pop , additionally it will return a paginated result instead of a single Task , this is because the pop endpoint now supports filtering with multiple tasks returns. Services that rely on the scheduler pop endpoint need to update their interfaces (Update services that rely on /pop endpoint of scheduler #3961)
• Push endpoint changed from /queues to schedulers/{id}/push, services that interface with the push endpoints (rocky) need to update their interfaces. (Update services that rely on /push endpoint of scheduler #3962)
• scan profile mutation message queue, for every organisation a message queue is created for scan profile mutations, this needs to be updated and all scan profile mutations for every organisation needs to be relayed on a single scan profile mutations message queue (Combine organisational scan profile mutation message queues #3963)
• raw data file received message queue (the as with scan profile mutations) (Combine raw data recieved message queues #3964)
• Model definitions updates: organisation fields are added to Task , Schedule , services using these models need to update their specifications. (Model definition updates: addition of organisation field to Task and Schedule models #3965)
• Batch status updates. Several places in the scheduler we can consider batch updating status field of task. Potentially exposing this as a endpoint. This because the task runner will be able to pop off multiple task from a scheduler, it might therefore be beneficial to batch update the tasks. (
• Deleting of organisations needs to be addressed, and what the protocol needs to be with regards of queued tasks, and schedules. (Discussion: how do we handle the deletion of organisations? #3966)

PR

#3839