RP Integration Points / Agent Isolation

[andre-merzky]

We frequently face use cases / project which require the integration of RP's task execution capabilities into 3rd part softwares. This is, in some sense, trivial as RP exposes an API which can be used for such purposes. However, the resulting solutions then inherit properties of RP which do not always match the use cases' boundary conditions. For example, RP has a client and agent module - integration via the RP API always requires the (rather heavy-weight) client module to be instantiated even though the task execution capabilities are actually provided by the agent. As another example, MongoDB is always necessary as it facilitates the communication between client and agent modules.

This discussion serves the purpose to consider additional integration points for such use cases. We repeatedly entertained the notion of 'agent isolation' as one possible approach: the agent module would be extracted from the RP stack in a way that it provides, in itself, a well defined integration API or endpoint and as such could provide task execution capabilities w/o necessitating the client module's integration. But other approaches might be possible, and this thread is open to discussing those also.

[andre-merzky]

PARSL:

• RP-PARSL integration is a representative use case
• most straight forward integration would be via ZMQ messages
• HTX: executor is client side, workers are shipped to compute nodes, PARSL is pre-installed
• PARSL provides documentation on integration with 3rd-party systems

HEP-Cloud: additional use case

Client purposes:

• launch agent
• send workload to agent / obtain state updates
  Client should use integration API, we can have multiple / modular clients.

Agent could have multiple interfaces: REST, ZMQ, File System.

TODO: derive Agent requirements (functional, non-functional) from use cases, then design agent interface, then define implementation steps.

[andre-merzky]

After discussion with @mtitov and @mturilli we agreed that the agent integration API can be developed incrementally, by

• adding it to the existing agent implementation
• coding stand-alone clients against that API for
  • launching the agent w/o RP client module
  • submitting work to the agent w/o RP client module
  • etc
• co-evolve those clients with the integration API
• once the set of clients using the new API is feature complete we can begin to dismantle the RP client module

[AymenFJA]

The most reasonable interface, at least based on the current use cases/requirements, is the ZMQ interface. ZMQ interface will allow us to reduce the code footprints as we already have our code base that is implemented based on ZMQ (unlike REST and File System Interfaces), and it will allow us to avoid the interaction with the File system when shipping the tasks (unlike the File system interface) since we need to ship many serialized objects between the user interface and the agent continuously.

[andre-merzky]

Scope of API:

• services need to be started / stopped, or get reconfigured during agent runtime
• non-functional: high volume calls - is ZMQ / Rest able to handle those for remote connections?