feature/plugins → develop: Plugin Support (WIP)

[lbeckman314]

⚙️ Overview

Adapted from RPC-based plugins in Go by Eli Bendersky and go-plugin examples 🚀

This PR adds initial support for Plugins using the go-plugin package from HashiCorp.

In this setup, the Plugin handles all user authentication, with the Server having no "knowledge" or record of user credentials/tokens (e.g. example-users.csv).

🔄 Sequence Diagram

Created with https://sequencediagram.org (source)

⚡ Quick Start

1. Start the Server

➜ cd plugins

➜ make
Building ./server...OK

➜ ./server
Listening on http://localhost:8080

2. Send Requests

In another terminal, send the request using of the examples below —

Required Inputs ⚙️

Input	Sent in...	Example
User	Authorization Header	Authorization: Bearer Alyssa P. Hacker
Task	Request Body	@example-tasks/hello-world.json

Example Users ⚙️

All "Authorized" users may be found in the "User Database" — example-users.csv:

• Alyssa P. Hacker, a Lisp hacker
• Ben Bitdiddle
• Cy D. Fect, a "reformed C programmer"
• Eva Lu Ator
• Lem E. Tweakit
• Louis Reasoner, a loose reasoner

Here the use of a simple text file to contain users is to show how a plugin can be written and integrated into Funnel — real world use cases might involve more complex mechanisms for tracking users (e.g OAuth)...

Example: Authorized User

Here we send a request to authenticate a user named 'Alyssa P. Hacker' who is an Authorized user (i.e. found in the "User Database" — example-users.csv):

➜ curl --header "Authorization: Bearer Alyssa P. Hacker" \
       --data @example-tasks/hello-world.json \
       http://localhost:8080

Response: {Alyssa P. Hacker <Alyssa's Secret>} ✅

Example: Unauthorized User

Here's an example of attempting to authenticate a user named 'Foo', representing an Unauthorized user:

➜ curl --header "Authorization: Bearer Foo" \
       --data @example-tasks/hello-world.json \
       http://localhost:8080

Error: User Foo not found ❌

FAQ 🕵️

Will plugins have the ability to update the worker configuration (e.g. S3 bucket, region, key ID and key secret)?

Yes, plugins will modify the local worker configuration the local worker used during the task's duration. This information is not stored globally or shared between different tasks or workflows.

Will plugins have the ability to access the TES task’s tags?

Yes — the entire TES task is passed from the Worker to the plugin. This allows the plugin to authorize users based on task information (like tags).

Will the Worker and Executor support hitting a configured S3-compatible endpoint instead of the AWS endpoint?

Yes, the Worker and Executor support both AWS S3 endpoints as well as custom S3 endpoints such as MinIO, Ceph, etc.

Will plugins have the ability to access the funnel configuration (e.g. a client_id and a client_secret for use by the plugin code)?

Yes, the plugins will have access to the Worker config and all values in it.

If we sent a JWT access token to funnel during task creation, would it be feasible for funnel to receive it and for the plugin code to have access to it?

Yes, the plugin supports passing any arbitrary data such as JWT tokens.

🚧 Next Steps

• Add Server integration
• Add updated sequence diagram of Server/Plugin interactions
• Add steps/docs for writing additional custom plugins to integrate with Funnel...
• Add real world use case example/docs
• Add expected use cases, examples, and docs

📚 Additional Resources

• https://github.com/hashicorp/go-plugin
• https://pkg.go.dev/github.com/hashicorp/go-plugin
• https://eli.thegreenplace.net/2023/rpc-based-plugins-in-go
• https://github.com/eliben/code-for-blog/tree/main/2023/go-plugin-htmlize-rpc

[netlify]

✅ Deploy Preview for ohsu-comp-bio-funnel canceled.

Name	Link
🔨 Latest commit	a8ed80a
🔍 Latest deploy log	https://app.netlify.com/sites/ohsu-comp-bio-funnel/deploys/67bfecffb6a5940009071e20

[lbeckman314]

Plugin Update 🌀

Example Config ✍️

config.yaml:

Plugins:
  Disabled: false
  Dir: plugin-binaries/
  Plugins:
    - auth

Example Steps 🚀

Server

➜ gh pr checkout 1150

➜ make install 

➜ funnel server run --config config.yaml
server               Server listening
httpPort             8000
rpcAddress           :9090
time                 2025-02-26T20:19:25-08:00

User not authorized: <nil>

Client

➜ funnel task create examples/hello-world.json
# Authorization Error

Next Steps 🚧

• Add support to configure endpoint that Plugin should target (currently hardcoded to mock auth database)
• Run gen3-workflow as a standalone service (mock Fence Auth?)
• Target /s3 endpoint