Replies: 8 comments 17 replies
-
I would keep it in requirements, but I don't think this is a critical decision
In a dev environment I would probably enabled the database-backed vault ? Generally speaking I would prefer to not call it tool secrets, and I wouldn't limit it to tools. You could think of workflow actions that are not tool steps that require secrets, like when you'd interact with an external service e.g. for exports after a workflow has completed. I would maybe make a user_secrets table with uuid as the primary key plus the corresponding CRUD endpoints. You can still filter by tool / tool version in the endpoint, but I wouldn't make it part of the path operation. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you Marius for the feedback! So, for the XML elements, what do you think about this?: <requirements>
<credentials name="service1" reference="some/unique/reference" required="true" label="My awesome API" description="Please provide credentials for blah…">
<secret name="my_token" inject_as_env="target_env_var_apikey" label="API Key" description="This is the API Key for blah…" />
</credentials>
<credentials name="service2" reference="some/other/unique/reference" required="false">
<secret name="s2_username" inject_as_env="target_env_var_user" label="Your username" />
<secret name="s2_password" inject_as_env="target_env_var_pass" label="Your password" />
</credentials>
</requirements>
For the database schema, what about this?: erDiagram
USER_SECRETS {
uuid id PK
int user_id
string service_reference UK
string secret_name
}
The
This should be generic enough. There's probably no need to store the tool ID and version in the database, as we can get this information from the tool's XML requirements and then check if the user has provided credentials for that service/secret. And finally the API endpoints could look like this:
|
Beta Was this translation helpful? Give feedback.
-
I didn't see any examples of this, but I'd be interested to, since the use case would be relevant for @abretaud and myself. E.g. the suite of Apollo tools would all want to re-use the same credentials because they all talk to the same backend, even across multiple tools. So I'd be looking at writing something like this, yes? <requirements>
<credentials name="Apollo" reference="gmod.org/apollo" required="true">
<secret name="server" inject_as_env="apollo_url" label="Your Apollo server" />
<secret name="username" inject_as_env="apollo_user" label="Your Apollo username" />
<secret name="password" inject_as_env="apollo_pass" label="Your Apollo password" />
</credentials>
</requirements>some questions based on this use case:
That UX I'm not sure if it makes sense for me, given how all of the tools I manage that require credentials, would be used. Apollo and the XNAT tool we're working on all have the built in assumption that they might want to talk to different servers at some point. So I think I expected this to be a tuple, sort of. That the user would be asked for credentials, they would provide a set of secrets for a specific credential, and then save that with a name. Then when the user needs to run the tool, they can select between tuples with their nicknames, to say "I want to use the dev apollo" or "the production apollo". |
Beta Was this translation helpful? Give feedback.
-
I agree with David's proposal. We've discussed this at the backend meeting and we're going to start out simple. Any sort of advanced sharing is out of scope initially.
This is not possible using tool shed installed tools. |
Beta Was this translation helpful? Give feedback.
-
oh, then I misunderstood "facilitate a team discussion", my apologies! If the scope is set in stone already, please feel free to ignore my comments. I genuinely understand needing to split up work into reasonable units. Just, if it's a unit of work that ends with "now tool authors should use this here is the documentation", just want to check it is something that's useful for me as a tool author? Do you see a path to sharing the credentials across tools in a future version? Can that sort of implementation decision be pushed off with migrations? (I'm not primarily a dev, I wouldn't know.) Would the existing credentials will be shown to the user? Or would it not find their credentials and they would have to re-enter them?
yes, but I have that. Currently. This would be a regression for me to switch to. I fear that this "advanced sharing" isn't an advanced use case, rather it is the most common one. The apollo and tripal tools work like that now, the user provides credentials once and they're re-used. The IUC Beacon2 suite does as well, OMERO as well. EGA is an outlier, but we've considered adding more tools to that suite for better user experience. |
Beta Was this translation helpful? Give feedback.
-
|
When a user complains it's too much work to pass credentials to a tool we can consider the next step, but I wouldn't invert this order with over-engineering a complex solution. Surely there can be a front-end hint asking if a similar credential type can be used, I don't think we need to plan this form the start. |
Beta Was this translation helpful? Give feedback.
-
|
If that's the decision, I understand. We can end the discussion. I just wanted to consider the requirements of those who are heavy tools-with-credentials users like me. I will probably stick to user preferences then, for my suites, and perhaps I can use this if I ever write individual tools. |
Beta Was this translation helpful? Give feedback.
-
|
It's not like there is a decision and we will not alter any of it, we only discussed this aspect at the backend meeting with the outcome that we will focus on a simple and safe implementation of a feature we currently do not have.
Would you agree that this is a workable compromise between being explicit and convenient ? The feature itself will be considerably more discoverable both by admins and users, so I hope that you'll consider it for your tool suites as well. Given there isn't much of an annotation you could of course continue to have the current mechanism as a fallback. |
Beta Was this translation helpful? Give feedback.
-
|
My only bit to add would be that it might be cool to have optional credentials, e.g. for services that allow anonymous login. |
Beta Was this translation helpful? Give feedback.
-
|
We will try to support |
Beta Was this translation helpful? Give feedback.
-
|
Cool. Would argue for |
Beta Was this translation helpful? Give feedback.
-
|
Also wanted to add this for ref: #15300 |
Beta Was this translation helpful? Give feedback.
-
|
Example tool XML requirements <requirements>
<credentials name="serviceA" reference="remote_server_A" optional="false" multiple="false" label="Credentials for Service A" description="Optional description of the service using credentials">
<variable name="server" inject_as_env="serviceA_url" label="Your Service1 server" description="You can set the server URL..."/>
<secret name="username" inject_as_env="serviceA_user" label="Your Service1 username" description="Your username is your email"/>
<secret name="password" inject_as_env="serviceA_pass" label="Your Service1 password" description="This needs to be a strong password"/>
</credentials>
<credentials name="serviceB" reference="remote_server_B" optional="true" multiple="true" label="Another service" description="Optional description of the service using credentials">
<secret name="api_token" inject_as_env="serviceB_token" label="API Token" description="This is the API token for service B"/>
</credentials>
</requirements>Database Model Design erDiagram
user_credentials {
int id PK
int user_id FK
string reference
string source_type
string source_id
int current_group_id FK
}
user_credentials_group {
int id PK
int user_credentials_id FK
string name
}
credential_variable {
int id PK
int user_credential_group_id FK
string name
string value
}
credential_secret {
int id PK
int user_credential_group_id FK
string name
boolean already_set
}
user_credentials ||--|{ user_credentials_group : "Has"
user_credentials_group ||--o{ credential_variable : "Has"
user_credentials_group ||--o{ credential_secret : "Has"
This schema introduces: API Endpoints
Here is an example of a Payload: {
"source_type": "tool",
"source_id": "test_tool",
"credentials": [
{
"reference": "test_service",
"current_group": "default",
"groups": [
{
"name": "default",
"variables": [{"name": "server", "value": "http://localhost:8080"}],
"secrets": [
{"name": "username", "value": "user"},
{"name": "password", "value": "pass"},
{"name": "token", "value": "key"}
]
}
]
}
]
}And this an example of a Response: [{
"user_id": "f2db41e1fa331b3e",
"id": "1cd8e2f6b131e891",
"source_type": "tool",
"source_id": "test_tool",
"reference": "test_service",
"current_group_name": "default",
"groups": {
"default": {
"id": "ebfb8f50c6abde6d",
"name": "default",
"variables": [{
"id": "f597429621d6eb2b",
"name": "server",
"value": "http://localhost:8080"
}],
"secrets": [{
"id": "ebfb8f50c6abde6d",
"name": "username",
"already_set": true
}, {
"id": "33b43b4e7093c91f",
"name": "password",
"already_set": true
}, {
"id": "a799d38679e985db",
"name": "token",
"already_set": true
}]
}
}
}]Questions
|
Beta Was this translation helpful? Give feedback.
-
|
I would be interested to know the tool test syntax. |
Beta Was this translation helpful? Give feedback.
-
|
That is a good point! We haven't thought about this yet and, I can't imagine right now how that would be done 🤔 Would asserting that a particular environment variable exists with certain content be sufficient? @bernt-matthias tapping into your deep experience with tools, how would you envision the tests for this? Any expert help would be greatly appreciated! |
Beta Was this translation helpful? Give feedback.
-
|
@arash77 First of all, thank you for providing a very detailed description of the model - this is very helpful! I have a few questions/comments on the db schema:
(I'll comment on the model definition in the PR) |
Beta Was this translation helpful? Give feedback.
-
|
Thank you so much @jdavcs! That is exactly the kind of feedback we were looking for :)
I can try to explain this one. The idea was that "variables" would store the value in the database as they are not considered sensitive, while "secrets" would just store whether "the value was stored in the vault" (already_set) but the actual value will always be in the vault and never retrieved in a DB query. We only need to know that the secret value was provided to display a proper placeholder to the user in the UI hence the |
Beta Was this translation helpful? Give feedback.
-
|
@davelopez Thank you for the explanation, now it makes sense. I'm wondering if we can simplify it a bit.. I may be missing a use case or some obvious limitation, etc. So, as I understand, both the secret and the variable represent a "thing" the user provides. The "thing" belongs to a Of course, we'd have some validation to ensure that if is_secret is true, value must be null. (deriving Also, if the above makes sense, I take back what I said about plural vs. singular with regard to "credentials": in this care the table would indeed store records of ONE thing - a single credential value per row (and how many such values constitute a single instance of "user credentials" is irrelevant). |
Beta Was this translation helpful? Give feedback.
-
|
Thank you again! That looks like a nice simplification. We will try it.
Both "variables" and "secrets" can be optional (for example if the tools use a sensible default or the service allows anonymous access, etc.). When you provide "some" of these credentials in the UI, and hit "Save" we are currently sending all of them, including the optional ones but with their values set to null. In the case of secrets, when we send a value of null it means "the secret has not changed". But I'm sure we can adapt to this simplified model and maybe send only what needs to be updated or set. |
Beta Was this translation helpful? Give feedback.
-
|
The config system had a similar kind of requirement: to tell the difference between a default value and an option explicitly set by the user - so the |
Beta Was this translation helpful? Give feedback.
-
Would it be an option to specify the credentials in the test itself? <test>
<credentials name="service1" >
<variable name="server" value="..."/>
<secret name="username" value="..."/>
<secret name="password" value="..."/>
</credentials>
...
</test>Problem might then be to store real credentials in a github CI (if a test uses a real service). Environment variables might be a good and easy alternative (could be handled with github secrets I guess). |
Beta Was this translation helpful? Give feedback.
-
|
We have thinking about this way but the problem as you mentioned is that the secrets should NOT be exposed. |
Beta Was this translation helpful? Give feedback.
-
|
or maybe we could skip how credentials are working (API/db/vault) and just force the test to use the env variable that we provide for it. |
Beta Was this translation helpful? Give feedback.
-
After an initial discussion with @bgruening and @arash77, this proposal expands on #1751 to facilitate a team discussion on design and implementation strategies. An initial prototype is underway at #19084.
Overview
The goal is to enable Galaxy tools to securely declare, request, and manage secret tokens or credentials that users can provide. This proposal outlines a way for tool authors to define credential requirements, a user interface (UI) to request and manage credentials, and the backend to store and retrieve them securely.
Declaring Tool Secrets
Some options for declaring tool secrets in the tool’s XML are:
Option A: Within the
<requirements>sectionTool authors will be able to declare that a tool requires credentials by adding a
<secret>element within the<requirements>section, for example:The fields are defined as follows:
type: Supports “token” for simple API keys and potentially “credentials” for username+password pairs.reference: A unique path-like string for the Vault key. This will default to a prefixed form of the tool ID and version to avoid accidental sharing but can be overridden for cases where secrets should be shared across tools.inject_as_env: Specifies the environment variable where the credentials should be injected.labelanddescription: Provide information to display in the UI when requesting credentials from users.required: A boolean that indicates if the credential is mandatory or if the tool can run with anonymous access.Option B: Within a dedicated
<secrets>section supporting multiple service credentialsWe could instead introduce a dedicated
<secrets>section, making it easier to distinguish credentials from other tool requirements.For tools that require multiple credentials or credentials with multiple parts (e.g., username and password):
Requesting Tool Secrets in the UI
In the Tool Form
When a tool that requires credentials is opened, a warning banner will appear at the top of the Tool Form. This banner will display a button that, when clicked, opens a dialog where users can input the required credentials. If credentials have already been provided, the banner will turn green with a message like “You have already provided credentials for this tool,” along with an option to edit them.
In the Workflow Run Form
When a workflow containing a tool that requires credentials is run, the user will be prompted to provide the necessary credentials before the workflow can be executed. This will be similar to the tool form, with a warning banner at the top of the form and a dialog to input credentials. When requesting credentials for a workflow, the system will check if the user has already provided credentials for all tools in the workflow (and sub-workflows) and only prompt for missing credentials.
Backend: Storing and Managing Tool Secrets
Once provided by the user, credentials will be stored in the user Vault, with references saved in a new table in the Galaxy database called
user_tool_secrets. This table will contain at the very minimum:This enables the system to check whether the user has provided credentials for a tool and to update the UI accordingly.
Users will manage their credentials in a new section in their user preferences. This section will list all tools for which the user has provided credentials and allow them to edit, delete, or copy credentials between tools if needed.
API Endpoints for Tool Secrets Management
To support these operations, we propose adding the following API endpoints for discussion:
/api/user/{id}/tool_secrets- Lists all credentials the user has provided (credentials themselves are not included)./api/user/{id}/tool_secrets/{tool_id}/{tool_version}- Verifies if credentials have been provided for a specific tool (no credential data returned)./api/user/{id}/tool_secrets/{tool_id}/{tool_version}- Allows users to provide credentials for a tool or copy them from another tool or version./api/user/{id}/tool_secrets/{tool_id}/{tool_version}- Updates credentials for a specific tool./api/user/{id}/tool_secrets/{tool_id}/{tool_version}- Deletes credentials for a specific tool.Open Questions
Q1*: Placement of
<secret>Element. Should we introduce a dedicated section in the tool’s XML instead of including secrets in the section? This might make the code more readable and organized.Q2*: Vault as a Requirement. Since all secrets should be securely stored, we assume the Vault is required. Should we make this a hard requirement or allow configurations that skip the Vault, storing secrets in user preferences instead? This flexibility might benefit testing or smaller Galaxy instances that don’t need a Vault, though it will make it not production-ready.
Other alternatives?
Please share your thoughts on the proposed design and suggest any alternatives or improvements.
Beta Was this translation helpful? Give feedback.
All reactions