feat(platform): OAuth support + API key management + GitHub blocks (#…

…8044)

## Config
- For Supabase, the back end needs `SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, and `SUPABASE_JWT_SECRET`
- For the GitHub integration to work, the back end needs `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET`
- For integrations OAuth flows to work in local development, the back end needs `FRONTEND_BASE_URL` to generate login URLs with accurate redirect URLs

## REST API
- Tweak output of OAuth `/login` endpoint: add `state_token` separately in response
- Add `POST /integrations/{provider}/credentials` (for API keys)
- Add `DELETE /integrations/{provider}/credentials/{cred_id}`

## Back end
- Add Supabase support to `AppService`
- Add `FRONTEND_BASE_URL` config option, mainly for local development use

### `autogpt_libs.supabase_integration_credentials_store`
- Add `CredentialsType` alias
- Add `.bearer()` helper methods to `APIKeyCredentials` and `OAuth2Credentials`

### Blocks
- Add `CredentialsField(..) -> CredentialsMetaInput`

## Front end
### UI components
- `CredentialsInput` for use on `CustomNode`: allows user to add/select credentials for a service.
  - `APIKeyCredentialsModal`: a dialog for creating API keys
  - `OAuth2FlowWaitingModal`: a dialog to indicate that the application is waiting for the user to log in to the 3rd party service in the provided pop-up window
- `NodeCredentialsInput`: wrapper for `CredentialsInput` with the "usual" interface of node input components
- New icons: `IconKey`, `IconKeyPlus`, `IconUser`, `IconUserPlus`

### Data model
- `CredentialsProvider`: introduces the app-level `CredentialsProvidersContext`, which acts as an application-wide store and cache for credentials metadata.
- `useCredentials` for use on `CustomNode`: uses `CredentialsProvidersContext` and provides node-specific credential data and provider-specific data/functions
- `/auth/integrations/oauth_callback` route to close the loop to the `CredentialsInput` after a user completes sign-in to the external service
- Add `BlockIOCredentialsSubSchema`

### API client
- Add `isAuthenticated` method
- Add methods for integration OAuth flow: `oAuthLogin`, `oAuthCallback`
- Add CRD methods for credentials: `createAPIKeyCredentials`, `listCredentials`, `getCredentials`, `deleteCredentials`
- Add mirrored types `CredentialsMetaResponse`, `CredentialsMetaInput`, `OAuth2Credentials`, `APIKeyCredentials`
- Add GitHub blocks + "DEVELOPER_TOOLS" category
- Add `**kwargs` to `Block.run(..)` signature to support additional kwargs
- Add support for loading blocks from nested modules (e.g. `blocks/github/issues.py`)

#### Executor
- Add strict support for `credentials` fields on blocks
- Fetch credentials for graph execution and pass them down through to the node execution

.github/workflows/platform-backend-ci.yml

@@ -105,7 +105,7 @@ jobs:
           LOG_LEVEL: ${{ runner.debug && 'DEBUG' || 'INFO' }}
           DATABASE_URL: ${{ steps.supabase.outputs.DB_URL }}
           SUPABASE_URL: ${{ steps.supabase.outputs.API_URL }}
-          SUPABASE_SERVICE_KEY: ${{ steps.supabase.outputs.SERVICE_ROLE_KEY }}
+          SUPABASE_SERVICE_ROLE_KEY: ${{ steps.supabase.outputs.SERVICE_ROLE_KEY }}
           SUPABASE_JWT_SECRET: ${{ steps.supabase.outputs.JWT_SECRET }}
     env:
       CI: true

autogpt_platform/autogpt_libs/autogpt_libs/auth/middleware.py

@@ -7,12 +7,13 @@
 from .jwt_utils import parse_jwt_token
 
 security = HTTPBearer()
+logger = logging.getLogger(__name__)
 
 
 async def auth_middleware(request: Request):
     if not settings.ENABLE_AUTH:
         # If authentication is disabled, allow the request to proceed
-        logging.warn("Auth disabled")
+        logger.warn("Auth disabled")
         return {}
 
     security = HTTPBearer()
@@ -24,7 +25,7 @@ async def auth_middleware(request: Request):
     try:
         payload = parse_jwt_token(credentials.credentials)
         request.state.user = payload
-        logging.info("Token decoded successfully")
+        logger.debug("Token decoded successfully")
     except ValueError as e:
         raise HTTPException(status_code=401, detail=str(e))
     return payload

autogpt_platform/autogpt_libs/autogpt_libs/supabase_integration_credentials_store/types.py

@@ -29,20 +29,29 @@ class OAuth2Credentials(_BaseCredentials):
     scopes: list[str]
     metadata: dict[str, Any] = Field(default_factory=dict)
 
+    def bearer(self) -> str:
+        return f"Bearer {self.access_token.get_secret_value()}"
+
 
 class APIKeyCredentials(_BaseCredentials):
     type: Literal["api_key"] = "api_key"
     api_key: SecretStr
     expires_at: Optional[int]
     """Unix timestamp (seconds) indicating when the API key expires (if at all)"""
 
+    def bearer(self) -> str:
+        return f"Bearer {self.api_key.get_secret_value()}"
+
 
 Credentials = Annotated[
     OAuth2Credentials | APIKeyCredentials,
     Field(discriminator="type"),
 ]
 
 
+CredentialsType = Literal["api_key", "oauth2"]
+
+
 class OAuthState(BaseModel):
     token: str
     provider: str

autogpt_platform/backend/.env.example

@@ -11,13 +11,30 @@ REDIS_HOST=localhost
 REDIS_PORT=6379
 REDIS_PASSWORD=password
 
-ENABLE_AUTH=false
 ENABLE_CREDIT=false
 APP_ENV="local"
 PYRO_HOST=localhost
 SENTRY_DSN=
-# This is needed when ENABLE_AUTH is true
-SUPABASE_JWT_SECRET=our-super-secret-jwt-token-with-at-least-32-characters-long
+
+## User auth with Supabase is required for any of the 3rd party integrations with auth to work.
+ENABLE_AUTH=false
+SUPABASE_URL=
+SUPABASE_SERVICE_ROLE_KEY=
+SUPABASE_JWT_SECRET=
+
+# For local development, you may need to set FRONTEND_BASE_URL for the OAuth flow for integrations to work.
+# FRONTEND_BASE_URL=http://localhost:3000
+
+## == INTEGRATION CREDENTIALS == ##
+# Each set of server side credentials is required for the corresponding 3rd party
+# integration to work.
+
+# For the OAuth callback URL, use <your_frontend_url>/auth/integrations/oauth_callback,
+# e.g. http://localhost:3000/auth/integrations/oauth_callback
+
+# GitHub OAuth App server credentials - https://github.com/settings/developers
+GITHUB_CLIENT_ID=
+GITHUB_CLIENT_SECRET=
 
 ## ===== OPTIONAL API KEYS ===== ##
 

autogpt_platform/backend/backend/blocks/__init__.py

@@ -1,4 +1,3 @@
-import glob
 import importlib
 import os
 import re
@@ -8,17 +7,17 @@
 
 # Dynamically load all modules under backend.blocks
 AVAILABLE_MODULES = []
-current_dir = os.path.dirname(__file__)
-modules = glob.glob(os.path.join(current_dir, "*.py"))
+current_dir = Path(__file__).parent
 modules = [
-    Path(f).stem
-    for f in modules
-    if os.path.isfile(f) and f.endswith(".py") and not f.endswith("__init__.py")
+    str(f.relative_to(current_dir))[:-3].replace(os.path.sep, ".")
+    for f in current_dir.rglob("*.py")
+    if f.is_file() and f.name != "__init__.py"
 ]
 for module in modules:
-    if not re.match("^[a-z_]+$", module):
+    if not re.match("^[a-z_.]+$", module):
         raise ValueError(
-            f"Block module {module} error: module name must be lowercase, separated by underscores, and contain only alphabet characters"
+            f"Block module {module} error: module name must be lowercase, "
+            "separated by underscores, and contain only alphabet characters"
         )
 
     importlib.import_module(f".{module}", package=__name__)

autogpt_platform/backend/backend/blocks/basic.py

@@ -57,7 +57,7 @@ def __init__(self):
             static_output=True,
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         yield "output", input_data.data or input_data.input
 
 
@@ -79,7 +79,7 @@ def __init__(self):
             test_output=("status", "printed"),
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         print(">>>>> Print: ", input_data.text)
         yield "status", "printed"
 
@@ -118,7 +118,7 @@ def __init__(self):
             categories={BlockCategory.BASIC},
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         obj = input_data.input
         key = input_data.key
 
@@ -200,7 +200,7 @@ def __init__(self):
             ui_type=BlockUIType.INPUT,
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         yield "result", input_data.value
 
 
@@ -283,7 +283,7 @@ def __init__(self):
             ui_type=BlockUIType.OUTPUT,
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         """
         Attempts to format the recorded_value using the fmt_string if provided.
         If formatting fails or no fmt_string is given, returns the original recorded_value.
@@ -343,7 +343,7 @@ def __init__(self):
             ],
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         try:
             # If no dictionary is provided, create a new one
             if input_data.dictionary is None:
@@ -414,7 +414,7 @@ def __init__(self):
             ],
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         try:
             # If no list is provided, create a new one
             if input_data.list is None:
@@ -455,5 +455,5 @@ def __init__(self):
             ui_type=BlockUIType.NOTE,
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         yield "output", input_data.text

autogpt_platform/backend/backend/blocks/block.py

@@ -31,7 +31,7 @@ def __init__(self):
             disabled=True,
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         code = input_data.code
 
         if search := re.search(r"class (\w+)\(Block\):", code):

autogpt_platform/backend/backend/blocks/branching.py

@@ -70,7 +70,7 @@ def __init__(self):
             ],
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         value1 = input_data.value1
         operator = input_data.operator
         value2 = input_data.value2

autogpt_platform/backend/backend/blocks/csv.py

@@ -40,7 +40,7 @@ def __init__(self):
             ],
         )
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         import csv
         from io import StringIO
 

autogpt_platform/backend/backend/blocks/discord.py

@@ -81,14 +81,14 @@ async def on_message(message):
 
         await client.start(token)
 
-    def run(self, input_data: "ReadDiscordMessagesBlock.Input") -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         while True:
             for output_name, output_value in self.__run(input_data):
                 yield output_name, output_value
             if not input_data.continuous_read:
                 break
 
-    def __run(self, input_data: "ReadDiscordMessagesBlock.Input") -> BlockOutput:
+    def __run(self, input_data: Input) -> BlockOutput:
         try:
             loop = asyncio.get_event_loop()
             future = self.run_bot(input_data.discord_bot_token.get_secret_value())
@@ -187,7 +187,7 @@ def chunk_message(self, message: str, limit: int = 2000) -> list:
         """Splits a message into chunks not exceeding the Discord limit."""
         return [message[i : i + limit] for i in range(0, len(message), limit)]
 
-    def run(self, input_data: "SendDiscordMessageBlock.Input") -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         try:
             loop = asyncio.get_event_loop()
             future = self.send_message(

autogpt_platform/backend/backend/blocks/email_block.py

@@ -88,7 +88,7 @@ def send_email(
         except Exception as e:
             return f"Failed to send email: {str(e)}"
 
-    def run(self, input_data: Input) -> BlockOutput:
+    def run(self, input_data: Input, **kwargs) -> BlockOutput:
         status = self.send_email(
             input_data.creds,
             input_data.to_email,

autogpt_platform/backend/backend/blocks/github/_auth.py

@@ -0,0 +1,54 @@
+from typing import Literal
+
+from autogpt_libs.supabase_integration_credentials_store.types import (
+    APIKeyCredentials,
+    OAuth2Credentials,
+)
+from pydantic import SecretStr
+
+from backend.data.model import CredentialsField, CredentialsMetaInput
+from backend.util.settings import Secrets
+
+secrets = Secrets()
+GITHUB_OAUTH_IS_CONFIGURED = bool(
+    secrets.github_client_id and secrets.github_client_secret
+)
+
+GithubCredentials = APIKeyCredentials | OAuth2Credentials
+GithubCredentialsInput = CredentialsMetaInput[
+    Literal["github"],
+    Literal["api_key", "oauth2"] if GITHUB_OAUTH_IS_CONFIGURED else Literal["api_key"],
+]
+
+
+def GithubCredentialsField(scope: str) -> GithubCredentialsInput:
+    """
+    Creates a GitHub credentials input on a block.
+
+    Params:
+        scope: The authorization scope needed for the block to work. ([list of available scopes](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/scopes-for-oauth-apps#available-scopes))
+    """  # noqa
+    return CredentialsField(
+        provider="github",
+        supported_credential_types=(
+            {"api_key", "oauth2"} if GITHUB_OAUTH_IS_CONFIGURED else {"api_key"}
+        ),
+        required_scopes={scope},
+        description="The GitHub integration can be used with OAuth, "
+        "or any API key with sufficient permissions for the blocks it is used on.",
+    )
+
+
+TEST_CREDENTIALS = APIKeyCredentials(
+    id="01234567-89ab-cdef-0123-456789abcdef",
+    provider="github",
+    api_key=SecretStr("mock-github-api-key"),
+    title="Mock GitHub API key",
+    expires_at=None,
+)
+TEST_CREDENTIALS_INPUT = {
+    "provider": TEST_CREDENTIALS.provider,
+    "id": TEST_CREDENTIALS.id,
+    "type": TEST_CREDENTIALS.type,
+    "title": TEST_CREDENTIALS.type,
+}