diff --git a/packages/core/src/README.md b/packages/core/src/README.md new file mode 100644 index 0000000000000..fb3184b86bab1 --- /dev/null +++ b/packages/core/src/README.md @@ -0,0 +1,103 @@ +# n8n Core Architecture + +This document provides a high-level overview of the n8n core architecture, with links to more detailed documentation for key subsystems. + +## System Architecture + +n8n's core package contains several critical subsystems that work together to provide the workflow execution engine, node loading, credential management, and other foundational features. + +```mermaid +graph TD + A[n8n Core] --> B[Execution Engine] + A --> C[Nodes Loader] + A --> D[Binary Data] + A --> E[Encryption] + A --> F[Error Handling] + A --> G[Logging] +``` + +## Key Subsystems + +### Execution Engine + +The [Execution Engine](./execution-engine/README.md) is responsible for executing workflows, including: + +- Running full and partial workflow executions +- Managing active workflows with triggers and pollers +- Handling execution contexts for different node types +- Process node-to-node data passing +- Error handling and recovery + +For detailed information, see the [Execution Engine Documentation](./execution-engine/README.md). + +### Nodes Loader + +The [Nodes Loader](./nodes-loader/README.md) system handles loading, managing, and providing access to n8n nodes and credentials: + +- Loading nodes and credentials from various sources +- Managing node and credential metadata +- Lazy loading for improved performance +- Isolated class loading + +For detailed information, see the [Nodes Loader Documentation](./nodes-loader/README.md). + +### Binary Data + +The Binary Data subsystem manages binary data handling throughout n8n: + +- File system and object storage management +- Binary data streaming and processing +- File type detection and handling +- Data deduplication + +### Encryption + +The Encryption subsystem provides services for: + +- Credential encryption/decryption +- Secure storage of sensitive data +- Key management +- Encryption scheme versioning + +### Error Handling + +The Error Handling subsystem provides: + +- Standardized error classes +- Error reporting and collection +- Workflow error recovery +- User-friendly error messages + +### Logging + +The Logging subsystem handles: + +- Structured logging +- Log level management +- Log routing (console, file, etc.) +- Sensitive data redaction + +## Design Principles + +The n8n core architecture adheres to several key design principles: + +1. **Separation of Concerns**: Each subsystem has clear responsibilities +2. **Dependency Injection**: Services are loosely coupled through dependency injection +3. **Extensibility**: Systems are designed to be extended with new functionality +4. **Type Safety**: Extensive use of TypeScript interfaces and type checking +5. **Performance Optimization**: Lazy loading, partial execution, and other optimizations +6. **Error Resilience**: Comprehensive error handling at multiple levels + +## Implementation Patterns + +Throughout the codebase, you'll find consistent implementation patterns: + +- **Service Pattern**: Most subsystems are implemented as injectable services +- **Factory Pattern**: Used for creating specialized contexts and instances +- **Strategy Pattern**: Allows for different implementations of core functionality +- **Decorator Pattern**: Used for adding behavior to classes +- **Repository Pattern**: Used for data access abstraction + +## Getting Started with Development + +To contribute to n8n core, start by understanding the subsystem relevant to your task. The linked documentation provides detailed explanations of each subsystem's architecture and components. diff --git a/packages/core/src/execution-engine/README.md b/packages/core/src/execution-engine/README.md new file mode 100644 index 0000000000000..b822270a8662b --- /dev/null +++ b/packages/core/src/execution-engine/README.md @@ -0,0 +1,191 @@ +# n8n Execution Engine Architecture + +This document provides a high-level overview of the n8n execution engine architecture and its components. + +## Core Components + +The execution engine consists of several key components that work together to execute workflows: + +```mermaid +graph TD + WE[WorkflowExecute] + AW[ActiveWorkflows] + TP[TriggersAndPollers] + EC[Execution Contexts] + LH[Lifecycle Hooks] + PEU[Partial Execution Utils] + + AW -->|manages| TP + TP -->|triggers| WE + WE -->|uses| EC + WE -->|calls| LH + WE -->|optimizes with| PEU +``` + +### WorkflowExecute + +The central component responsible for executing workflows. It manages: +- The node execution stack +- Data passing between nodes +- Error handling and retries +- Partial workflow execution + +### ActiveWorkflows + +Manages currently active workflows that have trigger or polling nodes: +- Activates and deactivates workflows +- Sets up triggers and polling operations +- Manages workflow state in memory + +### TriggersAndPollers + +Handles trigger and polling nodes: +- Sets up event listeners for triggers +- Manages scheduled polling operations +- Processes incoming webhook requests +- Initiates workflow executions + +### Execution Contexts + +Specialized contexts for different node types and execution scenarios: +- NodeExecutionContext: Base abstract class +- ExecuteContext: Regular node execution +- WebhookContext: Webhook handling +- TriggerContext: Trigger nodes +- PollContext: Polling nodes +- And several other specialized contexts + +### Execution Lifecycle Hooks + +Provides hooks for workflow and node execution events: +- Before/after node execution +- Before/after workflow execution +- Supports features like monitoring and debugging + +### Partial Execution Utilities + +Optimizes workflow execution by only running necessary parts: +- DirectedGraph: Graph representation of the workflow +- Finding start nodes and subgraphs +- Handling cyclic dependencies +- Recreating execution state for partial runs + +## Execution Flow + +The high-level execution flow in n8n follows these steps: + +```mermaid +sequenceDiagram + participant Client + participant AW as ActiveWorkflows + participant TP as TriggersAndPollers + participant WE as WorkflowExecute + participant EC as ExecutionContext + + Client->>AW: Activate workflow + AW->>TP: Setup triggers/pollers + + alt Regular Run + Client->>WE: Run workflow + WE->>WE: Execute nodes + else Triggered Run + Note over TP: Webhook/Event received + TP->>WE: Trigger execution + WE->>WE: Execute nodes + else Polling Run + Note over TP: Poll interval reached + TP->>TP: Poll for data + TP->>WE: Trigger if data found + WE->>WE: Execute nodes + end + + loop For each node + WE->>EC: Create execution context + EC->>EC: Resolve parameters + EC->>EC: Execute node logic + EC->>WE: Return results + WE->>WE: Process output + end + + WE->>Client: Return results +``` + +## Data Flow + +Data flows through the workflow execution as follows: + +```mermaid +flowchart TD + A[Node A Output] -->|Connection| B[Node B Input] + B -->|Processing| C[Node B Output] + C -->|Connection| D[Node C Input] + + subgraph "Data Structure" + X[INodeExecutionData[][]] + X -->|"Outer Array"| Y["Different outputs (main, error)"] + Y -->|"Inner Array"| Z["Data items"] + end + + subgraph "Paired Items" + I[Input Items] -->|"Maps to"| O[Output Items] + O -->|"Tracked via"| P["pairedItem property"] + end +``` + +## Execution Modes + +n8n supports different execution modes: + +1. **Manual Execution**: Triggered manually by a user +2. **Webhook Execution**: Triggered by an HTTP request +3. **Polling Execution**: Triggered by periodic polling +4. **Scheduled Execution**: Triggered at specific times + +## Partial Execution + +Partial execution is an optimization that re-executes only the necessary parts of a workflow: + +```mermaid +flowchart TD + A[Workflow Change] --> B["Create DirectedGraph"] + B --> C["Find modified nodes"] + C --> D["Find dependent nodes"] + C --> E["Find execution path"] + D --> F["Determine start nodes"] + E --> F + F --> G["Create execution subgraph"] + G --> H["Handle cycles"] + H --> I["Clean run data"] + I --> J["Recreate execution stack"] + J --> K["Run partial workflow"] +``` + +## Error Handling + +The execution engine provides multiple error handling mechanisms: + +1. **Node-level error handling**: + - continueOnFail: Continue execution despite node failures + - retryOnFail: Retry node execution multiple times + - onError output: Route errors to specific outputs + +2. **Workflow-level error handling**: + - Error workflows: Separate workflows triggered on error + - Error reporting: Centralized error collection and reporting + - Error node: Handle errors within the workflow + +## Extensibility + +The architecture is designed to be extensible: +- Specialized node execution contexts can be added +- Additional execution lifecycle hooks can be registered +- Custom error handlers can be implemented +- Custom trigger mechanisms can be developed + +## Key Design Principles + +1. **Separation of concerns**: Each component has specific responsibilities +2. **Dependency injection**: Components are loosely coupled +3. **Extensibility**: New functionality can be added with minimal changes +4. **Performance optimization**: Partial execution minimizes unnecessary work +5. **Error resilience**: Comprehensive error handling at multiple levels \ No newline at end of file diff --git a/packages/core/src/execution-engine/active-workflows.ts b/packages/core/src/execution-engine/active-workflows.ts index 1c1b5e2d141fb..24b04c7d59592 100644 --- a/packages/core/src/execution-engine/active-workflows.ts +++ b/packages/core/src/execution-engine/active-workflows.ts @@ -24,6 +24,39 @@ import type { IGetExecutePollFunctions, IGetExecuteTriggerFunctions } from './in import { ScheduledTaskManager } from './scheduled-task-manager'; import { TriggersAndPollers } from './triggers-and-pollers'; +/** + * Service that manages workflows that are currently active in the system. + * An active workflow is one that has trigger or polling nodes enabled and is + * waiting for events to trigger execution. + * + * ### Responsibilities + * + * - Activating and deactivating workflows + * - Managing trigger and polling nodes for active workflows + * - Tracking workflow state and webhooks + * - Handling scheduled tasks for polling nodes + * + * ### Workflow Activation Flow + * + * ```mermaid + * sequenceDiagram + * participant Client + * participant AW as ActiveWorkflows + * participant TAP as TriggersAndPollers + * participant STM as ScheduledTaskManager + * + * Client->>AW: add(workflow) + * AW->>AW: Find trigger/poll nodes + * AW->>TAP: For trigger nodes: setupTrigger() + * AW->>STM: For poll nodes: registerScheduledTasks() + * AW->>Client: Workflow activated + * ``` + * + * ## Related Components + * + * - {@link TriggersAndPollers}: Handles the trigger and polling node execution + * - {@link ScheduledTaskManager}: Manages scheduled tasks for polling nodes + */ @Service() export class ActiveWorkflows { constructor( diff --git a/packages/core/src/execution-engine/execution-lifecycle-hooks.ts b/packages/core/src/execution-engine/execution-lifecycle-hooks.ts index 047d5a20756ec..2549e25a185b1 100644 --- a/packages/core/src/execution-engine/execution-lifecycle-hooks.ts +++ b/packages/core/src/execution-engine/execution-lifecycle-hooks.ts @@ -54,14 +54,25 @@ s */ export type ExecutionLifecycleHookName = keyof ExecutionLifecyleHookHandlers; /** - * Contains hooks that trigger at specific events in an execution's lifecycle. Every hook has an array of callbacks to run. + * Manages and executes hooks for workflow and node execution lifecycle events. + * These hooks allow external code to perform actions before or after + * specific points in the execution process. * - * Common use cases include: + * ### Available Hooks + * + * - **nodeExecuteBefore**: Triggered before a node is executed + * - **nodeExecuteAfter**: Triggered after a node is executed + * - **workflowExecuteBefore**: Triggered before workflow execution starts + * - **workflowExecuteAfter**: Triggered after workflow execution completes + * + * ### These hooks are particularly useful for + * - Logging and observability * - Saving execution progress to database * - Pushing execution status updates to the frontend * - Recording workflow statistics * - Running external hooks for execution events * - Error and Cancellation handling and cleanup + * - Execution analytics * * @example * ```typescript @@ -70,6 +81,29 @@ export type ExecutionLifecycleHookName = keyof ExecutionLifecyleHookHandlers; * await saveToDatabase(executionId, fullRunData); *}); * ``` + * + * ### Hook Execution Flow + * + * ```mermaid + * sequenceDiagram + * participant WE as WorkflowExecute + * participant Hooks as ExecutionLifecycleHooks + * participant Handlers as Hook Handlers + * + * WE->>Hooks: workflowExecuteBefore() + * Hooks->>Handlers: Call all registered handlers + * + * loop For each node + * WE->>Hooks: nodeExecuteBefore(nodeName) + * Hooks->>Handlers: Call handlers + * WE->>WE: Execute node + * WE->>Hooks: nodeExecuteAfter(nodeName, data) + * Hooks->>Handlers: Call handlers + * end + * + * WE->>Hooks: workflowExecuteAfter() + * Hooks->>Handlers: Call all registered handlers + * ``` */ export class ExecutionLifecycleHooks { readonly handlers: ExecutionLifecyleHookHandlers = { diff --git a/packages/core/src/execution-engine/index.ts b/packages/core/src/execution-engine/index.ts index bb8adfb34cd89..1792070a19f07 100644 --- a/packages/core/src/execution-engine/index.ts +++ b/packages/core/src/execution-engine/index.ts @@ -1,8 +1,31 @@ -export * from './active-workflows'; +/** + * # n8n Execution Engine + * + * This module contains the core components of n8n's workflow execution engine. + * The execution engine is responsible for running workflows, managing active workflows, + * handling triggers, and providing execution contexts for different node types. + * + * ## Key Components + * + * - {@link ActiveWorkflows}: Manages workflows that are currently active/running + * - {@link WorkflowExecute}: Core workflow execution logic + * - {@link TriggersAndPollers}: Handles trigger nodes and polling operations + * - {@link ExecutionLifecycleHooks}: Provides hooks for workflow execution lifecycle events + * - {@link RoutingNode}: Routes execution based on node type and implementation + * + * ## Execution Modes + * + * - **Full Execution**: Executes the entire workflow from start to finish + * - **Partial Execution**: Executes only parts of the workflow that need to be updated + * + * @module executionEngine + */ + +export { ActiveWorkflows } from './active-workflows'; export * from './interfaces'; export * from './routing-node'; export * from './node-execution-context'; export * from './partial-execution-utils'; export * from './node-execution-context/utils/execution-metadata'; -export * from './workflow-execute'; +export { WorkflowExecute } from './workflow-execute'; export { ExecutionLifecycleHooks } from './execution-lifecycle-hooks'; diff --git a/packages/core/src/execution-engine/node-execution-context/base-execute-context.ts b/packages/core/src/execution-engine/node-execution-context/base-execute-context.ts index f2d0aa8653524..9cc59f9ebb916 100644 --- a/packages/core/src/execution-engine/node-execution-context/base-execute-context.ts +++ b/packages/core/src/execution-engine/node-execution-context/base-execute-context.ts @@ -33,6 +33,27 @@ import { BinaryDataService } from '@/binary-data/binary-data.service'; import { NodeExecutionContext } from './node-execution-context'; +/** + * Base class for execution contexts that handle regular node execution. + * Extends NodeExecutionContext with execution-specific functionality. + * + * ### Responsibilities + * + * - Managing binary data through BinaryDataService + * - Providing wait/resume functionality for async operations + * - Accessing input data for node execution + * - Supporting sub-workflow execution + * - Handling execution cancellation + * + * ### Context Hierarchy + * + * This class serves as the base for: + * - ExecuteContext: Used for regular node execution + * - ExecuteSingleContext: Used for single-item node execution + * + * Both derived contexts add specialized functionality for different + * execution scenarios while sharing common base execution capabilities. + */ export class BaseExecuteContext extends NodeExecutionContext { protected readonly binaryDataService = Container.get(BinaryDataService); diff --git a/packages/core/src/execution-engine/node-execution-context/index.ts b/packages/core/src/execution-engine/node-execution-context/index.ts index dcac1576d1a94..437f9a11d8759 100644 --- a/packages/core/src/execution-engine/node-execution-context/index.ts +++ b/packages/core/src/execution-engine/node-execution-context/index.ts @@ -1,3 +1,40 @@ +/** + * # Node Execution Contexts + * + * This module provides specialized execution contexts for different node types and execution scenarios. + * Each context type provides the appropriate environment and utilities for its specific execution needs. + * + * ## Context Types + * + * - **NodeExecutionContext**: Base abstract class for all contexts + * - **BaseExecuteContext**: Foundation for regular execution contexts + * - **ExecuteContext**: Regular node execution + * - **ExecuteSingleContext**: Single-item node execution + * - **WebhookContext**: Webhook node execution with HTTP request/response handling + * - **TriggerContext**: Trigger node execution for starting workflows + * - **PollContext**: Polling node execution for periodically checking for data + * - **LoadOptionsContext**: Dynamic options loading for node parameters + * - **HookContext**: Pre/post execution hooks + * - **CredentialsTestContext**: Testing node credentials + * - **SupplyDataContext**: Supplying data to nodes + * + * ## Context Hierarchy + * + * ```mermaid + * classDiagram + * NodeExecutionContext <|-- BaseExecuteContext + * NodeExecutionContext <|-- WebhookContext + * NodeExecutionContext <|-- TriggerContext + * NodeExecutionContext <|-- PollContext + * NodeExecutionContext <|-- LoadOptionsContext + * NodeExecutionContext <|-- HookContext + * NodeExecutionContext <|-- CredentialsTestContext + * NodeExecutionContext <|-- SupplyDataContext + * BaseExecuteContext <|-- ExecuteContext + * BaseExecuteContext <|-- ExecuteSingleContext + * ``` + */ + export { CredentialTestContext } from './credentials-test-context'; export { ExecuteContext } from './execute-context'; export { ExecuteSingleContext } from './execute-single-context'; diff --git a/packages/core/src/execution-engine/node-execution-context/node-execution-context.ts b/packages/core/src/execution-engine/node-execution-context/node-execution-context.ts index 662e24d6b6f18..f541b6ce40756 100644 --- a/packages/core/src/execution-engine/node-execution-context/node-execution-context.ts +++ b/packages/core/src/execution-engine/node-execution-context/node-execution-context.ts @@ -39,6 +39,36 @@ import { extractValue } from './utils/extract-value'; import { getAdditionalKeys } from './utils/get-additional-keys'; import { validateValueAgainstSchema } from './utils/validate-value-against-schema'; +/** + * Base abstract class for all node execution contexts in n8n. + * Provides core functionality for accessing and manipulating node data during execution. + * + * ### Responsibilities + * + * - Resolving node parameters (including expressions) + * - Accessing and validating credential data + * - Providing workflow data proxies for expressions + * - Type validation and conversion + * + * ### Hierarchy + * + * ```mermaid + * classDiagram + * NodeExecutionContext <|-- BaseExecuteContext + * NodeExecutionContext <|-- WebhookContext + * NodeExecutionContext <|-- TriggerContext + * NodeExecutionContext <|-- PollContext + * NodeExecutionContext <|-- LoadOptionsContext + * NodeExecutionContext <|-- HookContext + * NodeExecutionContext <|-- CredentialsTestContext + * NodeExecutionContext <|-- SupplyDataContext + * BaseExecuteContext <|-- ExecuteContext + * BaseExecuteContext <|-- ExecuteSingleContext + * ``` + * + * This class serves as the foundation for all specialized contexts that handle + * different node execution scenarios (regular execution, triggers, webhooks, etc.) + */ export abstract class NodeExecutionContext implements Omit { protected readonly instanceSettings = Container.get(InstanceSettings); diff --git a/packages/core/src/execution-engine/node-execution-context/webhook-context.ts b/packages/core/src/execution-engine/node-execution-context/webhook-context.ts index 73f71647533f3..f2407e9b4e9ee 100644 --- a/packages/core/src/execution-engine/node-execution-context/webhook-context.ts +++ b/packages/core/src/execution-engine/node-execution-context/webhook-context.ts @@ -25,6 +25,40 @@ import { getRequestHelperFunctions } from './utils/request-helper-functions'; import { returnJsonArray } from './utils/return-json-array'; import { getNodeWebhookUrl } from './utils/webhook-helper-functions'; +/** + * Specialized execution context for webhook nodes. + * Handles HTTP request/response for webhook-triggered workflows. + * + * ### Responsibilities + * + * - Providing access to HTTP request data (body, headers, query params) + * - Managing HTTP response sending + * - Handling webhook URLs and test webhooks + * - Supporting file uploads and binary data processing + * + * ### Response Modes + * + * - **Immediate**: Responds as soon as the webhook node executes + * - **ResponseNode**: Waits to get a response from a specific Response-to-* node + * - **LastNode**: Waits for the entire workflow to complete + * + * ### Webhook Flow + * + * ```mermaid + * sequenceDiagram + * participant Client + * participant Server + * participant WC as WebhookContext + * participant WE as WorkflowExecute + * + * Client->>Server: HTTP Request to webhook URL + * Server->>WC: Create context with request/response + * WC->>WC: Process request data + * WC->>WE: Trigger workflow execution + * WE-->>WC: Execute workflow (may be async) + * WC->>Client: Send response (immediate or deferred) + * ``` + */ export class WebhookContext extends NodeExecutionContext implements IWebhookFunctions { readonly helpers: IWebhookFunctions['helpers']; diff --git a/packages/core/src/execution-engine/partial-execution-utils/index.ts b/packages/core/src/execution-engine/partial-execution-utils/index.ts index d363f52302a04..218b9f2c82bbd 100644 --- a/packages/core/src/execution-engine/partial-execution-utils/index.ts +++ b/packages/core/src/execution-engine/partial-execution-utils/index.ts @@ -1,3 +1,38 @@ +/** + * # Partial Execution Utilities + * + * This module provides utilities for partial workflow execution - a performance + * optimization that allows n8n to execute only the necessary parts of a workflow + * instead of re-executing the entire workflow. + * + * ## Key Components + * + * - **DirectedGraph**: Core data structure representing workflow as a directed graph + * - **findStartNodes**: Identifies which nodes to start execution from + * - **findSubgraph**: Creates a minimal graph between trigger and destination nodes + * - **handleCycles**: Manages cyclic dependencies in the workflow graph + * - **cleanRunData**: Prepares run data for partial execution + * - **recreateNodeExecutionStack**: Rebuilds execution stack for partial runs + * + * ## Partial Execution Process + * + * ```mermaid + * flowchart TD + * A[Start] --> B[Find Trigger] + * B --> C[Create Directed Graph] + * C --> D[Find Subgraph] + * D --> E[Find Start Nodes] + * E --> F[Handle Cycles] + * F --> G[Clean Run Data] + * G --> H[Recreate Execution Stack] + * H --> I[Execute Partial Workflow] + * ``` + * + * This system enables efficient workflow updates where only the affected parts + * of a workflow need to be re-executed, significantly improving performance + * for large workflows. + */ + export { DirectedGraph } from './directed-graph'; export { findTriggerForPartialExecution } from './find-trigger-for-partial-execution'; export { findStartNodes } from './find-start-nodes'; diff --git a/packages/core/src/execution-engine/routing-node.ts b/packages/core/src/execution-engine/routing-node.ts index e62bdfd34cc2a..f0a462d7de700 100644 --- a/packages/core/src/execution-engine/routing-node.ts +++ b/packages/core/src/execution-engine/routing-node.ts @@ -41,6 +41,32 @@ import url from 'node:url'; import { type ExecuteContext, ExecuteSingleContext } from './node-execution-context'; +/** + * A utility class that helps route execution based on the node type and implementation. + * This component is responsible for determining the right execution path for different + * node types and handling their specific execution requirements. + * + * ## Responsibilities + * + * - Routing execution to the correct node implementation + * - Handling declarative REST API nodes + * - Providing node configuration for dynamic nodes + * - Executing node business logic with appropriate context + * + * ## Execution Flow + * + * ```mermaid + * flowchart TD + * A[Node Execution] --> B{Node Type} + * B -->|Declarative API| C[executeDeclarativeApi] + * B -->|Regular Node| D[executeStandardNode] + * C --> E[Pre-process parameters] + * D --> F[Execute node method] + * E --> G[Execute API request] + * F --> H[Return results] + * G --> H + * ``` + */ export class RoutingNode { constructor( private readonly context: ExecuteContext, diff --git a/packages/core/src/execution-engine/ssh-clients-manager.ts b/packages/core/src/execution-engine/ssh-clients-manager.ts index a29c5facc7d55..646df9abc2a9e 100644 --- a/packages/core/src/execution-engine/ssh-clients-manager.ts +++ b/packages/core/src/execution-engine/ssh-clients-manager.ts @@ -3,6 +3,33 @@ import type { SSHCredentials } from 'n8n-workflow'; import { createHash } from 'node:crypto'; import { Client, type ConnectConfig } from 'ssh2'; +/** + * Service that manages SSH client connections used during workflow execution. + * Handles connection pooling, lifecycle management, and cleanup of SSH connections. + * + * ### Responsibilities + * + * - Creating and caching SSH client connections + * - Reusing existing connections based on credentials + * - Cleaning up stale connections + * - Ensuring proper shutdown of connections + * + * ### Connection Lifecycle + * + * ```mermaid + * flowchart TD + * A[getClient(credentials)] --> B{Connection exists?} + * B -->|Yes| C[Update lastUsed timestamp] + * B -->|No| D[Create new SSH connection] + * C --> E[Return existing client] + * D --> F[Cache client] + * F --> E + * G[Periodic cleanup] --> H[Check all connections] + * H --> I{Connection stale?} + * I -->|Yes| J[Close connection] + * I -->|No| H + * ``` + */ @Service() export class SSHClientsManager { readonly clients = new Map(); diff --git a/packages/core/src/execution-engine/triggers-and-pollers.ts b/packages/core/src/execution-engine/triggers-and-pollers.ts index 9c90327149879..e0b8dfa857ec1 100644 --- a/packages/core/src/execution-engine/triggers-and-pollers.ts +++ b/packages/core/src/execution-engine/triggers-and-pollers.ts @@ -17,6 +17,38 @@ import assert from 'node:assert'; import type { IGetExecuteTriggerFunctions } from './interfaces'; +/** + * Service responsible for running trigger and polling nodes in workflows. + * These nodes are the entry points that start workflow executions based on + * external events or scheduled polling. + * + * ### Responsibilities + * + * - Running trigger nodes and setting up their event listeners + * - Executing polling functions at scheduled intervals + * - Handling webhook responses and triggering workflows + * - Managing the transition from trigger/poll events to workflow execution + * + * ### Trigger/Poll Flow + * + * ```mermaid + * flowchart TD + * A[Workflow Activated] --> B{Node Type} + * B -->|Trigger| C[runTrigger()] + * B -->|Poll| D[runPoll()] + * C --> E[Node sets up listeners] + * D --> F[Node polls at interval] + * E --> G[Event received] + * F --> H[Data found] + * G --> I[Trigger workflow execution] + * H --> I + * ``` + * + * ## Related Components + * + * - {@link ActiveWorkflows}: Manages which workflows have active triggers/pollers + * - {@link WorkflowExecute}: Executes workflows when triggers/polls provide data + */ @Service() export class TriggersAndPollers { /** diff --git a/packages/core/src/execution-engine/workflow-execute.ts b/packages/core/src/execution-engine/workflow-execute.ts index 9f777b68ae2da..1f79f2a1498f6 100644 --- a/packages/core/src/execution-engine/workflow-execute.ts +++ b/packages/core/src/execution-engine/workflow-execute.ts @@ -70,6 +70,45 @@ import { import { RoutingNode } from './routing-node'; import { TriggersAndPollers } from './triggers-and-pollers'; +/** + * The central engine responsible for executing workflows in n8n. This class handles: + * + * - Full workflow execution + * - Partial workflow execution (executing only parts of a workflow) + * - Node execution and inter-node data passing + * - Error handling and workflow retries + * - Execution cancellation + * + * ### Execution Architecture + * + * ```mermaid + * flowchart TD + * A[workflow.run()] --> B[Initialize Execution Stack] + * B --> C[Process Next Node from Stack] + * C --> D[Prepare Input Data w/ PairedItems] + * D --> E[Execute Node via runNode()] + * E --> F[Process Node Output] + * F --> G[Add Connected Nodes to Stack] + * G --> H{Stack Empty?} + * H -->|No| C + * H -->|Yes| I[Finalize Execution] + * ``` + * + * ### Partial Execution Mode + * + * Optimizes performance by executing only the necessary parts of a workflow. + * The system: + * 1. Creates a directed graph representation of the workflow + * 2. Identifies only the nodes that need to be re-executed + * 3. Handles cycles and complex data dependencies + * 4. Re-uses results from previously executed nodes when possible + * + * ### Data Passing + * + * Data flows between nodes using the "pairedItem" concept, which tracks which + * output items correspond to which input items, maintaining data lineage through + * the workflow execution. + */ export class WorkflowExecute { private status: ExecutionStatus = 'new'; diff --git a/packages/core/src/nodes-loader/README.md b/packages/core/src/nodes-loader/README.md new file mode 100644 index 0000000000000..f9391b9f3c52d --- /dev/null +++ b/packages/core/src/nodes-loader/README.md @@ -0,0 +1,216 @@ +# n8n Nodes Loader Architecture + +This document provides a comprehensive overview of the n8n nodes-loader system, which is responsible for loading, managing, and providing access to n8n nodes and credentials. + +## Core Components + +The nodes-loader system consists of several key components that work together to load node and credential types from various sources: + +```mermaid +graph TD + A[DirectoryLoader] --> B[CustomDirectoryLoader] + A --> C[PackageDirectoryLoader] + C --> D[LazyPackageDirectoryLoader] + E[loadClassInIsolation] --> A + F[Constants & Types] --> A +``` + +### DirectoryLoader (Base Class) + +The `DirectoryLoader` is an abstract base class that provides the core functionality for loading nodes and credentials from a directory. It handles: + +- Path resolution and file management +- Class loading and instantiation +- Type registration and validation +- Icon URL handling and normalization + +### CustomDirectoryLoader + +The `CustomDirectoryLoader` extends the `DirectoryLoader` to load custom nodes from a user-defined directory: + +- Used for loading user-created nodes and credentials +- Has special handling for "CUSTOM" package naming +- Scans directory for node and credential files + +### PackageDirectoryLoader + +The `PackageDirectoryLoader` loads nodes and credentials from an installed n8n package: + +- Reads package.json to identify node and credential files +- Validates package structure +- Loads node and credential classes +- Manages relationships between nodes and credentials + +### LazyPackageDirectoryLoader + +The `LazyPackageDirectoryLoader` extends the `PackageDirectoryLoader` to provide lazy loading capabilities: + +- Loads node and credential metadata without loading full implementations +- Improves startup performance by deferring full loading +- Supports including/excluding specific nodes +- Falls back to full loading when necessary + +### loadClassInIsolation + +The `loadClassInIsolation` utility loads node and credential classes in an isolated VM context: + +- Prevents cross-contamination between different node versions +- Improves stability by isolating node code +- Uses Node.js VM module for sandboxing + +## Data Structures + +### Node and Credential Types + +```mermaid +classDiagram + class INodeType { + +description: INodeTypeDescription + +execute(params): Promise~INodeExecutionData[][]~ + } + + class ICredentialType { + +name: string + +displayName: string + +properties: INodeProperties[] + +authenticate?(): Promise~any~ + } + + class DirectoryLoader { + +nodeTypes: Record~string, LoadedNodeType~ + +credentialTypes: Record~string, LoadedCredentialType~ + +known: KnownNodes + +types: TypesData + } + + INodeType -- DirectoryLoader + ICredentialType -- DirectoryLoader +``` + +The loaders manage two primary types of components: + +1. **Node Types**: Classes that implement `INodeType` interface + - Define node appearance, parameters, and behavior + - Include the `execute` method that contains the node's business logic + +2. **Credential Types**: Classes that implement `ICredentialType` interface + - Define credential fields and validation + - May include authentication methods + - Can be linked to specific nodes + +### Loading Process + +```mermaid +sequenceDiagram + participant App + participant Loader as DirectoryLoader + participant Files as FileSystem + participant VM as NodeVM + + App->>Loader: loadAll() + Loader->>Files: Read package.json + Files-->>Loader: Package metadata + Loader->>Files: Find node/credential files + Files-->>Loader: File paths + + loop For each file + Loader->>VM: loadClassInIsolation(filePath) + VM-->>Loader: Node/Credential instance + Loader->>Loader: Process instance (icons, metadata) + Loader->>Loader: Store in nodeTypes/credentialTypes + end + + Loader->>Loader: Build node/credential relationships + Loader-->>App: Loading complete +``` + +## Special Features + +### Lazy Loading + +The lazy loading system improves startup performance by: + +1. Loading only metadata (types, names, descriptions) initially +2. Deferring the loading of actual implementation code +3. Loading specific nodes on-demand when needed + +```mermaid +flowchart TD + A[Start] --> B{Is lazy loading enabled?} + B -->|Yes| C[Load metadata only] + B -->|No| D[Load all implementations] + C --> E{Node requested?} + E -->|Yes| F[Load specific node implementation] + E -->|No| G[Continue with metadata only] +``` + +### Node-Credential Relationships + +The system tracks relationships between nodes and credentials: + +- Which credentials are used by which nodes +- Which nodes are supported by which credentials +- Inheritance relationships between credentials + +This information is used to: +- Optimize node and credential loading +- Provide better user experience when configuring nodes +- Enable proper HTTP request handling + +## Usage Patterns + +### Loading All Nodes + +```typescript +// Create loader instance +const loader = new PackageDirectoryLoader('/path/to/nodes-package'); + +// Load all nodes and credentials +await loader.loadAll(); + +// Access loaded types +const nodeTypes = loader.nodeTypes; +const credentialTypes = loader.credentialTypes; +``` + +### Lazy Loading + +```typescript +// Create lazy loader instance +const loader = new LazyPackageDirectoryLoader('/path/to/nodes-package'); + +// Load metadata only +await loader.loadAll(); + +// Access specific node (will load implementation if needed) +const nodeType = loader.getNode('my-node'); +``` + +### Including/Excluding Nodes + +```typescript +// Create loader with include/exclude filters +const loader = new LazyPackageDirectoryLoader( + '/path/to/nodes-package', + ['package.excludedNode'], // Nodes to exclude + ['package.includedNode'] // Nodes to include +); + +await loader.loadAll(); +``` + +## Best Practices + +1. **Use LazyPackageDirectoryLoader** for better performance in most cases +2. **Reset loaders** when package directories change +3. **Handle errors gracefully** from node loading +4. **Use the nodesByCredential** information for credential management +5. **Respect package scopes** when filtering nodes + +## Implementation Details + +- The system uses Node.js VM module for isolation +- Path resolution is normalized across platforms +- Icon paths are transformed for web server accessibility +- Code handles package.json parsing errors gracefully +- Type registries ensure uniqueness of node and credential types \ No newline at end of file