document the engine

packages/core/src/README.md

@@ -0,0 +1,103 @@
+# n8n Core Architecture
+
+This document provides a high-level overview of the n8n core architecture, with links to more detailed documentation for key subsystems.
+
+## System Architecture
+
+n8n's core package contains several critical subsystems that work together to provide the workflow execution engine, node loading, credential management, and other foundational features.
+
+```mermaid
+graph TD
+    A[n8n Core] --> B[Execution Engine]
+    A --> C[Nodes Loader]
+    A --> D[Binary Data]
+    A --> E[Encryption]
+    A --> F[Error Handling]
+    A --> G[Logging]
+```
+
+## Key Subsystems
+
+### Execution Engine
+
+The [Execution Engine](./execution-engine/README.md) is responsible for executing workflows, including:
+
+- Running full and partial workflow executions
+- Managing active workflows with triggers and pollers
+- Handling execution contexts for different node types
+- Process node-to-node data passing
+- Error handling and recovery
+
+For detailed information, see the [Execution Engine Documentation](./execution-engine/README.md).
+
+### Nodes Loader
+
+The [Nodes Loader](./nodes-loader/README.md) system handles loading, managing, and providing access to n8n nodes and credentials:
+
+- Loading nodes and credentials from various sources
+- Managing node and credential metadata
+- Lazy loading for improved performance
+- Isolated class loading
+
+For detailed information, see the [Nodes Loader Documentation](./nodes-loader/README.md).
+
+### Binary Data
+
+The Binary Data subsystem manages binary data handling throughout n8n:
+
+- File system and object storage management
+- Binary data streaming and processing
+- File type detection and handling
+- Data deduplication
+
+### Encryption
+
+The Encryption subsystem provides services for:
+
+- Credential encryption/decryption
+- Secure storage of sensitive data
+- Key management
+- Encryption scheme versioning
+
+### Error Handling
+
+The Error Handling subsystem provides:
+
+- Standardized error classes
+- Error reporting and collection
+- Workflow error recovery
+- User-friendly error messages
+
+### Logging
+
+The Logging subsystem handles:
+
+- Structured logging
+- Log level management
+- Log routing (console, file, etc.)
+- Sensitive data redaction
+
+## Design Principles
+
+The n8n core architecture adheres to several key design principles:
+
+1. **Separation of Concerns**: Each subsystem has clear responsibilities
+2. **Dependency Injection**: Services are loosely coupled through dependency injection
+3. **Extensibility**: Systems are designed to be extended with new functionality
+4. **Type Safety**: Extensive use of TypeScript interfaces and type checking
+5. **Performance Optimization**: Lazy loading, partial execution, and other optimizations
+6. **Error Resilience**: Comprehensive error handling at multiple levels
+
+## Implementation Patterns
+
+Throughout the codebase, you'll find consistent implementation patterns:
+
+- **Service Pattern**: Most subsystems are implemented as injectable services
+- **Factory Pattern**: Used for creating specialized contexts and instances
+- **Strategy Pattern**: Allows for different implementations of core functionality
+- **Decorator Pattern**: Used for adding behavior to classes
+- **Repository Pattern**: Used for data access abstraction
+
+## Getting Started with Development
+
+To contribute to n8n core, start by understanding the subsystem relevant to your task. The linked documentation provides detailed explanations of each subsystem's architecture and components.

packages/core/src/execution-engine/README.md

@@ -0,0 +1,191 @@
+# n8n Execution Engine Architecture
+
+This document provides a high-level overview of the n8n execution engine architecture and its components.
+
+## Core Components
+
+The execution engine consists of several key components that work together to execute workflows:
+
+```mermaid
+graph TD
+    WE[WorkflowExecute]
+    AW[ActiveWorkflows]
+    TP[TriggersAndPollers]
+    EC[Execution Contexts]
+    LH[Lifecycle Hooks]
+    PEU[Partial Execution Utils]
+    
+    AW -->|manages| TP
+    TP -->|triggers| WE
+    WE -->|uses| EC
+    WE -->|calls| LH
+    WE -->|optimizes with| PEU
+```
+
+### WorkflowExecute
+
+The central component responsible for executing workflows. It manages:
+- The node execution stack
+- Data passing between nodes
+- Error handling and retries
+- Partial workflow execution
+
+### ActiveWorkflows
+
+Manages currently active workflows that have trigger or polling nodes:
+- Activates and deactivates workflows
+- Sets up triggers and polling operations
+- Manages workflow state in memory
+
+### TriggersAndPollers
+
+Handles trigger and polling nodes:
+- Sets up event listeners for triggers
+- Manages scheduled polling operations
+- Processes incoming webhook requests
+- Initiates workflow executions
+
+### Execution Contexts
+
+Specialized contexts for different node types and execution scenarios:
+- NodeExecutionContext: Base abstract class
+- ExecuteContext: Regular node execution
+- WebhookContext: Webhook handling
+- TriggerContext: Trigger nodes
+- PollContext: Polling nodes
+- And several other specialized contexts
+
+### Execution Lifecycle Hooks
+
+Provides hooks for workflow and node execution events:
+- Before/after node execution
+- Before/after workflow execution
+- Supports features like monitoring and debugging
+
+### Partial Execution Utilities
+
+Optimizes workflow execution by only running necessary parts:
+- DirectedGraph: Graph representation of the workflow
+- Finding start nodes and subgraphs
+- Handling cyclic dependencies
+- Recreating execution state for partial runs
+
+## Execution Flow
+
+The high-level execution flow in n8n follows these steps:
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant AW as ActiveWorkflows
+    participant TP as TriggersAndPollers
+    participant WE as WorkflowExecute
+    participant EC as ExecutionContext
+    
+    Client->>AW: Activate workflow
+    AW->>TP: Setup triggers/pollers
+    
+    alt Regular Run
+        Client->>WE: Run workflow
+        WE->>WE: Execute nodes
+    else Triggered Run
+        Note over TP: Webhook/Event received
+        TP->>WE: Trigger execution
+        WE->>WE: Execute nodes
+    else Polling Run
+        Note over TP: Poll interval reached
+        TP->>TP: Poll for data
+        TP->>WE: Trigger if data found
+        WE->>WE: Execute nodes
+    end
+    
+    loop For each node
+        WE->>EC: Create execution context
+        EC->>EC: Resolve parameters
+        EC->>EC: Execute node logic
+        EC->>WE: Return results
+        WE->>WE: Process output
+    end
+    
+    WE->>Client: Return results
+```
+
+## Data Flow
+
+Data flows through the workflow execution as follows:
+
+```mermaid
+flowchart TD
+    A[Node A Output] -->|Connection| B[Node B Input]
+    B -->|Processing| C[Node B Output]
+    C -->|Connection| D[Node C Input]
+    
+    subgraph "Data Structure"
+        X[INodeExecutionData[][]]
+        X -->|"Outer Array"| Y["Different outputs (main, error)"]
+        Y -->|"Inner Array"| Z["Data items"]
+    end
+    
+    subgraph "Paired Items"
+        I[Input Items] -->|"Maps to"| O[Output Items]
+        O -->|"Tracked via"| P["pairedItem property"]
+    end
+```
+
+## Execution Modes
+
+n8n supports different execution modes:
+
+1. **Manual Execution**: Triggered manually by a user
+2. **Webhook Execution**: Triggered by an HTTP request
+3. **Polling Execution**: Triggered by periodic polling
+4. **Scheduled Execution**: Triggered at specific times
+
+## Partial Execution
+
+Partial execution is an optimization that re-executes only the necessary parts of a workflow:
+
+```mermaid
+flowchart TD
+    A[Workflow Change] --> B["Create DirectedGraph"]
+    B --> C["Find modified nodes"]
+    C --> D["Find dependent nodes"]
+    C --> E["Find execution path"]
+    D --> F["Determine start nodes"]
+    E --> F
+    F --> G["Create execution subgraph"]
+    G --> H["Handle cycles"]
+    H --> I["Clean run data"]
+    I --> J["Recreate execution stack"]
+    J --> K["Run partial workflow"]
+```
+
+## Error Handling
+
+The execution engine provides multiple error handling mechanisms:
+
+1. **Node-level error handling**:
+   - continueOnFail: Continue execution despite node failures
+   - retryOnFail: Retry node execution multiple times
+   - onError output: Route errors to specific outputs
+
+2. **Workflow-level error handling**:
+   - Error workflows: Separate workflows triggered on error
+   - Error reporting: Centralized error collection and reporting
+   - Error node: Handle errors within the workflow
+
+## Extensibility
+
+The architecture is designed to be extensible:
+- Specialized node execution contexts can be added
+- Additional execution lifecycle hooks can be registered
+- Custom error handlers can be implemented
+- Custom trigger mechanisms can be developed
+
+## Key Design Principles
+
+1. **Separation of concerns**: Each component has specific responsibilities
+2. **Dependency injection**: Components are loosely coupled
+3. **Extensibility**: New functionality can be added with minimal changes
+4. **Performance optimization**: Partial execution minimizes unnecessary work
+5. **Error resilience**: Comprehensive error handling at multiple levels

packages/core/src/execution-engine/active-workflows.ts

@@ -24,6 +24,39 @@ import type { IGetExecutePollFunctions, IGetExecuteTriggerFunctions } from './in
 import { ScheduledTaskManager } from './scheduled-task-manager';
 import { TriggersAndPollers } from './triggers-and-pollers';
 
+/**
+ * Service that manages workflows that are currently active in the system.
+ * An active workflow is one that has trigger or polling nodes enabled and is
+ * waiting for events to trigger execution.
+ *
+ * ### Responsibilities
+ *
+ * - Activating and deactivating workflows
+ * - Managing trigger and polling nodes for active workflows
+ * - Tracking workflow state and webhooks
+ * - Handling scheduled tasks for polling nodes
+ *
+ * ### Workflow Activation Flow
+ *
+ * ```mermaid
+ * sequenceDiagram
+ *   participant Client
+ *   participant AW as ActiveWorkflows
+ *   participant TAP as TriggersAndPollers
+ *   participant STM as ScheduledTaskManager
+ *
+ *   Client->>AW: add(workflow)
+ *   AW->>AW: Find trigger/poll nodes
+ *   AW->>TAP: For trigger nodes: setupTrigger()
+ *   AW->>STM: For poll nodes: registerScheduledTasks()
+ *   AW->>Client: Workflow activated
+ * ```
+ *
+ * ## Related Components
+ *
+ * - {@link TriggersAndPollers}: Handles the trigger and polling node execution
+ * - {@link ScheduledTaskManager}: Manages scheduled tasks for polling nodes
+ */
 @Service()
 export class ActiveWorkflows {
 	constructor(

packages/core/src/execution-engine/execution-lifecycle-hooks.ts

@@ -54,14 +54,25 @@ s	 */
 export type ExecutionLifecycleHookName = keyof ExecutionLifecyleHookHandlers;
 
 /**
- * Contains hooks that trigger at specific events in an execution's lifecycle. Every hook has an array of callbacks to run.
+ * Manages and executes hooks for workflow and node execution lifecycle events.
+ * These hooks allow external code to perform actions before or after
+ * specific points in the execution process.
  *
- * Common use cases include:
+ * ### Available Hooks
+ *
+ * - **nodeExecuteBefore**: Triggered before a node is executed
+ * - **nodeExecuteAfter**: Triggered after a node is executed
+ * - **workflowExecuteBefore**: Triggered before workflow execution starts
+ * - **workflowExecuteAfter**: Triggered after workflow execution completes
+ *
+ * ### These hooks are particularly useful for
+ * - Logging and observability
  * - Saving execution progress to database
  * - Pushing execution status updates to the frontend
  * - Recording workflow statistics
  * - Running external hooks for execution events
  * - Error and Cancellation handling and cleanup
+ * - Execution analytics
  *
  * @example
  * ```typescript
@@ -70,6 +81,29 @@ export type ExecutionLifecycleHookName = keyof ExecutionLifecyleHookHandlers;
  *  await saveToDatabase(executionId, fullRunData);
  *});
  * ```
+ *
+ * ### Hook Execution Flow
+ *
+ * ```mermaid
+ * sequenceDiagram
+ *     participant WE as WorkflowExecute
+ *     participant Hooks as ExecutionLifecycleHooks
+ *     participant Handlers as Hook Handlers
+ *
+ *     WE->>Hooks: workflowExecuteBefore()
+ *     Hooks->>Handlers: Call all registered handlers
+ *
+ *     loop For each node
+ *         WE->>Hooks: nodeExecuteBefore(nodeName)
+ *         Hooks->>Handlers: Call handlers
+ *         WE->>WE: Execute node
+ *         WE->>Hooks: nodeExecuteAfter(nodeName, data)
+ *         Hooks->>Handlers: Call handlers
+ *     end
+ *
+ *     WE->>Hooks: workflowExecuteAfter()
+ *     Hooks->>Handlers: Call all registered handlers
+ * ```
  */
 export class ExecutionLifecycleHooks {
 	readonly handlers: ExecutionLifecyleHookHandlers = {