pipeline-backend manages all pipeline resources within Versatile Data
Pipeline (VDP) to streamline data from the
start component, through AI/Data/Application components and to the end
component.
In 💧 Instill VDP, a pipeline is a DAG (Directed Acyclic Graph) consisting of multiple components.
flowchart LR
s[Trigger] --> c1[OpenAI Component]
c1 --> c2[Stability AI Component]
c1 --> c3[MySQL Component]
c1 --> e[Response]
c2 --> e
A Component serves as an essential building block within a Pipeline.
See the component package documentation for more
details.
A pipeline recipe specifies how components are configured and how they are interconnected.
Recipes are defined in YAML language:
variable:
# pipeline input fields
output:
# pipeline output fields
component:
<component-id>:
type: <component-definition-id>
task: <task-id>
input:
# values for the input fields
condition: <condition> # conditional statement to execute or bypass the component
setup: <setup> # setup specification values required in AI, Data and Application componentsThe component development guide contains a full example recipe.
sequenceDiagram
participant u as User
participant gw as api-gateway
participant p as pipeline-backend
participant db as pipeline-db
u ->> gw: POST /users/<user>/pipelines
gw ->> p: forward
p ->> db: Store pipeline and its recipe
When a pipeline is triggered, the DAG will be computed in order to execute components in topological order.
sequenceDiagram
participant u as User
participant gw as api-gateway
participant p as pipeline-backend
participant db as pipeline-db
participant c as component
u ->> gw: POST /users/<user>/pipelines/<pipeline-id>/trigger
gw ->> p: forward
p ->> db: Get recipe
db ->> p: Recipe
loop over topological order of components
p->>c: ExecuteWithValidation
end
We welcome contributions from the community! Whether you're a developer, designer, writer, or user, there are multiple ways to contribute:
We foster a friendly and inclusive environment for issue reporting. Before creating an issue, check if it already exists. Use clear language and provide reproducible steps for bugs. Accurately tag the issue (bug, improvement, question, etc.).
Please refer to the Contributing Guidelines for more details. Your code-driven innovations are more than welcome!
We are committed to providing a respectful and welcoming atmosphere for all contributors. Please review our Code of Conduct to understand our standards.
We have implemented a streamlined Issues Triage Process aimed at swiftly categorizing new issues and pull requests (PRs), allowing us to take prompt and appropriate actions.
Head over to our Discussions for engaging conversations:
- General: Chat about anything related to our projects.
- Polls: Participate in community polls.
- Q&A: Seek help or ask questions; our community members and maintainers are here to assist.
- Show and Tell: Showcase projects you've created using our tools.
Alternatively, you can also join our vibrant Discord community and direct your queries to the #ask-for-help channel. We're dedicated to supporting you every step of the way.
See the LICENSE file for licensing information.