diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 52ccad4a5..f1aadf607 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -25,49 +25,47 @@ In VDP, a **pipeline** is a DAG (Directed Acyclic Graph) consisting of multiple ```mermaid flowchart LR - s[Trigger] --> c1[OpenAI Connector] - c1 --> c2[Stability AI Connector] - c1 --> c3[MySQL Connector] + s[Trigger] --> c1[OpenAI Component] + c1 --> c2[Stability AI Component] + c1 --> c3[MySQL Component] c1 --> e[Response] c2 --> e ``` ### Component -There are different types of component: -- **connector** - - Queries, processes or transmits the ingested data to a service or app. - - Users need to configure their connectors (e.g. by providing an API token to - a remote service). -- **operator** - - Performs data injection and manipulation. -- **iterator** - - Takes an array and executes an operation (defined by a set of nested - components) on each of its elements. -- **trigger / response** - - These special components provide an input / output interface to pipeline - triggers. - -#### Connector - -- **Connectors** are used by the pipeline to interact with an external service. - They are defined and initialized in the [connector](../pkg/connector) package. -- In order to set up a connector, you may need to introduce its **connection** - details in the connector properties. - - In order to prevent private keys from being unintentionally leaked when +There are different types of component: **AI**, **Data**, **Application**, **Operator** and **Iterator**. + +> **Note:** +> - For **AI**, **Data**, **Application** components, they are used by the pipeline to interact with an external service, you may need to introduce its **connection** details in the component connection properties. +> - In order to prevent private keys from being unintentionally leaked when sharing a pipeline, the connection properties only take reference to a **secret** (e.g. `${secrets.my-secret}`). - - You can create secrets from the console settings or through an [API +> - You can create secrets from the console settings or through an [API call](https://openapi.instill.tech/reference/pipelinepublicservice_createusersecret). +#### AI + +**AI** components play a crucial role in transforming unstructured data into formats that are easy to interpret and analyze, thereby facilitating the extraction of valuable insights. These components integrate with AI models from various providers, whether it's the primary Instill Model or those from third-party AI vendors. They are defined and initialized in the [ai](../ai) package. + +#### Data + +**Data** components play a crucial role in establishing connections with remote data sources, such as IoT devices (e.g., IP cameras), cloud storage services (e.g., GCP Cloud Storage, AWS S3), data warehouses, or vector databases (e.g., Pinecone). These connectors act as the bridge between VDP and various external data sources. Their primary function is to enable seamless data exchange, enhancing Instill VDP's capability to work with diverse data sources effectively. They are defined and initialized in the [data](../data) package. + +#### Application + +**Application** components are used to seamlessly integrate various 3rd-party application services. They are defined and initialized in the [application](../application) package. + #### Operator -- **Operators** perform data transformations inside the pipeline. They are defined - and initialized in the [operator](../pkg/operator) package. +**Operator** components perform data transformations inside the pipeline. They are defined + and initialized in the [operator](../operator) package. + +#### Iterator + +**Iterator** takes an array and executes an operation (defined by a set of nested components) +on each of its elements. -The key difference between `connector` and `operator` is that the former will -connect to an external service, so it's **I/O bound** while the latter is **CPU -bound**. Connectors don't process but transfer data. ### Recipe @@ -235,18 +233,18 @@ $ docker exec pipeline-backend go run ./cmd/main ```sh $ cd $WORKSPACE/component -$ mkdir -p pkg/operator/hello/v0 && cd $_ +$ mkdir -p operator/hello/v0 && cd $_ ``` -Components are isolated in their own packages under `pkg/connector` or -`pkg/operator`. The package is versioned so, in case a breaking change needs to +Components are isolated in their own packages under `ai`, `data`, `application` or +`operator`. The package is versioned so, in case a breaking change needs to be introduced (e.g. supporting a new major version in a vendor API), existing -pipelines using the previous version of the connector can keep being triggered. +pipelines using the previous version of the component can keep being triggered. At the end of this guide, this will be the structure of the package: ``` -pkg/operator/hello/v0 +operator/hello/v0 ├──assets │ └──hello.svg ├──config @@ -278,7 +276,7 @@ Create a `config` directory and add the files `definition.json` and "title": "Hello", "uid": "e05d3d71-779c-45f8-904d-e90a050ca3b2", "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/hello/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/operator/hello/v0", "description": "'Hello, world' operator used as a template for adding components", "release_stage": "RELEASE_STAGE_ALPHA" } @@ -293,7 +291,7 @@ This file defines the component properties: should be written in imperative tense. - `spec` contains the parameters required to configure the component and that are independent from its tasks. E.g., the API token of a vendor. In general, - only connectors need such parameters. + only AI, data or application components need such parameters. - `available_tasks` defines the tasks the component can perform. - When a component is created in a pipeline, one of the tasks has to be selected, i.e., a configured component can only execute one task. @@ -389,7 +387,7 @@ of a pipeline when configured to use this operator. ### Implement the component interfaces Pipeline communicates with components through the `IComponent`, `IConnector`, -`IOperator` and `IExecution` interfaces, defined in the [`base`](../pkg/base) +`IOperator` and `IExecution` interfaces, defined in the [`base`](../base) package. This package also defines base implementations for these interfaces, so the `hello` component will only need to override the following methods: - `CreateExecution(vars map[string]any, task string) (*ExecutionWrapper, error)` @@ -400,7 +398,7 @@ the `hello` component will only need to override the following methods: important function in the component. All the data manipulation will take place here. -Paste the following code into a `main.go` file in `pkg/operator/hello/v0`: +Paste the following code into a `main.go` file in `operator/hello/v0`: ```go package hello @@ -413,7 +411,7 @@ import ( "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) const ( @@ -590,31 +588,31 @@ func TestOperator_CreateExecution(t *testing.T) { The last step before being able to use the component in VDP is loading the `hello` operator. This is done in the `Init` function in -[`pkg/operator/main.go`](../pkg/operator/main.go): +[`component.go`](../component.go): ```go package operator import ( // ... - "github.com/instill-ai/component/pkg/operator/hello/v0" + "github.com/instill-ai/component/operator/hello/v0" ) // ... -func Init(logger *zap.Logger) *OperatorStore { +func Init(logger *zap.Logger) *Store { baseOp := base.BaseOperator{Logger: logger} once.Do(func() { - opStore = &OperatorStore{ + store = &Store{ operatorUIDMap: map[uuid.UUID]*operator{}, operatorIDMap: map[string]*operator{}, } // ... - opStore.Import(hello.Init(baseOp)) + store.ImportOperator(hello.Init(baseOp)) }) - return opStore + return store } ``` @@ -693,7 +691,7 @@ contain useful descriptions. The information described in `definition.json` and `tasks.json` is enough to understand how a component should be used. `compogen` is a tool that parses the component configuration and builds a `README.mdx` file document displaying its information in a human-readable way. To generate the -document, just add the following line on top of `pkg/operator/hello/v0/main.go`: +document, just add the following line on top of `operator/hello/v0/main.go`: ```go //go:generate compogen readme --operator ./config ./README.mdx @@ -717,7 +715,7 @@ its version should change following the Semantic Versioning guidelines. or a new input field with a default value. - Major versions are intended for backwards-incompatible changes. - At this point, since there might be pipelines using the previous version, a - new package MUST be created. E.g., `operator/pkg/json/v0` -> `operator/pkg/json/v1`. + new package MUST be created. E.g., `operator/json/v0` -> `operator/json/v1`. - Build and pre-release labels are discouraged, as components are shipped as part of Instill VDP and they aren't likely to need such fine-grained version control. diff --git a/pkg/connector/archetypeai/v0/README.mdx b/ai/archetypeai/v0/README.mdx similarity index 95% rename from pkg/connector/archetypeai/v0/README.mdx rename to ai/archetypeai/v0/README.mdx index 87186918f..bebe92b1e 100644 --- a/pkg/connector/archetypeai/v0/README.mdx +++ b/ai/archetypeai/v0/README.mdx @@ -18,7 +18,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/archetypeai/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/archetypeai/v0/config/definition.json). ## Connection diff --git a/pkg/connector/archetypeai/v0/assets/archetype-ai.svg b/ai/archetypeai/v0/assets/archetype-ai.svg similarity index 100% rename from pkg/connector/archetypeai/v0/assets/archetype-ai.svg rename to ai/archetypeai/v0/assets/archetype-ai.svg diff --git a/pkg/connector/archetypeai/v0/client.go b/ai/archetypeai/v0/client.go similarity index 90% rename from pkg/connector/archetypeai/v0/client.go rename to ai/archetypeai/v0/client.go index d528c8b0d..12e2c7e32 100644 --- a/pkg/connector/archetypeai/v0/client.go +++ b/ai/archetypeai/v0/client.go @@ -1,7 +1,7 @@ package archetypeai import ( - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/internal/util/httpclient" "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" ) diff --git a/pkg/connector/archetypeai/v0/config/definition.json b/ai/archetypeai/v0/config/definition.json similarity index 97% rename from pkg/connector/archetypeai/v0/config/definition.json rename to ai/archetypeai/v0/config/definition.json index 1e21687f6..1184ac20b 100644 --- a/pkg/connector/archetypeai/v0/config/definition.json +++ b/ai/archetypeai/v0/config/definition.json @@ -47,6 +47,6 @@ "vendor": "Archetype AI", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/archetypeai/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/ai/archetypeai/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/archetypeai/v0/config/tasks.json b/ai/archetypeai/v0/config/tasks.json similarity index 100% rename from pkg/connector/archetypeai/v0/config/tasks.json rename to ai/archetypeai/v0/config/tasks.json diff --git a/pkg/connector/archetypeai/v0/connector_test.go b/ai/archetypeai/v0/connector_test.go similarity index 98% rename from pkg/connector/archetypeai/v0/connector_test.go rename to ai/archetypeai/v0/connector_test.go index 3aba38673..ec72f22b2 100644 --- a/pkg/connector/archetypeai/v0/connector_test.go +++ b/ai/archetypeai/v0/connector_test.go @@ -9,8 +9,8 @@ import ( "testing" qt "github.com/frankban/quicktest" - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/base" + "github.com/instill-ai/component/internal/util/httpclient" "github.com/instill-ai/x/errmsg" "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" diff --git a/pkg/connector/archetypeai/v0/main.go b/ai/archetypeai/v0/main.go similarity index 97% rename from pkg/connector/archetypeai/v0/main.go rename to ai/archetypeai/v0/main.go index e827a91b0..363330bc9 100644 --- a/pkg/connector/archetypeai/v0/main.go +++ b/ai/archetypeai/v0/main.go @@ -12,9 +12,9 @@ import ( "google.golang.org/protobuf/types/known/structpb" "github.com/gofrs/uuid" - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/util" - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/base" + "github.com/instill-ai/component/internal/util" + "github.com/instill-ai/component/internal/util/httpclient" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/archetypeai/v0/structs.go b/ai/archetypeai/v0/structs.go similarity index 100% rename from pkg/connector/archetypeai/v0/structs.go rename to ai/archetypeai/v0/structs.go diff --git a/pkg/connector/huggingface/v0/README.mdx b/ai/huggingface/v0/README.mdx similarity index 99% rename from pkg/connector/huggingface/v0/README.mdx rename to ai/huggingface/v0/README.mdx index ea2f8f8a4..60f620f19 100644 --- a/pkg/connector/huggingface/v0/README.mdx +++ b/ai/huggingface/v0/README.mdx @@ -32,7 +32,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/huggingface/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/huggingface/v0/config/definition.json). ## Connection diff --git a/pkg/connector/huggingface/v0/assets/hugging-face.svg b/ai/huggingface/v0/assets/hugging-face.svg similarity index 100% rename from pkg/connector/huggingface/v0/assets/hugging-face.svg rename to ai/huggingface/v0/assets/hugging-face.svg diff --git a/pkg/connector/huggingface/v0/client.go b/ai/huggingface/v0/client.go similarity index 94% rename from pkg/connector/huggingface/v0/client.go rename to ai/huggingface/v0/client.go index e23fa51c9..bece2d8ae 100644 --- a/pkg/connector/huggingface/v0/client.go +++ b/ai/huggingface/v0/client.go @@ -6,7 +6,7 @@ import ( "strings" "github.com/go-resty/resty/v2" - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/internal/util/httpclient" "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" ) diff --git a/pkg/connector/huggingface/v0/config/definition.json b/ai/huggingface/v0/config/definition.json similarity index 98% rename from pkg/connector/huggingface/v0/config/definition.json rename to ai/huggingface/v0/config/definition.json index 9e6ae0d21..417cd7216 100644 --- a/pkg/connector/huggingface/v0/config/definition.json +++ b/ai/huggingface/v0/config/definition.json @@ -93,6 +93,6 @@ "vendor": "Hugging Face", "vendor_attributes": {}, "version": "0.1.1", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/huggingface/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/ai/huggingface/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/huggingface/v0/config/tasks.json b/ai/huggingface/v0/config/tasks.json similarity index 100% rename from pkg/connector/huggingface/v0/config/tasks.json rename to ai/huggingface/v0/config/tasks.json diff --git a/pkg/connector/huggingface/v0/connector_test.go b/ai/huggingface/v0/connector_test.go similarity index 98% rename from pkg/connector/huggingface/v0/connector_test.go rename to ai/huggingface/v0/connector_test.go index 9acd3d6a0..ef3435ce6 100644 --- a/pkg/connector/huggingface/v0/connector_test.go +++ b/ai/huggingface/v0/connector_test.go @@ -13,8 +13,8 @@ import ( "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/base" + "github.com/instill-ai/component/internal/util/httpclient" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/huggingface/v0/main.go b/ai/huggingface/v0/main.go similarity index 99% rename from pkg/connector/huggingface/v0/main.go rename to ai/huggingface/v0/main.go index 9c48d0e0f..b764a4fdc 100644 --- a/pkg/connector/huggingface/v0/main.go +++ b/ai/huggingface/v0/main.go @@ -12,7 +12,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/huggingface/v0/structs.go b/ai/huggingface/v0/structs.go similarity index 100% rename from pkg/connector/huggingface/v0/structs.go rename to ai/huggingface/v0/structs.go diff --git a/pkg/connector/instill/v0/README.mdx b/ai/instill/v0/README.mdx similarity index 99% rename from pkg/connector/instill/v0/README.mdx rename to ai/instill/v0/README.mdx index 1e833ff4f..163e518ff 100644 --- a/pkg/connector/instill/v0/README.mdx +++ b/ai/instill/v0/README.mdx @@ -26,7 +26,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/instill/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/instill/v0/config/definition.json). ## Supported Tasks diff --git a/pkg/connector/instill/v0/assets/instill-model.svg b/ai/instill/v0/assets/instill-model.svg similarity index 100% rename from pkg/connector/instill/v0/assets/instill-model.svg rename to ai/instill/v0/assets/instill-model.svg diff --git a/pkg/connector/instill/v0/client.go b/ai/instill/v0/client.go similarity index 100% rename from pkg/connector/instill/v0/client.go rename to ai/instill/v0/client.go diff --git a/pkg/connector/instill/v0/config/definition.json b/ai/instill/v0/config/definition.json similarity index 96% rename from pkg/connector/instill/v0/config/definition.json rename to ai/instill/v0/config/definition.json index c375661e2..c4654f0a0 100644 --- a/pkg/connector/instill/v0/config/definition.json +++ b/ai/instill/v0/config/definition.json @@ -26,6 +26,6 @@ "vendor": "Instill", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/instill/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/ai/instill/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/instill/v0/config/tasks.json b/ai/instill/v0/config/tasks.json similarity index 100% rename from pkg/connector/instill/v0/config/tasks.json rename to ai/instill/v0/config/tasks.json diff --git a/pkg/connector/instill/v0/image_classification.go b/ai/instill/v0/image_classification.go similarity index 98% rename from pkg/connector/instill/v0/image_classification.go rename to ai/instill/v0/image_classification.go index 76c5cd728..74df83176 100644 --- a/pkg/connector/instill/v0/image_classification.go +++ b/ai/instill/v0/image_classification.go @@ -8,7 +8,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha" ) diff --git a/pkg/connector/instill/v0/image_to_image.go b/ai/instill/v0/image_to_image.go similarity index 98% rename from pkg/connector/instill/v0/image_to_image.go rename to ai/instill/v0/image_to_image.go index 33ba1e549..fc36a6a2b 100644 --- a/pkg/connector/instill/v0/image_to_image.go +++ b/ai/instill/v0/image_to_image.go @@ -8,7 +8,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha" ) diff --git a/pkg/connector/instill/v0/instance_segmentation.go b/ai/instill/v0/instance_segmentation.go similarity index 98% rename from pkg/connector/instill/v0/instance_segmentation.go rename to ai/instill/v0/instance_segmentation.go index 83b17118b..52e04ba5e 100644 --- a/pkg/connector/instill/v0/instance_segmentation.go +++ b/ai/instill/v0/instance_segmentation.go @@ -8,7 +8,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha" ) diff --git a/pkg/connector/instill/v0/keypoint_detection.go b/ai/instill/v0/keypoint_detection.go similarity index 98% rename from pkg/connector/instill/v0/keypoint_detection.go rename to ai/instill/v0/keypoint_detection.go index 30be7546a..26fda96bd 100644 --- a/pkg/connector/instill/v0/keypoint_detection.go +++ b/ai/instill/v0/keypoint_detection.go @@ -8,7 +8,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha" ) diff --git a/pkg/connector/instill/v0/llm_utils.go b/ai/instill/v0/llm_utils.go similarity index 98% rename from pkg/connector/instill/v0/llm_utils.go rename to ai/instill/v0/llm_utils.go index afaf7d10b..ea7e4185d 100644 --- a/pkg/connector/instill/v0/llm_utils.go +++ b/ai/instill/v0/llm_utils.go @@ -3,7 +3,7 @@ package instill import ( "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha" ) diff --git a/pkg/connector/instill/v0/main.go b/ai/instill/v0/main.go similarity index 99% rename from pkg/connector/instill/v0/main.go rename to ai/instill/v0/main.go index 8fd21f205..e21e33178 100644 --- a/pkg/connector/instill/v0/main.go +++ b/ai/instill/v0/main.go @@ -13,7 +13,7 @@ import ( "google.golang.org/protobuf/proto" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" commonPB "github.com/instill-ai/protogen-go/common/task/v1alpha" mgmtPB "github.com/instill-ai/protogen-go/core/mgmt/v1beta" diff --git a/pkg/connector/instill/v0/object_detection.go b/ai/instill/v0/object_detection.go similarity index 98% rename from pkg/connector/instill/v0/object_detection.go rename to ai/instill/v0/object_detection.go index a7984d8dd..e144878d5 100644 --- a/pkg/connector/instill/v0/object_detection.go +++ b/ai/instill/v0/object_detection.go @@ -8,7 +8,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha" ) diff --git a/pkg/connector/instill/v0/ocr.go b/ai/instill/v0/ocr.go similarity index 100% rename from pkg/connector/instill/v0/ocr.go rename to ai/instill/v0/ocr.go diff --git a/pkg/connector/instill/v0/semantic_segmentation.go b/ai/instill/v0/semantic_segmentation.go similarity index 98% rename from pkg/connector/instill/v0/semantic_segmentation.go rename to ai/instill/v0/semantic_segmentation.go index 1239dd66b..5c4f03153 100644 --- a/pkg/connector/instill/v0/semantic_segmentation.go +++ b/ai/instill/v0/semantic_segmentation.go @@ -8,7 +8,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha" ) diff --git a/pkg/connector/instill/v0/text_generation.go b/ai/instill/v0/text_generation.go similarity index 100% rename from pkg/connector/instill/v0/text_generation.go rename to ai/instill/v0/text_generation.go diff --git a/pkg/connector/instill/v0/text_generation_chat.go b/ai/instill/v0/text_generation_chat.go similarity index 100% rename from pkg/connector/instill/v0/text_generation_chat.go rename to ai/instill/v0/text_generation_chat.go diff --git a/pkg/connector/instill/v0/text_to_image.go b/ai/instill/v0/text_to_image.go similarity index 98% rename from pkg/connector/instill/v0/text_to_image.go rename to ai/instill/v0/text_to_image.go index 0d22732e5..b29a47279 100644 --- a/pkg/connector/instill/v0/text_to_image.go +++ b/ai/instill/v0/text_to_image.go @@ -8,7 +8,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha" ) diff --git a/pkg/connector/instill/v0/unspecified.go b/ai/instill/v0/unspecified.go similarity index 100% rename from pkg/connector/instill/v0/unspecified.go rename to ai/instill/v0/unspecified.go diff --git a/pkg/connector/instill/v0/visual_question_answering.go b/ai/instill/v0/visual_question_answering.go similarity index 100% rename from pkg/connector/instill/v0/visual_question_answering.go rename to ai/instill/v0/visual_question_answering.go diff --git a/pkg/connector/integration_test.go b/ai/integration_test.go similarity index 93% rename from pkg/connector/integration_test.go rename to ai/integration_test.go index f1c73186f..da3a3b6d2 100644 --- a/pkg/connector/integration_test.go +++ b/ai/integration_test.go @@ -1,7 +1,7 @@ //go:build integration // +build integration -package pkg +package ai import ( "testing" @@ -11,8 +11,8 @@ import ( "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/openai/v0" + "github.com/instill-ai/component/ai/openai/v0" + "github.com/instill-ai/component/base" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/openai/v0/README.mdx b/ai/openai/v0/README.mdx similarity index 98% rename from pkg/connector/openai/v0/README.mdx rename to ai/openai/v0/README.mdx index 19361a4b0..61ab569b8 100644 --- a/pkg/connector/openai/v0/README.mdx +++ b/ai/openai/v0/README.mdx @@ -20,7 +20,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/openai/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/openai/v0/config/definition.json). ## Connection diff --git a/pkg/connector/openai/v0/assets/openai.svg b/ai/openai/v0/assets/openai.svg similarity index 100% rename from pkg/connector/openai/v0/assets/openai.svg rename to ai/openai/v0/assets/openai.svg diff --git a/pkg/connector/openai/v0/audio_transcriptions.go b/ai/openai/v0/audio_transcriptions.go similarity index 96% rename from pkg/connector/openai/v0/audio_transcriptions.go rename to ai/openai/v0/audio_transcriptions.go index 391748f9b..fb3ab6dfc 100644 --- a/pkg/connector/openai/v0/audio_transcriptions.go +++ b/ai/openai/v0/audio_transcriptions.go @@ -5,7 +5,7 @@ import ( "fmt" "mime/multipart" - "github.com/instill-ai/component/pkg/connector/util" + "github.com/instill-ai/component/internal/util" ) const ( diff --git a/pkg/connector/openai/v0/client.go b/ai/openai/v0/client.go similarity index 95% rename from pkg/connector/openai/v0/client.go rename to ai/openai/v0/client.go index c8b40b2f1..53ef63b1b 100644 --- a/pkg/connector/openai/v0/client.go +++ b/ai/openai/v0/client.go @@ -1,7 +1,7 @@ package openai import ( - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/internal/util/httpclient" "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" ) diff --git a/pkg/connector/openai/v0/config/definition.json b/ai/openai/v0/config/definition.json similarity index 98% rename from pkg/connector/openai/v0/config/definition.json rename to ai/openai/v0/config/definition.json index 577f8f495..a7d3cc28e 100644 --- a/pkg/connector/openai/v0/config/definition.json +++ b/ai/openai/v0/config/definition.json @@ -61,6 +61,6 @@ "vendor": "OpenAI", "vendor_attributes": {}, "version": "0.1.1", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/openai/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/ai/openai/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/openai/v0/config/openai.json b/ai/openai/v0/config/openai.json similarity index 100% rename from pkg/connector/openai/v0/config/openai.json rename to ai/openai/v0/config/openai.json diff --git a/pkg/connector/openai/v0/config/tasks.json b/ai/openai/v0/config/tasks.json similarity index 100% rename from pkg/connector/openai/v0/config/tasks.json rename to ai/openai/v0/config/tasks.json diff --git a/pkg/connector/openai/v0/connector_test.go b/ai/openai/v0/connector_test.go similarity index 98% rename from pkg/connector/openai/v0/connector_test.go rename to ai/openai/v0/connector_test.go index dd917cd15..8f4a7f430 100644 --- a/pkg/connector/openai/v0/connector_test.go +++ b/ai/openai/v0/connector_test.go @@ -13,9 +13,9 @@ import ( "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/util/httpclient" - "github.com/instill-ai/component/pkg/mock" + "github.com/instill-ai/component/base" + "github.com/instill-ai/component/internal/mock" + "github.com/instill-ai/component/internal/util/httpclient" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/openai/v0/list_models.go b/ai/openai/v0/list_models.go similarity index 100% rename from pkg/connector/openai/v0/list_models.go rename to ai/openai/v0/list_models.go diff --git a/pkg/connector/openai/v0/main.go b/ai/openai/v0/main.go similarity index 99% rename from pkg/connector/openai/v0/main.go rename to ai/openai/v0/main.go index d0db37d09..b9f276885 100644 --- a/pkg/connector/openai/v0/main.go +++ b/ai/openai/v0/main.go @@ -13,7 +13,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/openai/v0/text_embeddings.go b/ai/openai/v0/text_embeddings.go similarity index 100% rename from pkg/connector/openai/v0/text_embeddings.go rename to ai/openai/v0/text_embeddings.go diff --git a/pkg/connector/openai/v0/text_generation.go b/ai/openai/v0/text_generation.go similarity index 100% rename from pkg/connector/openai/v0/text_generation.go rename to ai/openai/v0/text_generation.go diff --git a/pkg/connector/openai/v0/text_to_image.go b/ai/openai/v0/text_to_image.go similarity index 100% rename from pkg/connector/openai/v0/text_to_image.go rename to ai/openai/v0/text_to_image.go diff --git a/pkg/connector/openai/v0/text_to_speech.go b/ai/openai/v0/text_to_speech.go similarity index 100% rename from pkg/connector/openai/v0/text_to_speech.go rename to ai/openai/v0/text_to_speech.go diff --git a/pkg/connector/stabilityai/v0/README.mdx b/ai/stabilityai/v0/README.mdx similarity index 98% rename from pkg/connector/stabilityai/v0/README.mdx rename to ai/stabilityai/v0/README.mdx index 37cd25190..d1b5dca62 100644 --- a/pkg/connector/stabilityai/v0/README.mdx +++ b/ai/stabilityai/v0/README.mdx @@ -17,7 +17,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/stabilityai/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/stabilityai/v0/config/definition.json). ## Connection diff --git a/pkg/connector/stabilityai/v0/assets/stability-ai.svg b/ai/stabilityai/v0/assets/stability-ai.svg similarity index 100% rename from pkg/connector/stabilityai/v0/assets/stability-ai.svg rename to ai/stabilityai/v0/assets/stability-ai.svg diff --git a/pkg/connector/stabilityai/v0/client.go b/ai/stabilityai/v0/client.go similarity index 94% rename from pkg/connector/stabilityai/v0/client.go rename to ai/stabilityai/v0/client.go index c1c16a03a..e3b69b3bd 100644 --- a/pkg/connector/stabilityai/v0/client.go +++ b/ai/stabilityai/v0/client.go @@ -1,7 +1,7 @@ package stabilityai import ( - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/internal/util/httpclient" "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" ) diff --git a/pkg/connector/stabilityai/v0/config/definition.json b/ai/stabilityai/v0/config/definition.json similarity index 97% rename from pkg/connector/stabilityai/v0/config/definition.json rename to ai/stabilityai/v0/config/definition.json index e2f94a522..4a136d55c 100644 --- a/pkg/connector/stabilityai/v0/config/definition.json +++ b/ai/stabilityai/v0/config/definition.json @@ -47,6 +47,6 @@ "vendor": "Stability AI", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/stabilityai/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/ai/stabilityai/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/stabilityai/v0/config/stabilityai.json b/ai/stabilityai/v0/config/stabilityai.json similarity index 100% rename from pkg/connector/stabilityai/v0/config/stabilityai.json rename to ai/stabilityai/v0/config/stabilityai.json diff --git a/pkg/connector/stabilityai/v0/config/tasks.json b/ai/stabilityai/v0/config/tasks.json similarity index 100% rename from pkg/connector/stabilityai/v0/config/tasks.json rename to ai/stabilityai/v0/config/tasks.json diff --git a/pkg/connector/stabilityai/v0/connector_test.go b/ai/stabilityai/v0/connector_test.go similarity index 98% rename from pkg/connector/stabilityai/v0/connector_test.go rename to ai/stabilityai/v0/connector_test.go index f95568f35..f46fa2058 100644 --- a/pkg/connector/stabilityai/v0/connector_test.go +++ b/ai/stabilityai/v0/connector_test.go @@ -12,8 +12,8 @@ import ( "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/base" + "github.com/instill-ai/component/internal/util/httpclient" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/stabilityai/v0/image_to_image.go b/ai/stabilityai/v0/image_to_image.go similarity index 98% rename from pkg/connector/stabilityai/v0/image_to_image.go rename to ai/stabilityai/v0/image_to_image.go index 98dc3da12..a9679d47b 100644 --- a/pkg/connector/stabilityai/v0/image_to_image.go +++ b/ai/stabilityai/v0/image_to_image.go @@ -5,8 +5,8 @@ import ( "fmt" "mime/multipart" - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/util" + "github.com/instill-ai/component/base" + "github.com/instill-ai/component/internal/util" "google.golang.org/protobuf/types/known/structpb" ) diff --git a/pkg/connector/stabilityai/v0/list_engines.go b/ai/stabilityai/v0/list_engines.go similarity index 100% rename from pkg/connector/stabilityai/v0/list_engines.go rename to ai/stabilityai/v0/list_engines.go diff --git a/pkg/connector/stabilityai/v0/main.go b/ai/stabilityai/v0/main.go similarity index 99% rename from pkg/connector/stabilityai/v0/main.go rename to ai/stabilityai/v0/main.go index 778549f3b..881333d5a 100644 --- a/pkg/connector/stabilityai/v0/main.go +++ b/ai/stabilityai/v0/main.go @@ -9,7 +9,7 @@ import ( "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/stabilityai/v0/text_to_image.go b/ai/stabilityai/v0/text_to_image.go similarity index 98% rename from pkg/connector/stabilityai/v0/text_to_image.go rename to ai/stabilityai/v0/text_to_image.go index b2b87dfae..020f4a676 100644 --- a/pkg/connector/stabilityai/v0/text_to_image.go +++ b/ai/stabilityai/v0/text_to_image.go @@ -3,7 +3,7 @@ package stabilityai import ( "fmt" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" "google.golang.org/protobuf/types/known/structpb" ) diff --git a/pkg/connector/googlesearch/v0/README.mdx b/application/googlesearch/v0/README.mdx similarity index 95% rename from pkg/connector/googlesearch/v0/README.mdx rename to application/googlesearch/v0/README.mdx index e86885123..975a5247e 100644 --- a/pkg/connector/googlesearch/v0/README.mdx +++ b/application/googlesearch/v0/README.mdx @@ -16,7 +16,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/googlesearch/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/googlesearch/v0/config/definition.json). ## Connection diff --git a/pkg/connector/googlesearch/v0/assets/google-search.svg b/application/googlesearch/v0/assets/google-search.svg similarity index 100% rename from pkg/connector/googlesearch/v0/assets/google-search.svg rename to application/googlesearch/v0/assets/google-search.svg diff --git a/pkg/connector/googlesearch/v0/config/definition.json b/application/googlesearch/v0/config/definition.json similarity index 98% rename from pkg/connector/googlesearch/v0/config/definition.json rename to application/googlesearch/v0/config/definition.json index 8357116e3..ee9054f6b 100644 --- a/pkg/connector/googlesearch/v0/config/definition.json +++ b/application/googlesearch/v0/config/definition.json @@ -54,6 +54,6 @@ "vendor": "Google", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/googlesearch/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/application/googlesearch/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/googlesearch/v0/config/tasks.json b/application/googlesearch/v0/config/tasks.json similarity index 100% rename from pkg/connector/googlesearch/v0/config/tasks.json rename to application/googlesearch/v0/config/tasks.json diff --git a/pkg/connector/googlesearch/v0/main.go b/application/googlesearch/v0/main.go similarity index 98% rename from pkg/connector/googlesearch/v0/main.go rename to application/googlesearch/v0/main.go index 3a18d07e3..ab0b95cce 100644 --- a/pkg/connector/googlesearch/v0/main.go +++ b/application/googlesearch/v0/main.go @@ -12,7 +12,7 @@ import ( "google.golang.org/api/option" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) const ( diff --git a/pkg/connector/googlesearch/v0/search.go b/application/googlesearch/v0/search.go similarity index 98% rename from pkg/connector/googlesearch/v0/search.go rename to application/googlesearch/v0/search.go index 630c1d4b2..559c5ebd5 100644 --- a/pkg/connector/googlesearch/v0/search.go +++ b/application/googlesearch/v0/search.go @@ -8,7 +8,7 @@ import ( "github.com/PuerkitoBio/goquery" "google.golang.org/api/customsearch/v1" - "github.com/instill-ai/component/pkg/connector/util" + "github.com/instill-ai/component/internal/util" ) const ( diff --git a/pkg/connector/numbers/v0/README.mdx b/application/numbers/v0/README.mdx similarity index 95% rename from pkg/connector/numbers/v0/README.mdx rename to application/numbers/v0/README.mdx index f3be5bffb..2170dc40b 100644 --- a/pkg/connector/numbers/v0/README.mdx +++ b/application/numbers/v0/README.mdx @@ -16,7 +16,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/numbers/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/numbers/v0/config/definition.json). ## Connection diff --git a/pkg/connector/numbers/v0/assets/numbers.svg b/application/numbers/v0/assets/numbers.svg similarity index 100% rename from pkg/connector/numbers/v0/assets/numbers.svg rename to application/numbers/v0/assets/numbers.svg diff --git a/pkg/connector/numbers/v0/config/definition.json b/application/numbers/v0/config/definition.json similarity index 98% rename from pkg/connector/numbers/v0/config/definition.json rename to application/numbers/v0/config/definition.json index 4f526778e..b27942bab 100644 --- a/pkg/connector/numbers/v0/config/definition.json +++ b/application/numbers/v0/config/definition.json @@ -45,6 +45,6 @@ "vendor": "Numbers Protocol", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/numbers/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/application/numbers/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/numbers/v0/config/tasks.json b/application/numbers/v0/config/tasks.json similarity index 100% rename from pkg/connector/numbers/v0/config/tasks.json rename to application/numbers/v0/config/tasks.json diff --git a/pkg/connector/numbers/v0/main.go b/application/numbers/v0/main.go similarity index 99% rename from pkg/connector/numbers/v0/main.go rename to application/numbers/v0/main.go index 14e8bc66f..6e7b316ef 100644 --- a/pkg/connector/numbers/v0/main.go +++ b/application/numbers/v0/main.go @@ -18,7 +18,7 @@ import ( "github.com/gofrs/uuid" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) const urlRegisterAsset = "https://api.numbersprotocol.io/api/v3/assets/" diff --git a/pkg/connector/redis/v0/README.mdx b/application/redis/v0/README.mdx similarity index 97% rename from pkg/connector/redis/v0/README.mdx rename to application/redis/v0/README.mdx index 126369b44..54862287a 100644 --- a/pkg/connector/redis/v0/README.mdx +++ b/application/redis/v0/README.mdx @@ -18,7 +18,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/redis/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/redis/v0/config/definition.json). ## Connection diff --git a/pkg/connector/redis/v0/assets/redis.svg b/application/redis/v0/assets/redis.svg similarity index 100% rename from pkg/connector/redis/v0/assets/redis.svg rename to application/redis/v0/assets/redis.svg diff --git a/pkg/connector/redis/v0/chat_history.go b/application/redis/v0/chat_history.go similarity index 100% rename from pkg/connector/redis/v0/chat_history.go rename to application/redis/v0/chat_history.go diff --git a/pkg/connector/redis/v0/client.go b/application/redis/v0/client.go similarity index 98% rename from pkg/connector/redis/v0/client.go rename to application/redis/v0/client.go index 95cedf460..355c89e86 100644 --- a/pkg/connector/redis/v0/client.go +++ b/application/redis/v0/client.go @@ -5,7 +5,7 @@ import ( "crypto/x509" "fmt" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" goredis "github.com/redis/go-redis/v9" "google.golang.org/protobuf/types/known/structpb" ) diff --git a/pkg/connector/redis/v0/config/definition.json b/application/redis/v0/config/definition.json similarity index 99% rename from pkg/connector/redis/v0/config/definition.json rename to application/redis/v0/config/definition.json index fd6423276..693ab165a 100644 --- a/pkg/connector/redis/v0/config/definition.json +++ b/application/redis/v0/config/definition.json @@ -208,6 +208,6 @@ "vendor": "Redis Labs", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/redis/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/application/redis/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/redis/v0/config/tasks.json b/application/redis/v0/config/tasks.json similarity index 100% rename from pkg/connector/redis/v0/config/tasks.json rename to application/redis/v0/config/tasks.json diff --git a/pkg/connector/redis/v0/main.go b/application/redis/v0/main.go similarity index 98% rename from pkg/connector/redis/v0/main.go rename to application/redis/v0/main.go index 494824587..28098d584 100644 --- a/pkg/connector/redis/v0/main.go +++ b/application/redis/v0/main.go @@ -9,7 +9,7 @@ import ( "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) const ( diff --git a/pkg/connector/restapi/v0/README.mdx b/application/restapi/v0/README.mdx similarity index 98% rename from pkg/connector/restapi/v0/README.mdx rename to application/restapi/v0/README.mdx index 3457d8c27..96e7cdeb5 100644 --- a/pkg/connector/restapi/v0/README.mdx +++ b/application/restapi/v0/README.mdx @@ -22,7 +22,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/restapi/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/restapi/v0/config/definition.json). ## Connection diff --git a/pkg/connector/restapi/v0/assets/restapi.svg b/application/restapi/v0/assets/restapi.svg similarity index 100% rename from pkg/connector/restapi/v0/assets/restapi.svg rename to application/restapi/v0/assets/restapi.svg diff --git a/pkg/connector/restapi/v0/auth.go b/application/restapi/v0/auth.go similarity index 96% rename from pkg/connector/restapi/v0/auth.go rename to application/restapi/v0/auth.go index 7d6ecc8ab..1b67fc659 100644 --- a/pkg/connector/restapi/v0/auth.go +++ b/application/restapi/v0/auth.go @@ -3,7 +3,7 @@ package restapi import ( "fmt" - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/internal/util/httpclient" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/restapi/v0/client.go b/application/restapi/v0/client.go similarity index 91% rename from pkg/connector/restapi/v0/client.go rename to application/restapi/v0/client.go index 264522021..401829dd2 100644 --- a/pkg/connector/restapi/v0/client.go +++ b/application/restapi/v0/client.go @@ -1,7 +1,7 @@ package restapi import ( - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/internal/util/httpclient" "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" ) diff --git a/pkg/connector/restapi/v0/config/definition.json b/application/restapi/v0/config/definition.json similarity index 99% rename from pkg/connector/restapi/v0/config/definition.json rename to application/restapi/v0/config/definition.json index d3b88850e..d58a6e8db 100644 --- a/pkg/connector/restapi/v0/config/definition.json +++ b/application/restapi/v0/config/definition.json @@ -197,6 +197,6 @@ "vendor": "", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/restapi/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/application/restapi/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/restapi/v0/config/tasks.json b/application/restapi/v0/config/tasks.json similarity index 100% rename from pkg/connector/restapi/v0/config/tasks.json rename to application/restapi/v0/config/tasks.json diff --git a/pkg/connector/restapi/v0/connector_test.go b/application/restapi/v0/connector_test.go similarity index 97% rename from pkg/connector/restapi/v0/connector_test.go rename to application/restapi/v0/connector_test.go index c14ea9a29..9d3fafe6a 100644 --- a/pkg/connector/restapi/v0/connector_test.go +++ b/application/restapi/v0/connector_test.go @@ -13,8 +13,8 @@ import ( "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/base" + "github.com/instill-ai/component/internal/util/httpclient" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/restapi/v0/main.go b/application/restapi/v0/main.go similarity index 99% rename from pkg/connector/restapi/v0/main.go rename to application/restapi/v0/main.go index 12886b81e..d6f63f733 100644 --- a/pkg/connector/restapi/v0/main.go +++ b/application/restapi/v0/main.go @@ -10,7 +10,7 @@ import ( "net/http" "sync" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" "github.com/instill-ai/x/errmsg" "google.golang.org/protobuf/proto" "google.golang.org/protobuf/types/known/structpb" diff --git a/pkg/connector/slack/v0/README.mdx b/application/slack/v0/README.mdx similarity index 95% rename from pkg/connector/slack/v0/README.mdx rename to application/slack/v0/README.mdx index 9f2dca30a..30b4aba4c 100644 --- a/pkg/connector/slack/v0/README.mdx +++ b/application/slack/v0/README.mdx @@ -17,7 +17,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/slack/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/slack/v0/config/definition.json). ## Connection diff --git a/pkg/connector/slack/v0/apiFunctions.go b/application/slack/v0/apiFunctions.go similarity index 100% rename from pkg/connector/slack/v0/apiFunctions.go rename to application/slack/v0/apiFunctions.go diff --git a/pkg/connector/slack/v0/assets/slack.svg b/application/slack/v0/assets/slack.svg similarity index 100% rename from pkg/connector/slack/v0/assets/slack.svg rename to application/slack/v0/assets/slack.svg diff --git a/pkg/connector/slack/v0/client.go b/application/slack/v0/client.go similarity index 100% rename from pkg/connector/slack/v0/client.go rename to application/slack/v0/client.go diff --git a/pkg/connector/slack/v0/config/definition.json b/application/slack/v0/config/definition.json similarity index 98% rename from pkg/connector/slack/v0/config/definition.json rename to application/slack/v0/config/definition.json index 6f13ecf89..d526309ca 100644 --- a/pkg/connector/slack/v0/config/definition.json +++ b/application/slack/v0/config/definition.json @@ -46,6 +46,6 @@ "vendor": "Slack", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/slack/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/application/slack/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/application/slack/v0/config/tasks.json b/application/slack/v0/config/tasks.json new file mode 100644 index 000000000..e0b0ac627 --- /dev/null +++ b/application/slack/v0/config/tasks.json @@ -0,0 +1,239 @@ +{ + "TASK_READ_MESSAGE": { + "instillShortDescription": "Get the latest message since specific date", + "input": { + "description": "Please input the channel name and the date that we want to start to read", + "instillUIOrder": 0, + "properties": { + "channel_name": { + "description": "A channel name display in Slack", + "instillAcceptFormats": [ + "string" + ], + "instillUIMultiline": true, + "instillUIOrder": 0, + "instillUpstreamTypes": [ + "value", + "reference", + "template" + ], + "title": "Channel Name", + "type": "string" + }, + "start_to_read_date": { + "description": "earliest date in all read messages", + "instillAcceptFormats": [ + "string" + ], + "instillUIMultiline": true, + "instillUIOrder": 1, + "instillUpstreamTypes": [ + "value", + "reference", + "template" + ], + "title": "Start to read date", + "type": "string" + }, + "is_public_channel": { + "description": "Whether all members can read the channel", + "instillAcceptFormats": [ + "boolean" + ], + "instillUIMultiline": true, + "instillUIOrder": 2, + "instillUpstreamTypes": [ + "value", + "reference", + "template" + ], + "title": "Public channel", + "type": "boolean" + } + }, + "required": [ + "channel_name" + ], + "title": "Input", + "type": "object" + }, + "output": { + "description": "All messages in Slack channel", + "instillUIOrder": 0, + "properties": { + "conversations": { + "description": "An array of conversations with thread messages", + "instillUIOrder": 0, + "title": "Conversations", + "type": "array", + "items": { + "title": "conversation details", + "type": "object", + "properties": { + "user_id": { + "description": "unique id from Slack", + "instillFormat": "string", + "instillUIOrder": 1, + "title": "User UID", + "type": "string" + }, + "message": { + "description": "message to start a conversation", + "instillFormat": "string", + "instillUIOrder": 2, + "title": "Start Conversation Message", + "type": "string" + }, + "start_date": { + "description": "when a conversation starts", + "instillFormat": "string", + "instillUIOrder": 3, + "required": [], + "title": "Start Date", + "type": "string" + }, + "last_date": { + "description": "Date of the last message", + "instillFormat": "string", + "instillUIOrder": 4, + "required": [], + "title": "Last Date", + "type": "string" + }, + "thread_reply_messages": { + "description": "replies in a conversation", + "instillFormat": "array", + "instillUIOrder": 5, + "title": "Replied messages", + "type": "array", + "items": { + "title": "relied details", + "type": "object", + "properties": { + "user_id": { + "description": "unique id from Slack", + "instillFormat": "string", + "instillUIOrder": 1, + "title": "User UID", + "type": "string" + }, + "datetime": { + "description": "replied datetime", + "instillFormat": "string", + "instillUIOrder": 2, + "title": "Replied Time", + "type": "string" + }, + "message": { + "description": "message to reply a conversation", + "instillFormat": "string", + "instillUIOrder": 3, + "title": "Replied Message", + "type": "string" + } + }, + "required": [ + "user_id", + "datetime", + "message" + ] + } + } + }, + "required": [ + "user_id", + "message", + "start_date" + ] + } + } + }, + "required": [ + "conversations" + ], + "title": "Output", + "type": "object" + } + }, + "TASK_WRITE_MESSAGE": { + "instillShortDescription": "send message to a specific channel", + "title": "Send Message", + "input": { + "description": "Input", + "instillUIOrder": 0, + "properties": { + "channel_name": { + "description": "A channel name display in Slack", + "instillAcceptFormats": [ + "string" + ], + "instillUIMultiline": true, + "instillUIOrder": 0, + "instillUpstreamTypes": [ + "value", + "reference", + "template" + ], + "title": "Channel Name", + "type": "string" + }, + "message": { + "description": "message to be sent to the target channel", + "instillAcceptFormats": [ + "string" + ], + "instillUIMultiline": true, + "instillUIOrder": 1, + "instillUpstreamTypes": [ + "value", + "reference", + "template" + ], + "title": "Message", + "type": "string" + }, + "is_public_channel": { + "description": "Whether all members can read the channel", + "instillAcceptFormats": [ + "boolean" + ], + "instillUIMultiline": true, + "instillUIOrder": 2, + "instillUpstreamTypes": [ + "value", + "reference", + "template" + ], + "title": "Public channel", + "type": "boolean" + } + }, + "required": [ + "channel_name", + "message" + ], + "title": "Input", + "type": "object" + }, + "output": { + "description": "The greeting sentence", + "instillUIOrder": 0, + "properties": { + "result": { + "description": "result for sending message", + "instillEditOnNodeFields": [], + "instillUIOrder": 0, + "required": [], + "title": "Result", + "type": "string", + "instillFormat": "string" + } + }, + "required": [ + "result" + ], + "title": "Output", + "type": "object" + } + } +} diff --git a/pkg/connector/slack/v0/connector_test.go b/application/slack/v0/connector_test.go similarity index 98% rename from pkg/connector/slack/v0/connector_test.go rename to application/slack/v0/connector_test.go index 8e2e3d48c..76a180517 100644 --- a/pkg/connector/slack/v0/connector_test.go +++ b/application/slack/v0/connector_test.go @@ -7,7 +7,7 @@ import ( "time" qt "github.com/frankban/quicktest" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" ) diff --git a/pkg/connector/slack/v0/main.go b/application/slack/v0/main.go similarity index 98% rename from pkg/connector/slack/v0/main.go rename to application/slack/v0/main.go index a2085210f..bf8e1f048 100644 --- a/pkg/connector/slack/v0/main.go +++ b/application/slack/v0/main.go @@ -9,7 +9,7 @@ import ( "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" "github.com/instill-ai/x/errmsg" "github.com/slack-go/slack" ) diff --git a/pkg/connector/slack/v0/mockSlack.go b/application/slack/v0/mockSlack.go similarity index 100% rename from pkg/connector/slack/v0/mockSlack.go rename to application/slack/v0/mockSlack.go diff --git a/pkg/connector/slack/v0/structs.go b/application/slack/v0/structs.go similarity index 100% rename from pkg/connector/slack/v0/structs.go rename to application/slack/v0/structs.go diff --git a/pkg/connector/slack/v0/taskFunctions.go b/application/slack/v0/taskFunctions.go similarity index 98% rename from pkg/connector/slack/v0/taskFunctions.go rename to application/slack/v0/taskFunctions.go index 0b79752b0..9fe897dbc 100644 --- a/pkg/connector/slack/v0/taskFunctions.go +++ b/application/slack/v0/taskFunctions.go @@ -5,7 +5,7 @@ import ( "sync" "time" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" "github.com/slack-go/slack" "google.golang.org/protobuf/types/known/structpb" ) @@ -17,7 +17,7 @@ func (e *execution) readMessage(in *structpb.Struct) (*structpb.Struct, error) { if err := base.ConvertFromStructpb(in, ¶ms); err != nil { return nil, err } - + targetChannelID, err := loopChannelListAPI(e, params.IsPublicChannel, params.ChannelName) if err != nil { @@ -80,7 +80,7 @@ func (e *execution) readMessage(in *structpb.Struct) (*structpb.Struct, error) { if readTaskResp.Conversations == nil { readTaskResp.Conversations = []Conversation{} } - + out, err := base.ConvertToStructpb(readTaskResp) if err != nil { return nil, err diff --git a/pkg/connector/website/v0/README.mdx b/application/website/v0/README.mdx similarity index 94% rename from pkg/connector/website/v0/README.mdx rename to application/website/v0/README.mdx index ad4bb54ce..af4a3f152 100644 --- a/pkg/connector/website/v0/README.mdx +++ b/application/website/v0/README.mdx @@ -16,7 +16,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/website/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/website/v0/config/definition.json). ## Supported Tasks diff --git a/pkg/connector/website/v0/assets/website.svg b/application/website/v0/assets/website.svg similarity index 100% rename from pkg/connector/website/v0/assets/website.svg rename to application/website/v0/assets/website.svg diff --git a/pkg/connector/website/v0/config/definition.json b/application/website/v0/config/definition.json similarity index 94% rename from pkg/connector/website/v0/config/definition.json rename to application/website/v0/config/definition.json index 117797b1f..318eaa91a 100644 --- a/pkg/connector/website/v0/config/definition.json +++ b/application/website/v0/config/definition.json @@ -15,6 +15,6 @@ "vendor": "", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/website/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/application/website/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/website/v0/config/tasks.json b/application/website/v0/config/tasks.json similarity index 100% rename from pkg/connector/website/v0/config/tasks.json rename to application/website/v0/config/tasks.json diff --git a/pkg/connector/website/v0/main.go b/application/website/v0/main.go similarity index 97% rename from pkg/connector/website/v0/main.go rename to application/website/v0/main.go index 9e5191a7d..a141a6fc9 100644 --- a/pkg/connector/website/v0/main.go +++ b/application/website/v0/main.go @@ -9,7 +9,7 @@ import ( "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) const ( diff --git a/pkg/connector/website/v0/scrape_website.go b/application/website/v0/scrape_website.go similarity index 98% rename from pkg/connector/website/v0/scrape_website.go rename to application/website/v0/scrape_website.go index 261f3dddb..55efa0a9e 100644 --- a/pkg/connector/website/v0/scrape_website.go +++ b/application/website/v0/scrape_website.go @@ -9,7 +9,7 @@ import ( "github.com/PuerkitoBio/goquery" "github.com/gocolly/colly/v2" - "github.com/instill-ai/component/pkg/connector/util" + "github.com/instill-ai/component/internal/util" ) type PageInfo struct { diff --git a/pkg/base/component.go b/base/component.go similarity index 99% rename from pkg/base/component.go rename to base/component.go index fd36d0262..645f90e17 100644 --- a/pkg/base/component.go +++ b/base/component.go @@ -6,7 +6,6 @@ import ( "strings" "github.com/gofrs/uuid" - "github.com/instill-ai/component/pkg/jsonref" "github.com/lestrrat-go/jsref/provider" "go.uber.org/zap" "golang.org/x/text/cases" @@ -15,6 +14,8 @@ import ( "google.golang.org/protobuf/proto" "google.golang.org/protobuf/types/known/structpb" + "github.com/instill-ai/component/internal/jsonref" + pipelinePB "github.com/instill-ai/protogen-go/vdp/pipeline/v1beta" ) diff --git a/pkg/base/connector.go b/base/connector.go similarity index 100% rename from pkg/base/connector.go rename to base/connector.go diff --git a/pkg/base/connector_test.go b/base/connector_test.go similarity index 100% rename from pkg/base/connector_test.go rename to base/connector_test.go diff --git a/pkg/base/execution.go b/base/execution.go similarity index 100% rename from pkg/base/execution.go rename to base/execution.go diff --git a/pkg/base/formats.go b/base/formats.go similarity index 100% rename from pkg/base/formats.go rename to base/formats.go diff --git a/pkg/base/openapi.go b/base/openapi.go similarity index 100% rename from pkg/base/openapi.go rename to base/openapi.go diff --git a/pkg/base/operator.go b/base/operator.go similarity index 100% rename from pkg/base/operator.go rename to base/operator.go diff --git a/pkg/base/operator_test.go b/base/operator_test.go similarity index 100% rename from pkg/base/operator_test.go rename to base/operator_test.go diff --git a/pkg/base/testdata/connectorAdditional.json b/base/testdata/connectorAdditional.json similarity index 100% rename from pkg/base/testdata/connectorAdditional.json rename to base/testdata/connectorAdditional.json diff --git a/pkg/base/testdata/connectorDef.json b/base/testdata/connectorDef.json similarity index 98% rename from pkg/base/testdata/connectorDef.json rename to base/testdata/connectorDef.json index f2c45c08e..785800b45 100644 --- a/pkg/base/testdata/connectorDef.json +++ b/base/testdata/connectorDef.json @@ -1,6 +1,6 @@ { "version": "1.0.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/base", + "source_url": "https://github.com/instill-ai/component/blob/main/base", "available_tasks": [ "TASK_TEXT_EMBEDDINGS" ], diff --git a/pkg/base/testdata/connectorTasks.json b/base/testdata/connectorTasks.json similarity index 100% rename from pkg/base/testdata/connectorTasks.json rename to base/testdata/connectorTasks.json diff --git a/pkg/base/testdata/operatorDef.json b/base/testdata/operatorDef.json similarity index 96% rename from pkg/base/testdata/operatorDef.json rename to base/testdata/operatorDef.json index 85c9d346e..c0afa3cc6 100644 --- a/pkg/base/testdata/operatorDef.json +++ b/base/testdata/operatorDef.json @@ -1,6 +1,6 @@ { "version": "1.0.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/base", + "source_url": "https://github.com/instill-ai/component/blob/main/base", "available_tasks": [ "TASK_MARSHAL" ], diff --git a/pkg/base/testdata/operatorTasks.json b/base/testdata/operatorTasks.json similarity index 100% rename from pkg/base/testdata/operatorTasks.json rename to base/testdata/operatorTasks.json diff --git a/pkg/base/testdata/wantConnectorDefinition.json b/base/testdata/wantConnectorDefinition.json similarity index 99% rename from pkg/base/testdata/wantConnectorDefinition.json rename to base/testdata/wantConnectorDefinition.json index 3325ce5d5..4511aa950 100644 --- a/pkg/base/testdata/wantConnectorDefinition.json +++ b/base/testdata/wantConnectorDefinition.json @@ -206,5 +206,5 @@ } ], "version": "1.0.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/base" + "source_url": "https://github.com/instill-ai/component/blob/main/base" } diff --git a/pkg/base/testdata/wantOperatorDefinition.json b/base/testdata/wantOperatorDefinition.json similarity index 99% rename from pkg/base/testdata/wantOperatorDefinition.json rename to base/testdata/wantOperatorDefinition.json index de3735119..b58e9e5fb 100644 --- a/pkg/base/testdata/wantOperatorDefinition.json +++ b/base/testdata/wantOperatorDefinition.json @@ -128,7 +128,7 @@ }, "public": true, "version": "1.0.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/base", + "source_url": "https://github.com/instill-ai/component/blob/main/base", "tasks": [ { "name": "TASK_MARSHAL", diff --git a/pkg/base/usage.go b/base/usage.go similarity index 100% rename from pkg/base/usage.go rename to base/usage.go diff --git a/component.go b/component.go new file mode 100644 index 000000000..c5c0c3307 --- /dev/null +++ b/component.go @@ -0,0 +1,216 @@ +package component + +import ( + "fmt" + "sync" + + "github.com/gofrs/uuid" + "go.uber.org/zap" + "google.golang.org/protobuf/types/known/structpb" + + "github.com/instill-ai/component/ai/archetypeai/v0" + "github.com/instill-ai/component/ai/huggingface/v0" + "github.com/instill-ai/component/ai/instill/v0" + "github.com/instill-ai/component/ai/openai/v0" + "github.com/instill-ai/component/ai/stabilityai/v0" + "github.com/instill-ai/component/application/googlesearch/v0" + "github.com/instill-ai/component/application/numbers/v0" + "github.com/instill-ai/component/application/redis/v0" + "github.com/instill-ai/component/application/restapi/v0" + "github.com/instill-ai/component/application/slack/v0" + "github.com/instill-ai/component/application/website/v0" + "github.com/instill-ai/component/base" + "github.com/instill-ai/component/data/bigquery/v0" + "github.com/instill-ai/component/data/googlecloudstorage/v0" + "github.com/instill-ai/component/data/pinecone/v0" + "github.com/instill-ai/component/operator/base64/v0" + "github.com/instill-ai/component/operator/image/v0" + "github.com/instill-ai/component/operator/json/v0" + "github.com/instill-ai/component/operator/text/v0" + + pipelinePB "github.com/instill-ai/protogen-go/vdp/pipeline/v1beta" +) + +var ( + once sync.Once + compStore *Store +) + +// Store holds in-memory information about the initialized components. +type Store struct { + operatorUIDs []uuid.UUID + operatorUIDMap map[uuid.UUID]*operator + operatorIDMap map[string]*operator + + connectorUIDs []uuid.UUID + connectorUIDMap map[uuid.UUID]*connector + connectorIDMap map[string]*connector +} + +type operator struct { + op base.IOperator +} + +type connector struct { + con base.IConnector +} + +// ConnectionSecrets contains the global connection secrets of each +// implemented connector (referenced by ID). Connectors may use these secrets +// to skip the connector configuration step and have a ready-to-run +// connection. +type ConnectionSecrets map[string]map[string]any + +// Init initializes the components implemented in this repository and loads +// their information to memory. +func Init( + logger *zap.Logger, + secrets ConnectionSecrets, + usageHandlerCreators map[string]base.UsageHandlerCreator, +) *Store { + baseOp := base.Operator{Logger: logger} + baseConn := base.Connector{Logger: logger} + + once.Do(func() { + compStore = &Store{ + operatorUIDMap: map[uuid.UUID]*operator{}, + operatorIDMap: map[string]*operator{}, + connectorUIDMap: map[uuid.UUID]*connector{}, + connectorIDMap: map[string]*connector{}, + } + compStore.ImportOperator(base64.Init(baseOp)) + compStore.ImportOperator(json.Init(baseOp)) + compStore.ImportOperator(image.Init(baseOp)) + compStore.ImportOperator(text.Init(baseOp)) + + { + // StabilityAI + conn := stabilityai.Init(baseConn) + + // Secret doesn't allow hyphens + conn = conn.WithSecrets(secrets["stabilityai"]). + WithUsageHandlerCreator(usageHandlerCreators[conn.GetID()]) + compStore.ImportConnector(conn) + } + + compStore.ImportConnector(instill.Init(baseConn)) + compStore.ImportConnector(huggingface.Init(baseConn)) + + { + // OpenAI + conn := openai.Init(baseConn) + conn = conn.WithSecrets(secrets[conn.GetID()]). + WithUsageHandlerCreator(usageHandlerCreators[conn.GetID()]) + compStore.ImportConnector(conn) + } + + compStore.ImportConnector(archetypeai.Init(baseConn)) + compStore.ImportConnector(numbers.Init(baseConn)) + compStore.ImportConnector(bigquery.Init(baseConn)) + compStore.ImportConnector(googlecloudstorage.Init(baseConn)) + compStore.ImportConnector(googlesearch.Init(baseConn)) + compStore.ImportConnector(pinecone.Init(baseConn)) + compStore.ImportConnector(redis.Init(baseConn)) + compStore.ImportConnector(restapi.Init(baseConn)) + compStore.ImportConnector(website.Init(baseConn)) + compStore.ImportConnector(slack.Init(baseConn)) + + }) + return compStore +} + +// ImportOperator loads the operator definitions into memory. +func (s *Store) ImportOperator(op base.IOperator) { + o := &operator{op: op} + s.operatorUIDMap[op.GetUID()] = o + s.operatorIDMap[op.GetID()] = o + s.operatorUIDs = append(s.operatorUIDs, op.GetUID()) +} + +// ImportConnector loads the connector definitions into memory. +func (s *Store) ImportConnector(con base.IConnector) { + c := &connector{con: con} + s.connectorUIDMap[con.GetUID()] = c + s.connectorIDMap[con.GetID()] = c + s.connectorUIDs = append(s.connectorUIDs, con.GetUID()) +} + +// CreateExecution initializes the execution of a connector given its UID. +func (s *Store) CreateExecution(defUID uuid.UUID, sysVars map[string]any, connection *structpb.Struct, task string) (*base.ExecutionWrapper, error) { + if op, ok := s.operatorUIDMap[defUID]; ok { + return op.op.CreateExecution(sysVars, task) + } + if con, ok := s.connectorUIDMap[defUID]; ok { + return con.con.CreateExecution(sysVars, connection, task) + } + return nil, fmt.Errorf("component definition not found") +} + +// GetOperatorDefinitionByUID returns a operator definition by its UID. +func (s *Store) GetOperatorDefinitionByUID(defUID uuid.UUID, sysVars map[string]any, component *pipelinePB.OperatorComponent) (*pipelinePB.OperatorDefinition, error) { + if op, ok := s.operatorUIDMap[defUID]; ok { + return op.op.GetOperatorDefinition(sysVars, component) + } + return nil, fmt.Errorf("operator definition not found") +} + +// GetOperatorDefinitionByID returns a operator definition by its ID. +func (s *Store) GetOperatorDefinitionByID(defID string, sysVars map[string]any, component *pipelinePB.OperatorComponent) (*pipelinePB.OperatorDefinition, error) { + if op, ok := s.operatorIDMap[defID]; ok { + return op.op.GetOperatorDefinition(sysVars, component) + } + return nil, fmt.Errorf("operator definition not found") +} + +// ListOperatorDefinitions returns all the loaded operator definitions. +func (s *Store) ListOperatorDefinitions(sysVars map[string]any, returnTombstone bool) []*pipelinePB.OperatorDefinition { + defs := []*pipelinePB.OperatorDefinition{} + for _, uid := range s.operatorUIDs { + op := s.operatorUIDMap[uid] + def, err := op.op.GetOperatorDefinition(sysVars, nil) + if err == nil { + if !def.Tombstone || returnTombstone { + defs = append(defs, def) + } + } + } + return defs +} + +// GetConnectorDefinitionByUID returns a connector definition by its UID. +func (s *Store) GetConnectorDefinitionByUID(defUID uuid.UUID, sysVars map[string]any, component *pipelinePB.ConnectorComponent) (*pipelinePB.ConnectorDefinition, error) { + if con, ok := s.connectorUIDMap[defUID]; ok { + return con.con.GetConnectorDefinition(sysVars, component) + } + return nil, fmt.Errorf("connector definition not found") +} + +// GetConnectorDefinitionByID returns a connector definition by its ID. +func (s *Store) GetConnectorDefinitionByID(defID string, sysVars map[string]any, component *pipelinePB.ConnectorComponent) (*pipelinePB.ConnectorDefinition, error) { + if con, ok := s.connectorIDMap[defID]; ok { + return con.con.GetConnectorDefinition(sysVars, component) + } + return nil, fmt.Errorf("connector definition not found") +} + +// ListConnectorDefinitions returns all the loaded connector definitions. +func (s *Store) ListConnectorDefinitions(sysVars map[string]any, returnTombstone bool) []*pipelinePB.ConnectorDefinition { + defs := []*pipelinePB.ConnectorDefinition{} + for _, uid := range s.connectorUIDs { + con := s.connectorUIDMap[uid] + def, err := con.con.GetConnectorDefinition(sysVars, nil) + if err == nil { + if !def.Tombstone || returnTombstone { + defs = append(defs, def) + } + } + } + return defs +} + +func (s *Store) IsSecretField(defUID uuid.UUID, target string) (bool, error) { + if con, ok := s.connectorUIDMap[defUID]; ok { + return con.con.IsSecretField(target), nil + } + return false, fmt.Errorf("connector definition not found") +} diff --git a/pkg/connector/bigquery/v0/README.mdx b/data/bigquery/v0/README.mdx similarity index 93% rename from pkg/connector/bigquery/v0/README.mdx rename to data/bigquery/v0/README.mdx index 155769fb1..72714a03a 100644 --- a/pkg/connector/bigquery/v0/README.mdx +++ b/data/bigquery/v0/README.mdx @@ -16,7 +16,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/bigquery/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/bigquery/v0/config/definition.json). ## Connection diff --git a/pkg/connector/bigquery/v0/assets/bigquery.svg b/data/bigquery/v0/assets/bigquery.svg similarity index 100% rename from pkg/connector/bigquery/v0/assets/bigquery.svg rename to data/bigquery/v0/assets/bigquery.svg diff --git a/pkg/connector/bigquery/v0/config/definition.json b/data/bigquery/v0/config/definition.json similarity index 98% rename from pkg/connector/bigquery/v0/config/definition.json rename to data/bigquery/v0/config/definition.json index dd6b4f212..4392259a3 100644 --- a/pkg/connector/bigquery/v0/config/definition.json +++ b/data/bigquery/v0/config/definition.json @@ -88,6 +88,6 @@ "vendor": "Google", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/bigquery/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/data/bigquery/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/bigquery/v0/config/tasks.json b/data/bigquery/v0/config/tasks.json similarity index 100% rename from pkg/connector/bigquery/v0/config/tasks.json rename to data/bigquery/v0/config/tasks.json diff --git a/pkg/connector/bigquery/v0/insert.go b/data/bigquery/v0/insert.go similarity index 100% rename from pkg/connector/bigquery/v0/insert.go rename to data/bigquery/v0/insert.go diff --git a/pkg/connector/bigquery/v0/main.go b/data/bigquery/v0/main.go similarity index 98% rename from pkg/connector/bigquery/v0/main.go rename to data/bigquery/v0/main.go index 5d4598cee..d0de96515 100644 --- a/pkg/connector/bigquery/v0/main.go +++ b/data/bigquery/v0/main.go @@ -12,7 +12,7 @@ import ( "google.golang.org/api/option" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) const ( diff --git a/pkg/connector/googlecloudstorage/v0/README.mdx b/data/googlecloudstorage/v0/README.mdx similarity index 94% rename from pkg/connector/googlecloudstorage/v0/README.mdx rename to data/googlecloudstorage/v0/README.mdx index af00b26a2..3e61bcbbf 100644 --- a/pkg/connector/googlecloudstorage/v0/README.mdx +++ b/data/googlecloudstorage/v0/README.mdx @@ -16,7 +16,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/googlecloudstorage/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/googlecloudstorage/v0/config/definition.json). ## Connection diff --git a/pkg/connector/googlecloudstorage/v0/assets/gcs.svg b/data/googlecloudstorage/v0/assets/gcs.svg similarity index 100% rename from pkg/connector/googlecloudstorage/v0/assets/gcs.svg rename to data/googlecloudstorage/v0/assets/gcs.svg diff --git a/pkg/connector/googlecloudstorage/v0/config/definition.json b/data/googlecloudstorage/v0/config/definition.json similarity index 97% rename from pkg/connector/googlecloudstorage/v0/config/definition.json rename to data/googlecloudstorage/v0/config/definition.json index 96d6317cb..0a5d06237 100644 --- a/pkg/connector/googlecloudstorage/v0/config/definition.json +++ b/data/googlecloudstorage/v0/config/definition.json @@ -61,6 +61,6 @@ "vendor": "Google", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/googlecloudstorage/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/data/googlecloudstorage/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/googlecloudstorage/v0/config/tasks.json b/data/googlecloudstorage/v0/config/tasks.json similarity index 100% rename from pkg/connector/googlecloudstorage/v0/config/tasks.json rename to data/googlecloudstorage/v0/config/tasks.json diff --git a/pkg/connector/googlecloudstorage/v0/main.go b/data/googlecloudstorage/v0/main.go similarity index 98% rename from pkg/connector/googlecloudstorage/v0/main.go rename to data/googlecloudstorage/v0/main.go index efed59303..b296a3271 100644 --- a/pkg/connector/googlecloudstorage/v0/main.go +++ b/data/googlecloudstorage/v0/main.go @@ -11,7 +11,7 @@ import ( "google.golang.org/api/option" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) const ( diff --git a/pkg/connector/googlecloudstorage/v0/upload.go b/data/googlecloudstorage/v0/upload.go similarity index 97% rename from pkg/connector/googlecloudstorage/v0/upload.go rename to data/googlecloudstorage/v0/upload.go index 553b24258..680631896 100644 --- a/pkg/connector/googlecloudstorage/v0/upload.go +++ b/data/googlecloudstorage/v0/upload.go @@ -8,7 +8,7 @@ import ( "cloud.google.com/go/iam" "cloud.google.com/go/storage" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) func uploadToGCS(client *storage.Client, bucketName, objectName, data string) error { diff --git a/pkg/connector/pinecone/v0/README.mdx b/data/pinecone/v0/README.mdx similarity index 96% rename from pkg/connector/pinecone/v0/README.mdx rename to data/pinecone/v0/README.mdx index 10b173b4f..b7b3a49cc 100644 --- a/pkg/connector/pinecone/v0/README.mdx +++ b/data/pinecone/v0/README.mdx @@ -17,7 +17,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/pinecone/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/pinecone/v0/config/definition.json). ## Connection diff --git a/pkg/connector/pinecone/v0/assets/pinecone.svg b/data/pinecone/v0/assets/pinecone.svg similarity index 100% rename from pkg/connector/pinecone/v0/assets/pinecone.svg rename to data/pinecone/v0/assets/pinecone.svg diff --git a/pkg/connector/pinecone/v0/config/definition.json b/data/pinecone/v0/config/definition.json similarity index 98% rename from pkg/connector/pinecone/v0/config/definition.json rename to data/pinecone/v0/config/definition.json index 7146a6322..6baee0ddd 100644 --- a/pkg/connector/pinecone/v0/config/definition.json +++ b/data/pinecone/v0/config/definition.json @@ -61,6 +61,6 @@ "vendor": "Pinecone", "vendor_attributes": {}, "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/pinecone/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/data/pinecone/v0", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/connector/pinecone/v0/config/tasks.json b/data/pinecone/v0/config/tasks.json similarity index 100% rename from pkg/connector/pinecone/v0/config/tasks.json rename to data/pinecone/v0/config/tasks.json diff --git a/pkg/connector/pinecone/v0/connector_test.go b/data/pinecone/v0/connector_test.go similarity index 98% rename from pkg/connector/pinecone/v0/connector_test.go rename to data/pinecone/v0/connector_test.go index 25ae7a2f2..93030bc9a 100644 --- a/pkg/connector/pinecone/v0/connector_test.go +++ b/data/pinecone/v0/connector_test.go @@ -13,8 +13,8 @@ import ( "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/base" + "github.com/instill-ai/component/internal/util/httpclient" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/connector/pinecone/v0/main.go b/data/pinecone/v0/main.go similarity index 96% rename from pkg/connector/pinecone/v0/main.go rename to data/pinecone/v0/main.go index b4e44f81d..f20ff5ff7 100644 --- a/pkg/connector/pinecone/v0/main.go +++ b/data/pinecone/v0/main.go @@ -9,8 +9,8 @@ import ( "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/util/httpclient" + "github.com/instill-ai/component/base" + "github.com/instill-ai/component/internal/util/httpclient" ) const ( diff --git a/pkg/connector/pinecone/v0/structs.go b/data/pinecone/v0/structs.go similarity index 100% rename from pkg/connector/pinecone/v0/structs.go rename to data/pinecone/v0/structs.go diff --git a/pkg/jsonref/interface.go b/internal/jsonref/interface.go similarity index 100% rename from pkg/jsonref/interface.go rename to internal/jsonref/interface.go diff --git a/pkg/jsonref/options.go b/internal/jsonref/options.go similarity index 100% rename from pkg/jsonref/options.go rename to internal/jsonref/options.go diff --git a/pkg/jsonref/ref.go b/internal/jsonref/ref.go similarity index 100% rename from pkg/jsonref/ref.go rename to internal/jsonref/ref.go diff --git a/internal/mock/generator.go b/internal/mock/generator.go new file mode 100644 index 000000000..0cf10adcb --- /dev/null +++ b/internal/mock/generator.go @@ -0,0 +1,3 @@ +package mock + +//go:generate minimock -g -i github.com/instill-ai/component/base.UsageHandler -o ./ -s "_mock.gen.go" diff --git a/pkg/mock/usage_handler_mock.gen.go b/internal/mock/usage_handler_mock.gen.go similarity index 100% rename from pkg/mock/usage_handler_mock.gen.go rename to internal/mock/usage_handler_mock.gen.go diff --git a/pkg/connector/util/helper.go b/internal/util/helper.go similarity index 97% rename from pkg/connector/util/helper.go rename to internal/util/helper.go index a7c79eec1..0e410fbb1 100644 --- a/pkg/connector/util/helper.go +++ b/internal/util/helper.go @@ -9,7 +9,7 @@ import ( md "github.com/JohannesKaufmann/html-to-markdown" "github.com/PuerkitoBio/goquery" "github.com/h2non/filetype" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) func GetFileExt(fileData []byte) string { diff --git a/pkg/connector/util/helper_test.go b/internal/util/helper_test.go similarity index 100% rename from pkg/connector/util/helper_test.go rename to internal/util/helper_test.go diff --git a/pkg/connector/util/httpclient/httpclient.go b/internal/util/httpclient/httpclient.go similarity index 100% rename from pkg/connector/util/httpclient/httpclient.go rename to internal/util/httpclient/httpclient.go diff --git a/pkg/connector/util/httpclient/httpclient_test.go b/internal/util/httpclient/httpclient_test.go similarity index 100% rename from pkg/connector/util/httpclient/httpclient_test.go rename to internal/util/httpclient/httpclient_test.go diff --git a/pkg/operator/base64/v0/README.mdx b/operator/base64/v0/README.mdx similarity index 92% rename from pkg/operator/base64/v0/README.mdx rename to operator/base64/v0/README.mdx index f02364ade..dcafb58b8 100644 --- a/pkg/operator/base64/v0/README.mdx +++ b/operator/base64/v0/README.mdx @@ -17,7 +17,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/operator/base64/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/operator/base64/v0/config/definition.json). ## Supported Tasks diff --git a/pkg/operator/base64/v0/assets/base64.svg b/operator/base64/v0/assets/base64.svg similarity index 100% rename from pkg/operator/base64/v0/assets/base64.svg rename to operator/base64/v0/assets/base64.svg diff --git a/pkg/operator/base64/v0/config/definition.json b/operator/base64/v0/config/definition.json similarity index 95% rename from pkg/operator/base64/v0/config/definition.json rename to operator/base64/v0/config/definition.json index 477cb447c..5214f6b6f 100644 --- a/pkg/operator/base64/v0/config/definition.json +++ b/operator/base64/v0/config/definition.json @@ -14,7 +14,7 @@ "tombstone": false, "uid": "3a836447-c211-4134-9cc5-ad45e1cc467e", "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/base64/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/operator/base64/v0", "description": "Encode or decode a string in Base64 format", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/operator/base64/v0/config/tasks.json b/operator/base64/v0/config/tasks.json similarity index 100% rename from pkg/operator/base64/v0/config/tasks.json rename to operator/base64/v0/config/tasks.json diff --git a/pkg/operator/base64/v0/encoding_test.go b/operator/base64/v0/encoding_test.go similarity index 100% rename from pkg/operator/base64/v0/encoding_test.go rename to operator/base64/v0/encoding_test.go diff --git a/pkg/operator/base64/v0/main.go b/operator/base64/v0/main.go similarity index 98% rename from pkg/operator/base64/v0/main.go rename to operator/base64/v0/main.go index ef46994a8..6ae7c41a6 100644 --- a/pkg/operator/base64/v0/main.go +++ b/operator/base64/v0/main.go @@ -12,7 +12,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) const ( diff --git a/pkg/operator/image/v0/README.mdx b/operator/image/v0/README.mdx similarity index 97% rename from pkg/operator/image/v0/README.mdx rename to operator/image/v0/README.mdx index 6a9194061..1f7834015 100644 --- a/pkg/operator/image/v0/README.mdx +++ b/operator/image/v0/README.mdx @@ -21,7 +21,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/operator/image/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/operator/image/v0/config/definition.json). ## Supported Tasks diff --git a/pkg/operator/image/v0/assets/image.svg b/operator/image/v0/assets/image.svg similarity index 100% rename from pkg/operator/image/v0/assets/image.svg rename to operator/image/v0/assets/image.svg diff --git a/pkg/operator/image/v0/config/definition.json b/operator/image/v0/config/definition.json similarity index 96% rename from pkg/operator/image/v0/config/definition.json rename to operator/image/v0/config/definition.json index 17036415d..97460d708 100644 --- a/pkg/operator/image/v0/config/definition.json +++ b/operator/image/v0/config/definition.json @@ -17,7 +17,7 @@ "title": "Image", "uid": "e9eb8fc8-f249-4e11-ad50-5035d79ffc18", "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/image/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/operator/image/v0", "description": "Manipulate image files", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/operator/image/v0/config/tasks.json b/operator/image/v0/config/tasks.json similarity index 100% rename from pkg/operator/image/v0/config/tasks.json rename to operator/image/v0/config/tasks.json diff --git a/pkg/operator/image/v0/draw.go b/operator/image/v0/draw.go similarity index 100% rename from pkg/operator/image/v0/draw.go rename to operator/image/v0/draw.go diff --git a/pkg/operator/image/v0/draw_test.go b/operator/image/v0/draw_test.go similarity index 100% rename from pkg/operator/image/v0/draw_test.go rename to operator/image/v0/draw_test.go diff --git a/pkg/operator/image/v0/font.go b/operator/image/v0/font.go similarity index 100% rename from pkg/operator/image/v0/font.go rename to operator/image/v0/font.go diff --git a/pkg/operator/image/v0/main.go b/operator/image/v0/main.go similarity index 98% rename from pkg/operator/image/v0/main.go rename to operator/image/v0/main.go index e3d23d5d3..44a6db72b 100644 --- a/pkg/operator/image/v0/main.go +++ b/operator/image/v0/main.go @@ -16,7 +16,7 @@ import ( "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) var ( diff --git a/pkg/operator/image/v0/testdata/cls-dog.json b/operator/image/v0/testdata/cls-dog.json similarity index 100% rename from pkg/operator/image/v0/testdata/cls-dog.json rename to operator/image/v0/testdata/cls-dog.json diff --git a/pkg/operator/image/v0/testdata/det-coco-1.json b/operator/image/v0/testdata/det-coco-1.json similarity index 100% rename from pkg/operator/image/v0/testdata/det-coco-1.json rename to operator/image/v0/testdata/det-coco-1.json diff --git a/pkg/operator/image/v0/testdata/det-coco-2.json b/operator/image/v0/testdata/det-coco-2.json similarity index 100% rename from pkg/operator/image/v0/testdata/det-coco-2.json rename to operator/image/v0/testdata/det-coco-2.json diff --git a/pkg/operator/image/v0/testdata/inst-seg-coco-1.json b/operator/image/v0/testdata/inst-seg-coco-1.json similarity index 100% rename from pkg/operator/image/v0/testdata/inst-seg-coco-1.json rename to operator/image/v0/testdata/inst-seg-coco-1.json diff --git a/pkg/operator/image/v0/testdata/inst-seg-coco-2.json b/operator/image/v0/testdata/inst-seg-coco-2.json similarity index 100% rename from pkg/operator/image/v0/testdata/inst-seg-coco-2.json rename to operator/image/v0/testdata/inst-seg-coco-2.json diff --git a/pkg/operator/image/v0/testdata/inst-seg-stomata.json b/operator/image/v0/testdata/inst-seg-stomata.json similarity index 100% rename from pkg/operator/image/v0/testdata/inst-seg-stomata.json rename to operator/image/v0/testdata/inst-seg-stomata.json diff --git a/pkg/operator/image/v0/testdata/kp-coco-1.json b/operator/image/v0/testdata/kp-coco-1.json similarity index 100% rename from pkg/operator/image/v0/testdata/kp-coco-1.json rename to operator/image/v0/testdata/kp-coco-1.json diff --git a/pkg/operator/image/v0/testdata/kp-coco-2.json b/operator/image/v0/testdata/kp-coco-2.json similarity index 100% rename from pkg/operator/image/v0/testdata/kp-coco-2.json rename to operator/image/v0/testdata/kp-coco-2.json diff --git a/pkg/operator/image/v0/testdata/ocr-mm.json b/operator/image/v0/testdata/ocr-mm.json similarity index 100% rename from pkg/operator/image/v0/testdata/ocr-mm.json rename to operator/image/v0/testdata/ocr-mm.json diff --git a/pkg/operator/image/v0/testdata/sem-seg-cityscape.json b/operator/image/v0/testdata/sem-seg-cityscape.json similarity index 100% rename from pkg/operator/image/v0/testdata/sem-seg-cityscape.json rename to operator/image/v0/testdata/sem-seg-cityscape.json diff --git a/pkg/operator/json/v0/README.mdx b/operator/json/v0/README.mdx similarity index 95% rename from pkg/operator/json/v0/README.mdx rename to operator/json/v0/README.mdx index cc61bea6a..924b53db8 100644 --- a/pkg/operator/json/v0/README.mdx +++ b/operator/json/v0/README.mdx @@ -18,7 +18,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/operator/json/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/operator/json/v0/config/definition.json). ## Supported Tasks diff --git a/pkg/operator/json/v0/assets/json.svg b/operator/json/v0/assets/json.svg similarity index 100% rename from pkg/operator/json/v0/assets/json.svg rename to operator/json/v0/assets/json.svg diff --git a/pkg/operator/json/v0/config/definition.json b/operator/json/v0/config/definition.json similarity index 95% rename from pkg/operator/json/v0/config/definition.json rename to operator/json/v0/config/definition.json index 2c272fb1b..b20855d7d 100644 --- a/pkg/operator/json/v0/config/definition.json +++ b/operator/json/v0/config/definition.json @@ -14,7 +14,7 @@ "title": "JSON", "uid": "28f53d15-6150-46e6-99aa-f76b70a926c0", "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/json/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/operator/json/v0", "description": "Manipulate and convert JSON objects", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/operator/json/v0/config/tasks.json b/operator/json/v0/config/tasks.json similarity index 100% rename from pkg/operator/json/v0/config/tasks.json rename to operator/json/v0/config/tasks.json diff --git a/pkg/operator/json/v0/main.go b/operator/json/v0/main.go similarity index 98% rename from pkg/operator/json/v0/main.go rename to operator/json/v0/main.go index dc122b628..b3e0220cb 100644 --- a/pkg/operator/json/v0/main.go +++ b/operator/json/v0/main.go @@ -12,7 +12,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/operator/json/v0/operator_test.go b/operator/json/v0/operator_test.go similarity index 98% rename from pkg/operator/json/v0/operator_test.go rename to operator/json/v0/operator_test.go index b46333b14..f1ff216b1 100644 --- a/pkg/operator/json/v0/operator_test.go +++ b/operator/json/v0/operator_test.go @@ -10,7 +10,7 @@ import ( "go.uber.org/zap" "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" "github.com/instill-ai/x/errmsg" ) diff --git a/pkg/operator/text/v0/README.mdx b/operator/text/v0/README.mdx similarity index 95% rename from pkg/operator/text/v0/README.mdx rename to operator/text/v0/README.mdx index d74f5ce18..9119faa3c 100644 --- a/pkg/operator/text/v0/README.mdx +++ b/operator/text/v0/README.mdx @@ -17,7 +17,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/operator/text/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/operator/text/v0/config/definition.json). ## Supported Tasks diff --git a/pkg/operator/text/v0/assets/text.svg b/operator/text/v0/assets/text.svg similarity index 100% rename from pkg/operator/text/v0/assets/text.svg rename to operator/text/v0/assets/text.svg diff --git a/pkg/operator/text/v0/config/definition.json b/operator/text/v0/config/definition.json similarity index 95% rename from pkg/operator/text/v0/config/definition.json rename to operator/text/v0/config/definition.json index b4937d6a3..a7fafc0ff 100644 --- a/pkg/operator/text/v0/config/definition.json +++ b/operator/text/v0/config/definition.json @@ -13,7 +13,7 @@ "title": "Text", "uid": "5b7aca5b-1ae3-477f-bf60-d34e1c993c87", "version": "0.1.1", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/text/v0", + "source_url": "https://github.com/instill-ai/component/blob/main/operator/text/v0", "description": "Extract and manipulate text from different sources", "release_stage": "RELEASE_STAGE_ALPHA" } diff --git a/pkg/operator/text/v0/config/tasks.json b/operator/text/v0/config/tasks.json similarity index 100% rename from pkg/operator/text/v0/config/tasks.json rename to operator/text/v0/config/tasks.json diff --git a/pkg/operator/text/v0/convert.go b/operator/text/v0/convert.go similarity index 97% rename from pkg/operator/text/v0/convert.go rename to operator/text/v0/convert.go index d5a33cd13..a6f44eea9 100644 --- a/pkg/operator/text/v0/convert.go +++ b/operator/text/v0/convert.go @@ -9,7 +9,7 @@ import ( "code.sajari.com/docconv" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) // ConvertToTextInput defines the input for convert to text task diff --git a/pkg/operator/text/v0/convert_test.go b/operator/text/v0/convert_test.go similarity index 100% rename from pkg/operator/text/v0/convert_test.go rename to operator/text/v0/convert_test.go diff --git a/pkg/operator/text/v0/main.go b/operator/text/v0/main.go similarity index 98% rename from pkg/operator/text/v0/main.go rename to operator/text/v0/main.go index a845418ab..9ff8b38d5 100644 --- a/pkg/operator/text/v0/main.go +++ b/operator/text/v0/main.go @@ -10,7 +10,7 @@ import ( "google.golang.org/protobuf/types/known/structpb" - "github.com/instill-ai/component/pkg/base" + "github.com/instill-ai/component/base" ) const ( diff --git a/pkg/operator/text/v0/split.go b/operator/text/v0/split.go similarity index 100% rename from pkg/operator/text/v0/split.go rename to operator/text/v0/split.go diff --git a/pkg/operator/text/v0/split_test.go b/operator/text/v0/split_test.go similarity index 100% rename from pkg/operator/text/v0/split_test.go rename to operator/text/v0/split_test.go diff --git a/pkg/operator/text/v0/testdata/test.docx b/operator/text/v0/testdata/test.docx similarity index 100% rename from pkg/operator/text/v0/testdata/test.docx rename to operator/text/v0/testdata/test.docx diff --git a/pkg/operator/text/v0/testdata/test.html b/operator/text/v0/testdata/test.html similarity index 100% rename from pkg/operator/text/v0/testdata/test.html rename to operator/text/v0/testdata/test.html diff --git a/pkg/operator/text/v0/testdata/test.jpg b/operator/text/v0/testdata/test.jpg similarity index 100% rename from pkg/operator/text/v0/testdata/test.jpg rename to operator/text/v0/testdata/test.jpg diff --git a/pkg/operator/text/v0/testdata/test.odt b/operator/text/v0/testdata/test.odt similarity index 100% rename from pkg/operator/text/v0/testdata/test.odt rename to operator/text/v0/testdata/test.odt diff --git a/pkg/operator/text/v0/testdata/test.pdf b/operator/text/v0/testdata/test.pdf similarity index 100% rename from pkg/operator/text/v0/testdata/test.pdf rename to operator/text/v0/testdata/test.pdf diff --git a/pkg/operator/text/v0/testdata/test.png b/operator/text/v0/testdata/test.png similarity index 100% rename from pkg/operator/text/v0/testdata/test.png rename to operator/text/v0/testdata/test.png diff --git a/pkg/operator/text/v0/testdata/test.rtf b/operator/text/v0/testdata/test.rtf similarity index 100% rename from pkg/operator/text/v0/testdata/test.rtf rename to operator/text/v0/testdata/test.rtf diff --git a/pkg/operator/text/v0/testdata/test.tif b/operator/text/v0/testdata/test.tif similarity index 100% rename from pkg/operator/text/v0/testdata/test.tif rename to operator/text/v0/testdata/test.tif diff --git a/pkg/operator/text/v0/testdata/test.txt b/operator/text/v0/testdata/test.txt similarity index 100% rename from pkg/operator/text/v0/testdata/test.txt rename to operator/text/v0/testdata/test.txt diff --git a/pkg/connector/airbyte/v0/assets/airbyte-destination.svg b/pkg/connector/airbyte/v0/assets/airbyte-destination.svg deleted file mode 100644 index c19d229cb..000000000 --- a/pkg/connector/airbyte/v0/assets/airbyte-destination.svg +++ /dev/null @@ -1,3 +0,0 @@ - - - diff --git a/pkg/connector/airbyte/v0/config/definition.json b/pkg/connector/airbyte/v0/config/definition.json deleted file mode 100644 index b19470cb2..000000000 --- a/pkg/connector/airbyte/v0/config/definition.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "available_tasks": [], - "custom": false, - "documentation_url": "https://docs.airbyte.com/integrations/destinations", - "icon": "Airbyte/airbyte.svg", - "icon_url": "", - "id": "airbyte-destination", - "public": true, - "version": "0.1.0", - "release_stage": "RELEASE_STAGE_ALPHA", - "spec": { - "connection_specification": { - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "Destination", - "type": "object" - } - }, - "title": "Airbyte Destination", - "tombstone": true, - "type": "CONNECTOR_TYPE_DATA", - "uid": "975678a2-5117-48a4-a135-019619dee18e", - "vendor": "Airbyte", - "vendor_attributes": {} -} diff --git a/pkg/connector/airbyte/v0/config/tasks.json b/pkg/connector/airbyte/v0/config/tasks.json deleted file mode 100644 index 0967ef424..000000000 --- a/pkg/connector/airbyte/v0/config/tasks.json +++ /dev/null @@ -1 +0,0 @@ -{} diff --git a/pkg/connector/airbyte/v0/main.go b/pkg/connector/airbyte/v0/main.go deleted file mode 100644 index f57d3baef..000000000 --- a/pkg/connector/airbyte/v0/main.go +++ /dev/null @@ -1,54 +0,0 @@ -package airbyte - -import ( - "context" - _ "embed" - "fmt" - "sync" - - "google.golang.org/protobuf/types/known/structpb" - - "github.com/instill-ai/component/pkg/base" -) - -//go:embed config/definition.json -var definitionJSON []byte - -//go:embed config/tasks.json -var tasksJSON []byte - -var once sync.Once -var con *connector - -type connector struct { - base.Connector -} - -type execution struct { - base.ConnectorExecution -} - -func Init(bc base.Connector) *connector { - once.Do(func() { - con = &connector{Connector: bc} - err := con.LoadConnectorDefinition(definitionJSON, tasksJSON, nil) - if err != nil { - panic(err) - } - }) - return con -} - -func (c *connector) CreateExecution(sysVars map[string]any, connection *structpb.Struct, task string) (*base.ExecutionWrapper, error) { - return &base.ExecutionWrapper{Execution: &execution{ - ConnectorExecution: base.ConnectorExecution{Connector: c, SystemVariables: sysVars, Connection: connection, Task: task}, - }}, nil -} - -func (e *execution) Execute(context.Context, []*structpb.Struct) ([]*structpb.Struct, error) { - return nil, fmt.Errorf("the Airbyte connector has been removed") -} - -func (c *connector) Test(map[string]any, *structpb.Struct) error { - return fmt.Errorf("the Airbyte connector has been removed") -} diff --git a/pkg/connector/main.go b/pkg/connector/main.go deleted file mode 100644 index 66de6f92d..000000000 --- a/pkg/connector/main.go +++ /dev/null @@ -1,159 +0,0 @@ -package connector - -import ( - "fmt" - "sync" - - "github.com/gofrs/uuid" - "go.uber.org/zap" - "google.golang.org/protobuf/types/known/structpb" - - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/connector/airbyte/v0" - "github.com/instill-ai/component/pkg/connector/archetypeai/v0" - "github.com/instill-ai/component/pkg/connector/bigquery/v0" - "github.com/instill-ai/component/pkg/connector/googlecloudstorage/v0" - "github.com/instill-ai/component/pkg/connector/googlesearch/v0" - "github.com/instill-ai/component/pkg/connector/huggingface/v0" - "github.com/instill-ai/component/pkg/connector/instill/v0" - "github.com/instill-ai/component/pkg/connector/numbers/v0" - "github.com/instill-ai/component/pkg/connector/openai/v0" - "github.com/instill-ai/component/pkg/connector/pinecone/v0" - "github.com/instill-ai/component/pkg/connector/redis/v0" - "github.com/instill-ai/component/pkg/connector/restapi/v0" - "github.com/instill-ai/component/pkg/connector/slack/v0" - "github.com/instill-ai/component/pkg/connector/stabilityai/v0" - "github.com/instill-ai/component/pkg/connector/website/v0" - - pipelinePB "github.com/instill-ai/protogen-go/vdp/pipeline/v1beta" -) - -var ( - once sync.Once - conStore *Store -) - -// Store holds in-memory information about the initialized connectors. -type Store struct { - connectorUIDs []uuid.UUID - connectorUIDMap map[uuid.UUID]*connector - connectorIDMap map[string]*connector -} - -type connector struct { - con base.IConnector -} - -// ConnectionSecrets contains the global connection secrets of each -// implemented connector (referenced by ID). Connectors may use these secrets -// to skip the connector configuration step and have a ready-to-run -// connection. -type ConnectionSecrets map[string]map[string]any - -// Init initializes the different connector components and loads their -// information to memory. -func Init( - logger *zap.Logger, - secrets ConnectionSecrets, - usageHandlerCreators map[string]base.UsageHandlerCreator, -) *Store { - - baseConn := base.Connector{Logger: logger} - - once.Do(func() { - conStore = &Store{ - connectorUIDMap: map[uuid.UUID]*connector{}, - connectorIDMap: map[string]*connector{}, - } - - { - // StabilityAI - conn := stabilityai.Init(baseConn) - - // Secret doesn't allow hyphens - conn = conn.WithSecrets(secrets["stabilityai"]). - WithUsageHandlerCreator(usageHandlerCreators[conn.GetID()]) - conStore.Import(conn) - } - - conStore.Import(instill.Init(baseConn)) - conStore.Import(huggingface.Init(baseConn)) - - { - // OpenAI - conn := openai.Init(baseConn) - conn = conn.WithSecrets(secrets[conn.GetID()]). - WithUsageHandlerCreator(usageHandlerCreators[conn.GetID()]) - conStore.Import(conn) - } - - conStore.Import(archetypeai.Init(baseConn)) - conStore.Import(numbers.Init(baseConn)) - conStore.Import(airbyte.Init(baseConn)) - conStore.Import(bigquery.Init(baseConn)) - conStore.Import(googlecloudstorage.Init(baseConn)) - conStore.Import(googlesearch.Init(baseConn)) - conStore.Import(pinecone.Init(baseConn)) - conStore.Import(redis.Init(baseConn)) - conStore.Import(restapi.Init(baseConn)) - conStore.Import(website.Init(baseConn)) - conStore.Import(slack.Init(baseConn)) - - }) - - return conStore -} - -// Import loads the connector definitions into memory. -func (cs *Store) Import(con base.IConnector) { - c := &connector{con: con} - cs.connectorUIDMap[con.GetUID()] = c - cs.connectorIDMap[con.GetID()] = c - cs.connectorUIDs = append(cs.connectorUIDs, con.GetUID()) -} - -// CreateExecution initializes the execution of a connector given its UID. -func (cs *Store) CreateExecution(defUID uuid.UUID, sysVars map[string]any, connection *structpb.Struct, task string) (*base.ExecutionWrapper, error) { - if con, ok := cs.connectorUIDMap[defUID]; ok { - return con.con.CreateExecution(sysVars, connection, task) - } - return nil, fmt.Errorf("connector definition not found") -} - -// GetConnectorDefinitionByUID returns a connector definition by its UID. -func (cs *Store) GetConnectorDefinitionByUID(defUID uuid.UUID, sysVars map[string]any, component *pipelinePB.ConnectorComponent) (*pipelinePB.ConnectorDefinition, error) { - if con, ok := cs.connectorUIDMap[defUID]; ok { - return con.con.GetConnectorDefinition(sysVars, component) - } - return nil, fmt.Errorf("connector definition not found") -} - -// GetConnectorDefinitionByID returns a connector definition by its ID. -func (cs *Store) GetConnectorDefinitionByID(defID string, sysVars map[string]any, component *pipelinePB.ConnectorComponent) (*pipelinePB.ConnectorDefinition, error) { - if con, ok := cs.connectorIDMap[defID]; ok { - return con.con.GetConnectorDefinition(sysVars, component) - } - return nil, fmt.Errorf("connector definition not found") -} - -// ListConnectorDefinitions returns all the loaded connector definitions. -func (cs *Store) ListConnectorDefinitions(sysVars map[string]any, returnTombstone bool) []*pipelinePB.ConnectorDefinition { - defs := []*pipelinePB.ConnectorDefinition{} - for _, uid := range cs.connectorUIDs { - con := cs.connectorUIDMap[uid] - def, err := con.con.GetConnectorDefinition(sysVars, nil) - if err == nil { - if !def.Tombstone || returnTombstone { - defs = append(defs, def) - } - } - } - return defs -} - -func (cs *Store) IsSecretField(defUID uuid.UUID, target string) (bool, error) { - if con, ok := cs.connectorUIDMap[defUID]; ok { - return con.con.IsSecretField(target), nil - } - return false, fmt.Errorf("connector definition not found") -} diff --git a/pkg/connector/slack/v0/config/tasks.json b/pkg/connector/slack/v0/config/tasks.json deleted file mode 100644 index bbfbc9323..000000000 --- a/pkg/connector/slack/v0/config/tasks.json +++ /dev/null @@ -1,239 +0,0 @@ -{ - "TASK_READ_MESSAGE": { - "instillShortDescription": "Get the latest message since specific date", - "input": { - "description": "Please input the channel name and the date that we want to start to read", - "instillUIOrder": 0, - "properties": { - "channel_name": { - "description": "A channel name display in Slack", - "instillAcceptFormats": [ - "string" - ], - "instillUIMultiline": true, - "instillUIOrder": 0, - "instillUpstreamTypes": [ - "value", - "reference", - "template" - ], - "title": "Channel Name", - "type": "string" - }, - "start_to_read_date": { - "description": "earliest date in all read messages", - "instillAcceptFormats": [ - "string" - ], - "instillUIMultiline": true, - "instillUIOrder": 1, - "instillUpstreamTypes": [ - "value", - "reference", - "template" - ], - "title": "Start to read date", - "type": "string" - }, - "is_public_channel": { - "description": "Whether all members can read the channel", - "instillAcceptFormats": [ - "boolean" - ], - "instillUIMultiline": true, - "instillUIOrder": 2, - "instillUpstreamTypes": [ - "value", - "reference", - "template" - ], - "title": "Public channel", - "type": "boolean" - } - }, - "required": [ - "channel_name" - ], - "title": "Input", - "type": "object" - }, - "output": { - "description": "All messages in Slack channel", - "instillUIOrder": 0, - "properties": { - "conversations": { - "description": "An array of conversations with thread messages", - "instillUIOrder": 0, - "title": "Conversations", - "type": "array", - "items": { - "title": "conversation details", - "type": "object", - "properties": { - "user_id": { - "description": "unique id from Slack", - "instillFormat": "string", - "instillUIOrder": 1, - "title": "User UID", - "type": "string" - }, - "message": { - "description": "message to start a conversation", - "instillFormat": "string", - "instillUIOrder": 2, - "title": "Start Conversation Message", - "type": "string" - }, - "start_date": { - "description": "when a conversation starts", - "instillFormat": "string", - "instillUIOrder": 3, - "required": [], - "title": "Start Date", - "type": "string" - }, - "last_date": { - "description": "Date of the last message", - "instillFormat": "string", - "instillUIOrder": 4, - "required": [], - "title": "Last Date", - "type": "string" - }, - "thread_reply_messages": { - "description": "replies in a conversation", - "instillFormat": "array", - "instillUIOrder": 5, - "title": "Replied messages", - "type": "array", - "items": { - "title": "relied details", - "type": "object", - "properties": { - "user_id": { - "description": "unique id from Slack", - "instillFormat": "string", - "instillUIOrder": 1, - "title": "User UID", - "type": "string" - }, - "datetime": { - "description": "replied datetime", - "instillFormat": "string", - "instillUIOrder": 2, - "title": "Replied Time", - "type": "string" - }, - "message": { - "description": "message to reply a conversation", - "instillFormat": "string", - "instillUIOrder": 3, - "title": "Replied Message", - "type": "string" - } - }, - "required": [ - "user_id", - "datetime", - "message" - ] - } - } - }, - "required": [ - "user_id", - "message", - "start_date" - ] - } - } - }, - "required": [ - "conversations" - ], - "title": "Output", - "type": "object" - } - }, - "TASK_WRITE_MESSAGE": { - "instillShortDescription": "send message to a specific channel", - "title": "Send Message", - "input": { - "description": "Input", - "instillUIOrder": 0, - "properties": { - "channel_name": { - "description": "A channel name display in Slack", - "instillAcceptFormats": [ - "string" - ], - "instillUIMultiline": true, - "instillUIOrder": 0, - "instillUpstreamTypes": [ - "value", - "reference", - "template" - ], - "title": "Channel Name", - "type": "string" - }, - "message": { - "description": "message to be sent to the target channel", - "instillAcceptFormats": [ - "string" - ], - "instillUIMultiline": true, - "instillUIOrder": 1, - "instillUpstreamTypes": [ - "value", - "reference", - "template" - ], - "title": "Message", - "type": "string" - }, - "is_public_channel": { - "description": "Whether all members can read the channel", - "instillAcceptFormats": [ - "boolean" - ], - "instillUIMultiline": true, - "instillUIOrder": 2, - "instillUpstreamTypes": [ - "value", - "reference", - "template" - ], - "title": "Public channel", - "type": "boolean" - } - }, - "required": [ - "channel_name", - "message" - ], - "title": "Input", - "type": "object" - }, - "output": { - "description": "The greeting sentence", - "instillUIOrder": 0, - "properties": { - "result": { - "description": "result for sending message", - "instillEditOnNodeFields": [], - "instillUIOrder": 0, - "required": [], - "title": "Result", - "type": "string", - "instillFormat": "string" - } - }, - "required": [ - "result" - ], - "title": "Output", - "type": "object" - } - } - } \ No newline at end of file diff --git a/pkg/mock/generator.go b/pkg/mock/generator.go deleted file mode 100644 index f64b8dfe9..000000000 --- a/pkg/mock/generator.go +++ /dev/null @@ -1,3 +0,0 @@ -package mock - -//go:generate minimock -g -i github.com/instill-ai/component/pkg/base.UsageHandler -o ./ -s "_mock.gen.go" diff --git a/pkg/operator/end/v0/assets/end.svg b/pkg/operator/end/v0/assets/end.svg deleted file mode 100644 index b62a05707..000000000 --- a/pkg/operator/end/v0/assets/end.svg +++ /dev/null @@ -1,3 +0,0 @@ - - - diff --git a/pkg/operator/end/v0/config/definition.json b/pkg/operator/end/v0/config/definition.json deleted file mode 100644 index bdb368e74..000000000 --- a/pkg/operator/end/v0/config/definition.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "available_tasks": [ - "TASK_END" - ], - "custom": false, - "documentation_url": "https://www.instill.tech/docs/latest/vdp/operators/op-end", - "icon": "assets/end.svg", - "icon_url": "", - "id": "end", - "public": true, - "spec": {}, - "title": "End", - "tombstone": true, - "uid": "4f39c8bc-8617-495d-80de-80d0f5397516", - "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/end/v0", - "description": "Create an output interface in a pipeline when triggered synchronously", - "release_stage": "RELEASE_STAGE_ALPHA" -} diff --git a/pkg/operator/end/v0/config/tasks.json b/pkg/operator/end/v0/config/tasks.json deleted file mode 100644 index 45149f505..000000000 --- a/pkg/operator/end/v0/config/tasks.json +++ /dev/null @@ -1,55 +0,0 @@ -{ - "TASK_END": { - "input": { - "additionalProperties": false, - "description": "Input", - "instillEditOnNodeFields": [], - "instillUIOrder": 0, - "patternProperties": { - "^[a-z_][-a-z_0-9]{0,31}$": { - "description": "Arbitrary data", - "instillAcceptFormats": [ - "*" - ], - "instillUIOrder": 0, - "instillUpstreamTypes": [ - "value", - "reference", - "template" - ], - "title": "Key" - } - }, - "required": [], - "title": "Input", - "type": "object" - }, - "metadata": { - "additionalProperties": false, - "patternProperties": { - "^[a-z_][-a-z_0-9]{0,31}$": { - "properties": { - "description": { - "type": "string" - }, - "title": { - "type": "string" - } - }, - "title": "Metadata Item", - "type": "object" - } - }, - "title": "Metadata", - "type": "object" - }, - "output": { - "description": "Output", - "instillEditOnNodeFields": [], - "instillUIOrder": 0, - "required": [], - "title": "Output", - "type": "object" - } - } -} diff --git a/pkg/operator/end/v0/main.go b/pkg/operator/end/v0/main.go deleted file mode 100644 index 939ce87ab..000000000 --- a/pkg/operator/end/v0/main.go +++ /dev/null @@ -1,54 +0,0 @@ -package end - -import ( - "context" - _ "embed" - "fmt" - "sync" - - "google.golang.org/protobuf/types/known/structpb" - - "github.com/instill-ai/component/pkg/base" -) - -//go:embed config/definition.json -var definitionJSON []byte - -//go:embed config/tasks.json -var tasksJSON []byte - -var once sync.Once -var op *operator - -type operator struct { - base.Operator -} - -type execution struct { - base.OperatorExecution -} - -func Init(bo base.Operator) *operator { - once.Do(func() { - op = &operator{Operator: bo} - err := op.LoadOperatorDefinition(definitionJSON, tasksJSON, nil) - if err != nil { - panic(err) - } - }) - return op -} - -func (o *operator) CreateExecution(sysVars map[string]any, task string) (*base.ExecutionWrapper, error) { - return &base.ExecutionWrapper{Execution: &execution{ - OperatorExecution: base.OperatorExecution{Operator: o, SystemVariables: sysVars, Task: task}, - }}, nil -} - -func (e *execution) Execute(_ context.Context, inputs []*structpb.Struct) ([]*structpb.Struct, error) { - return nil, fmt.Errorf("the Airbyte operator has been removed") -} - -func (o *operator) Test(sysVars map[string]any, connection *structpb.Struct) error { - return fmt.Errorf("the Airbyte operator has been removed") -} diff --git a/pkg/operator/main.go b/pkg/operator/main.go deleted file mode 100644 index 20777c262..000000000 --- a/pkg/operator/main.go +++ /dev/null @@ -1,103 +0,0 @@ -package operator - -import ( - "fmt" - "sync" - - "github.com/gofrs/uuid" - "go.uber.org/zap" - - "github.com/instill-ai/component/pkg/base" - "github.com/instill-ai/component/pkg/operator/base64/v0" - "github.com/instill-ai/component/pkg/operator/end/v0" - "github.com/instill-ai/component/pkg/operator/image/v0" - "github.com/instill-ai/component/pkg/operator/json/v0" - "github.com/instill-ai/component/pkg/operator/start/v0" - "github.com/instill-ai/component/pkg/operator/text/v0" - - pipelinePB "github.com/instill-ai/protogen-go/vdp/pipeline/v1beta" -) - -var ( - once sync.Once - opStore *Store -) - -// Store holds in-memory information about the initialized operators. -type Store struct { - operatorUIDs []uuid.UUID - operatorUIDMap map[uuid.UUID]*operator - operatorIDMap map[string]*operator -} - -type operator struct { - op base.IOperator -} - -// Init initializes the different operator components and loads their -// information to memory. -func Init(logger *zap.Logger) *Store { - baseOp := base.Operator{Logger: logger} - - once.Do(func() { - opStore = &Store{ - operatorUIDMap: map[uuid.UUID]*operator{}, - operatorIDMap: map[string]*operator{}, - } - opStore.Import(start.Init(baseOp)) // deprecated - opStore.Import(end.Init(baseOp)) // deprecated - opStore.Import(base64.Init(baseOp)) - opStore.Import(json.Init(baseOp)) - opStore.Import(image.Init(baseOp)) - opStore.Import(text.Init(baseOp)) - - }) - return opStore -} - -// Import loads the operator definitions into memory. -func (os *Store) Import(op base.IOperator) { - o := &operator{op: op} - os.operatorUIDMap[op.GetUID()] = o - os.operatorIDMap[op.GetID()] = o - os.operatorUIDs = append(os.operatorUIDs, op.GetUID()) -} - -// CreateExecution initializes the execution of a operator given its UID. -func (os *Store) CreateExecution(defUID uuid.UUID, sysVars map[string]any, task string) (*base.ExecutionWrapper, error) { - if op, ok := os.operatorUIDMap[defUID]; ok { - return op.op.CreateExecution(sysVars, task) - } - return nil, fmt.Errorf("operator definition not found") -} - -// GetOperatorDefinitionByUID returns a operator definition by its UID. -func (os *Store) GetOperatorDefinitionByUID(defUID uuid.UUID, sysVars map[string]any, component *pipelinePB.OperatorComponent) (*pipelinePB.OperatorDefinition, error) { - if op, ok := os.operatorUIDMap[defUID]; ok { - return op.op.GetOperatorDefinition(sysVars, component) - } - return nil, fmt.Errorf("operator definition not found") -} - -// GetOperatorDefinitionByID returns a operator definition by its ID. -func (os *Store) GetOperatorDefinitionByID(defID string, sysVars map[string]any, component *pipelinePB.OperatorComponent) (*pipelinePB.OperatorDefinition, error) { - if op, ok := os.operatorIDMap[defID]; ok { - return op.op.GetOperatorDefinition(sysVars, component) - } - return nil, fmt.Errorf("operator definition not found") -} - -// ListOperatorDefinitions returns all the loaded operator definitions. -func (os *Store) ListOperatorDefinitions(sysVars map[string]any, returnTombstone bool) []*pipelinePB.OperatorDefinition { - defs := []*pipelinePB.OperatorDefinition{} - for _, uid := range os.operatorUIDs { - op := os.operatorUIDMap[uid] - def, err := op.op.GetOperatorDefinition(sysVars, nil) - if err == nil { - if !def.Tombstone || returnTombstone { - defs = append(defs, def) - } - } - } - return defs -} diff --git a/pkg/operator/start/v0/assets/start.svg b/pkg/operator/start/v0/assets/start.svg deleted file mode 100644 index 418a3093c..000000000 --- a/pkg/operator/start/v0/assets/start.svg +++ /dev/null @@ -1,3 +0,0 @@ - - - diff --git a/pkg/operator/start/v0/config/definition.json b/pkg/operator/start/v0/config/definition.json deleted file mode 100644 index 3e64b86d6..000000000 --- a/pkg/operator/start/v0/config/definition.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "available_tasks": [ - "TASK_START" - ], - "custom": false, - "documentation_url": "https://www.instill.tech/docs/latest/vdp/operators/op-start", - "icon": "assets/start.svg", - "icon_url": "", - "id": "start", - "public": true, - "spec": {}, - "title": "Start", - "tombstone": true, - "uid": "2ac8be70-0f7a-4b61-a33d-098b8acfa6f3", - "version": "0.1.0", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/start/v0", - "description": "Define data formats customized for their AI-first applications", - "release_stage": "RELEASE_STAGE_ALPHA" -} diff --git a/pkg/operator/start/v0/config/tasks.json b/pkg/operator/start/v0/config/tasks.json deleted file mode 100644 index 25b0aee71..000000000 --- a/pkg/operator/start/v0/config/tasks.json +++ /dev/null @@ -1,64 +0,0 @@ -{ - "TASK_START": { - "input": { - "description": "Input", - "instillEditOnNodeFields": [], - "instillUIOrder": 0, - "required": [], - "title": "Input", - "type": "object" - }, - "metadata": { - "additionalProperties": false, - "patternProperties": { - "^[a-z_][-a-z_0-9]{0,31}$": { - "properties": { - "description": { - "type": "string" - }, - "instillFormat": { - "type": "string" - }, - "items": { - "properties": { - "instillFormat": { - "type": "string" - }, - "type": { - "type": "string" - } - }, - "required": [ - "type", - "instillFormat" - ], - "type": "object" - }, - "title": { - "type": "string" - }, - "type": { - "type": "string" - } - }, - "required": [ - "type", - "instillFormat" - ], - "title": "Metadata Item", - "type": "object" - } - }, - "title": "Metadata", - "type": "object" - }, - "output": { - "description": "Output", - "instillEditOnNodeFields": [], - "instillUIOrder": 0, - "required": [], - "title": "Output", - "type": "object" - } - } -} diff --git a/pkg/operator/start/v0/main.go b/pkg/operator/start/v0/main.go deleted file mode 100644 index 669968d2c..000000000 --- a/pkg/operator/start/v0/main.go +++ /dev/null @@ -1,54 +0,0 @@ -package start - -import ( - "context" - _ "embed" - "fmt" - "sync" - - "google.golang.org/protobuf/types/known/structpb" - - "github.com/instill-ai/component/pkg/base" -) - -//go:embed config/definition.json -var definitionJSON []byte - -//go:embed config/tasks.json -var tasksJSON []byte - -var once sync.Once -var op *operator - -type operator struct { - base.Operator -} - -type execution struct { - base.OperatorExecution -} - -func Init(bo base.Operator) *operator { - once.Do(func() { - op = &operator{Operator: bo} - err := op.LoadOperatorDefinition(definitionJSON, tasksJSON, nil) - if err != nil { - panic(err) - } - }) - return op -} - -func (o *operator) CreateExecution(sysVars map[string]any, task string) (*base.ExecutionWrapper, error) { - return &base.ExecutionWrapper{Execution: &execution{ - OperatorExecution: base.OperatorExecution{Operator: o, SystemVariables: sysVars, Task: task}, - }}, nil -} - -func (e *execution) Execute(_ context.Context, inputs []*structpb.Struct) ([]*structpb.Struct, error) { - return nil, fmt.Errorf("the Airbyte operator has been removed") -} - -func (o *operator) Test(sysVars map[string]any, connection *structpb.Struct) error { - return fmt.Errorf("the Airbyte operator has been removed") -} diff --git a/tools/compogen/README.md b/tools/compogen/README.md index 598bb51b7..3c74bd034 100644 --- a/tools/compogen/README.md +++ b/tools/compogen/README.md @@ -55,7 +55,7 @@ Certain optional fields modify the document behaviour: ## TODO -- Support `oneOf` schemas for resource properties, present in, e.g., the [REST API](https://github.com/instill-ai/component/blob/main/pkg/connector/restapi/v0/config/definition.json#L26) connectors. +- Support `oneOf` schemas for resource properties, present in, e.g., the [REST API](https://github.com/instill-ai/component/blob/main/application/restapi/v0/config/definition.json#L26) connectors. - We might leverage some Go implementation of JSON schema. Some candidates: - [santhosh-tekuri/jsonschema](https://pkg.go.dev/github.com/santhosh-tekuri/jsonschema/v5#Schema) - [omissis/go-jsonschema](https://github.com/omissis/go-jsonschema/blob/934012d/pkg/schemas/model.go#L107) diff --git a/tools/compogen/cmd/testdata/readme-connector.txt b/tools/compogen/cmd/testdata/readme-connector.txt index 4851f3b73..c0f4bee78 100644 --- a/tools/compogen/cmd/testdata/readme-connector.txt +++ b/tools/compogen/cmd/testdata/readme-connector.txt @@ -46,7 +46,7 @@ cmp pkg/dummy/README.mdx want-readme.mdx } }, "release_stage": "RELEASE_STAGE_COMING_SOON", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/dummy/v0" + "source_url": "https://github.com/instill-ai/component/blob/main/data/dummy/v0" } -- tasks.json -- { @@ -96,7 +96,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/dummy/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/dummy/v0/config/definition.json). ## Connection diff --git a/tools/compogen/cmd/testdata/readme-operator.txt b/tools/compogen/cmd/testdata/readme-operator.txt index 3950643c9..05e1258c6 100644 --- a/tools/compogen/cmd/testdata/readme-operator.txt +++ b/tools/compogen/cmd/testdata/readme-operator.txt @@ -34,7 +34,7 @@ cmp pkg/dummy/README.mdx want-readme.mdx "title": "Dummy", "description": "Perform an action", "release_stage": "RELEASE_STAGE_BETA", - "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/dummy/v0" + "source_url": "https://github.com/instill-ai/component/blob/main/operator/dummy/v0" } -- tasks.json -- { @@ -169,7 +169,7 @@ It can carry out the following tasks: ## Configuration -The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/operator/dummy/v0/config/definition.json). +The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/operator/dummy/v0/config/definition.json). ## Supported Tasks diff --git a/tools/compogen/pkg/gen/readme.go b/tools/compogen/pkg/gen/readme.go index d20afc935..b81287e65 100644 --- a/tools/compogen/pkg/gen/readme.go +++ b/tools/compogen/pkg/gen/readme.go @@ -16,7 +16,7 @@ import ( "github.com/go-playground/validator/v10" "github.com/russross/blackfriday/v2" - component "github.com/instill-ai/component/pkg/base" + componentbase "github.com/instill-ai/component/base" ) const ( @@ -53,7 +53,7 @@ func (g *READMEGenerator) parseDefinition(configDir string) (d definition, err e return d, err } - renderedDefinitionJSON, err := component.RenderJSON(definitionJSON, nil) + renderedDefinitionJSON, err := componentbase.RenderJSON(definitionJSON, nil) if err != nil { return d, err } @@ -94,7 +94,7 @@ func (g *READMEGenerator) parseTasks(configDir string) (map[string]task, error) } - renderedTasksJSON, err := component.RenderJSON(tasksJSON, additionalJSONs) + renderedTasksJSON, err := componentbase.RenderJSON(tasksJSON, additionalJSONs) if err != nil { return nil, err } @@ -237,7 +237,7 @@ func parseREADMETasks(availableTasks []string, tasks map[string]task) ([]readmeT } if rt.Title = t.Title; rt.Title == "" { - rt.Title = component.TaskIDToTitle(at) + rt.Title = componentbase.TaskIDToTitle(at) } readmeTasks[i] = rt