diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
index 52ccad4a5..f1aadf607 100644
--- a/.github/CONTRIBUTING.md
+++ b/.github/CONTRIBUTING.md
@@ -25,49 +25,47 @@ In VDP, a **pipeline** is a DAG (Directed Acyclic Graph) consisting of multiple
```mermaid
flowchart LR
- s[Trigger] --> c1[OpenAI Connector]
- c1 --> c2[Stability AI Connector]
- c1 --> c3[MySQL Connector]
+ s[Trigger] --> c1[OpenAI Component]
+ c1 --> c2[Stability AI Component]
+ c1 --> c3[MySQL Component]
c1 --> e[Response]
c2 --> e
```
### Component
-There are different types of component:
-- **connector**
- - Queries, processes or transmits the ingested data to a service or app.
- - Users need to configure their connectors (e.g. by providing an API token to
- a remote service).
-- **operator**
- - Performs data injection and manipulation.
-- **iterator**
- - Takes an array and executes an operation (defined by a set of nested
- components) on each of its elements.
-- **trigger / response**
- - These special components provide an input / output interface to pipeline
- triggers.
-
-#### Connector
-
-- **Connectors** are used by the pipeline to interact with an external service.
- They are defined and initialized in the [connector](../pkg/connector) package.
-- In order to set up a connector, you may need to introduce its **connection**
- details in the connector properties.
- - In order to prevent private keys from being unintentionally leaked when
+There are different types of component: **AI**, **Data**, **Application**, **Operator** and **Iterator**.
+
+> **Note:**
+> - For **AI**, **Data**, **Application** components, they are used by the pipeline to interact with an external service, you may need to introduce its **connection** details in the component connection properties.
+> - In order to prevent private keys from being unintentionally leaked when
sharing a pipeline, the connection properties only take reference to a
**secret** (e.g. `${secrets.my-secret}`).
- - You can create secrets from the console settings or through an [API
+> - You can create secrets from the console settings or through an [API
call](https://openapi.instill.tech/reference/pipelinepublicservice_createusersecret).
+#### AI
+
+**AI** components play a crucial role in transforming unstructured data into formats that are easy to interpret and analyze, thereby facilitating the extraction of valuable insights. These components integrate with AI models from various providers, whether it's the primary Instill Model or those from third-party AI vendors. They are defined and initialized in the [ai](../ai) package.
+
+#### Data
+
+**Data** components play a crucial role in establishing connections with remote data sources, such as IoT devices (e.g., IP cameras), cloud storage services (e.g., GCP Cloud Storage, AWS S3), data warehouses, or vector databases (e.g., Pinecone). These connectors act as the bridge between VDP and various external data sources. Their primary function is to enable seamless data exchange, enhancing Instill VDP's capability to work with diverse data sources effectively. They are defined and initialized in the [data](../data) package.
+
+#### Application
+
+**Application** components are used to seamlessly integrate various 3rd-party application services. They are defined and initialized in the [application](../application) package.
+
#### Operator
-- **Operators** perform data transformations inside the pipeline. They are defined
- and initialized in the [operator](../pkg/operator) package.
+**Operator** components perform data transformations inside the pipeline. They are defined
+ and initialized in the [operator](../operator) package.
+
+#### Iterator
+
+**Iterator** takes an array and executes an operation (defined by a set of nested components)
+on each of its elements.
-The key difference between `connector` and `operator` is that the former will
-connect to an external service, so it's **I/O bound** while the latter is **CPU
-bound**. Connectors don't process but transfer data.
### Recipe
@@ -235,18 +233,18 @@ $ docker exec pipeline-backend go run ./cmd/main
```sh
$ cd $WORKSPACE/component
-$ mkdir -p pkg/operator/hello/v0 && cd $_
+$ mkdir -p operator/hello/v0 && cd $_
```
-Components are isolated in their own packages under `pkg/connector` or
-`pkg/operator`. The package is versioned so, in case a breaking change needs to
+Components are isolated in their own packages under `ai`, `data`, `application` or
+`operator`. The package is versioned so, in case a breaking change needs to
be introduced (e.g. supporting a new major version in a vendor API), existing
-pipelines using the previous version of the connector can keep being triggered.
+pipelines using the previous version of the component can keep being triggered.
At the end of this guide, this will be the structure of the package:
```
-pkg/operator/hello/v0
+operator/hello/v0
├──assets
│ └──hello.svg
├──config
@@ -278,7 +276,7 @@ Create a `config` directory and add the files `definition.json` and
"title": "Hello",
"uid": "e05d3d71-779c-45f8-904d-e90a050ca3b2",
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/hello/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/operator/hello/v0",
"description": "'Hello, world' operator used as a template for adding components",
"release_stage": "RELEASE_STAGE_ALPHA"
}
@@ -293,7 +291,7 @@ This file defines the component properties:
should be written in imperative tense.
- `spec` contains the parameters required to configure the component and that
are independent from its tasks. E.g., the API token of a vendor. In general,
- only connectors need such parameters.
+ only AI, data or application components need such parameters.
- `available_tasks` defines the tasks the component can perform.
- When a component is created in a pipeline, one of the tasks has to be
selected, i.e., a configured component can only execute one task.
@@ -389,7 +387,7 @@ of a pipeline when configured to use this operator.
### Implement the component interfaces
Pipeline communicates with components through the `IComponent`, `IConnector`,
-`IOperator` and `IExecution` interfaces, defined in the [`base`](../pkg/base)
+`IOperator` and `IExecution` interfaces, defined in the [`base`](../base)
package. This package also defines base implementations for these interfaces, so
the `hello` component will only need to override the following methods:
- `CreateExecution(vars map[string]any, task string) (*ExecutionWrapper, error)`
@@ -400,7 +398,7 @@ the `hello` component will only need to override the following methods:
important function in the component. All the data manipulation will take place
here.
-Paste the following code into a `main.go` file in `pkg/operator/hello/v0`:
+Paste the following code into a `main.go` file in `operator/hello/v0`:
```go
package hello
@@ -413,7 +411,7 @@ import (
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
const (
@@ -590,31 +588,31 @@ func TestOperator_CreateExecution(t *testing.T) {
The last step before being able to use the component in VDP is loading the
`hello` operator. This is done in the `Init` function in
-[`pkg/operator/main.go`](../pkg/operator/main.go):
+[`component.go`](../component.go):
```go
package operator
import (
// ...
- "github.com/instill-ai/component/pkg/operator/hello/v0"
+ "github.com/instill-ai/component/operator/hello/v0"
)
// ...
-func Init(logger *zap.Logger) *OperatorStore {
+func Init(logger *zap.Logger) *Store {
baseOp := base.BaseOperator{Logger: logger}
once.Do(func() {
- opStore = &OperatorStore{
+ store = &Store{
operatorUIDMap: map[uuid.UUID]*operator{},
operatorIDMap: map[string]*operator{},
}
// ...
- opStore.Import(hello.Init(baseOp))
+ store.ImportOperator(hello.Init(baseOp))
})
- return opStore
+ return store
}
```
@@ -693,7 +691,7 @@ contain useful descriptions. The information described in `definition.json` and
`tasks.json` is enough to understand how a component should be used. `compogen`
is a tool that parses the component configuration and builds a `README.mdx` file
document displaying its information in a human-readable way. To generate the
-document, just add the following line on top of `pkg/operator/hello/v0/main.go`:
+document, just add the following line on top of `operator/hello/v0/main.go`:
```go
//go:generate compogen readme --operator ./config ./README.mdx
@@ -717,7 +715,7 @@ its version should change following the Semantic Versioning guidelines.
or a new input field with a default value.
- Major versions are intended for backwards-incompatible changes.
- At this point, since there might be pipelines using the previous version, a
- new package MUST be created. E.g., `operator/pkg/json/v0` -> `operator/pkg/json/v1`.
+ new package MUST be created. E.g., `operator/json/v0` -> `operator/json/v1`.
- Build and pre-release labels are discouraged, as components are shipped as
part of Instill VDP and they aren't likely to need such fine-grained version
control.
diff --git a/pkg/connector/archetypeai/v0/README.mdx b/ai/archetypeai/v0/README.mdx
similarity index 95%
rename from pkg/connector/archetypeai/v0/README.mdx
rename to ai/archetypeai/v0/README.mdx
index 87186918f..bebe92b1e 100644
--- a/pkg/connector/archetypeai/v0/README.mdx
+++ b/ai/archetypeai/v0/README.mdx
@@ -18,7 +18,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/archetypeai/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/archetypeai/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/archetypeai/v0/assets/archetype-ai.svg b/ai/archetypeai/v0/assets/archetype-ai.svg
similarity index 100%
rename from pkg/connector/archetypeai/v0/assets/archetype-ai.svg
rename to ai/archetypeai/v0/assets/archetype-ai.svg
diff --git a/pkg/connector/archetypeai/v0/client.go b/ai/archetypeai/v0/client.go
similarity index 90%
rename from pkg/connector/archetypeai/v0/client.go
rename to ai/archetypeai/v0/client.go
index d528c8b0d..12e2c7e32 100644
--- a/pkg/connector/archetypeai/v0/client.go
+++ b/ai/archetypeai/v0/client.go
@@ -1,7 +1,7 @@
package archetypeai
import (
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/internal/util/httpclient"
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
)
diff --git a/pkg/connector/archetypeai/v0/config/definition.json b/ai/archetypeai/v0/config/definition.json
similarity index 97%
rename from pkg/connector/archetypeai/v0/config/definition.json
rename to ai/archetypeai/v0/config/definition.json
index 1e21687f6..1184ac20b 100644
--- a/pkg/connector/archetypeai/v0/config/definition.json
+++ b/ai/archetypeai/v0/config/definition.json
@@ -47,6 +47,6 @@
"vendor": "Archetype AI",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/archetypeai/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/ai/archetypeai/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/archetypeai/v0/config/tasks.json b/ai/archetypeai/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/archetypeai/v0/config/tasks.json
rename to ai/archetypeai/v0/config/tasks.json
diff --git a/pkg/connector/archetypeai/v0/connector_test.go b/ai/archetypeai/v0/connector_test.go
similarity index 98%
rename from pkg/connector/archetypeai/v0/connector_test.go
rename to ai/archetypeai/v0/connector_test.go
index 3aba38673..ec72f22b2 100644
--- a/pkg/connector/archetypeai/v0/connector_test.go
+++ b/ai/archetypeai/v0/connector_test.go
@@ -9,8 +9,8 @@ import (
"testing"
qt "github.com/frankban/quicktest"
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/base"
+ "github.com/instill-ai/component/internal/util/httpclient"
"github.com/instill-ai/x/errmsg"
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
diff --git a/pkg/connector/archetypeai/v0/main.go b/ai/archetypeai/v0/main.go
similarity index 97%
rename from pkg/connector/archetypeai/v0/main.go
rename to ai/archetypeai/v0/main.go
index e827a91b0..363330bc9 100644
--- a/pkg/connector/archetypeai/v0/main.go
+++ b/ai/archetypeai/v0/main.go
@@ -12,9 +12,9 @@ import (
"google.golang.org/protobuf/types/known/structpb"
"github.com/gofrs/uuid"
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/util"
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/base"
+ "github.com/instill-ai/component/internal/util"
+ "github.com/instill-ai/component/internal/util/httpclient"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/archetypeai/v0/structs.go b/ai/archetypeai/v0/structs.go
similarity index 100%
rename from pkg/connector/archetypeai/v0/structs.go
rename to ai/archetypeai/v0/structs.go
diff --git a/pkg/connector/huggingface/v0/README.mdx b/ai/huggingface/v0/README.mdx
similarity index 99%
rename from pkg/connector/huggingface/v0/README.mdx
rename to ai/huggingface/v0/README.mdx
index ea2f8f8a4..60f620f19 100644
--- a/pkg/connector/huggingface/v0/README.mdx
+++ b/ai/huggingface/v0/README.mdx
@@ -32,7 +32,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/huggingface/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/huggingface/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/huggingface/v0/assets/hugging-face.svg b/ai/huggingface/v0/assets/hugging-face.svg
similarity index 100%
rename from pkg/connector/huggingface/v0/assets/hugging-face.svg
rename to ai/huggingface/v0/assets/hugging-face.svg
diff --git a/pkg/connector/huggingface/v0/client.go b/ai/huggingface/v0/client.go
similarity index 94%
rename from pkg/connector/huggingface/v0/client.go
rename to ai/huggingface/v0/client.go
index e23fa51c9..bece2d8ae 100644
--- a/pkg/connector/huggingface/v0/client.go
+++ b/ai/huggingface/v0/client.go
@@ -6,7 +6,7 @@ import (
"strings"
"github.com/go-resty/resty/v2"
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/internal/util/httpclient"
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
)
diff --git a/pkg/connector/huggingface/v0/config/definition.json b/ai/huggingface/v0/config/definition.json
similarity index 98%
rename from pkg/connector/huggingface/v0/config/definition.json
rename to ai/huggingface/v0/config/definition.json
index 9e6ae0d21..417cd7216 100644
--- a/pkg/connector/huggingface/v0/config/definition.json
+++ b/ai/huggingface/v0/config/definition.json
@@ -93,6 +93,6 @@
"vendor": "Hugging Face",
"vendor_attributes": {},
"version": "0.1.1",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/huggingface/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/ai/huggingface/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/huggingface/v0/config/tasks.json b/ai/huggingface/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/huggingface/v0/config/tasks.json
rename to ai/huggingface/v0/config/tasks.json
diff --git a/pkg/connector/huggingface/v0/connector_test.go b/ai/huggingface/v0/connector_test.go
similarity index 98%
rename from pkg/connector/huggingface/v0/connector_test.go
rename to ai/huggingface/v0/connector_test.go
index 9acd3d6a0..ef3435ce6 100644
--- a/pkg/connector/huggingface/v0/connector_test.go
+++ b/ai/huggingface/v0/connector_test.go
@@ -13,8 +13,8 @@ import (
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/base"
+ "github.com/instill-ai/component/internal/util/httpclient"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/huggingface/v0/main.go b/ai/huggingface/v0/main.go
similarity index 99%
rename from pkg/connector/huggingface/v0/main.go
rename to ai/huggingface/v0/main.go
index 9c48d0e0f..b764a4fdc 100644
--- a/pkg/connector/huggingface/v0/main.go
+++ b/ai/huggingface/v0/main.go
@@ -12,7 +12,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/huggingface/v0/structs.go b/ai/huggingface/v0/structs.go
similarity index 100%
rename from pkg/connector/huggingface/v0/structs.go
rename to ai/huggingface/v0/structs.go
diff --git a/pkg/connector/instill/v0/README.mdx b/ai/instill/v0/README.mdx
similarity index 99%
rename from pkg/connector/instill/v0/README.mdx
rename to ai/instill/v0/README.mdx
index 1e833ff4f..163e518ff 100644
--- a/pkg/connector/instill/v0/README.mdx
+++ b/ai/instill/v0/README.mdx
@@ -26,7 +26,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/instill/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/instill/v0/config/definition.json).
## Supported Tasks
diff --git a/pkg/connector/instill/v0/assets/instill-model.svg b/ai/instill/v0/assets/instill-model.svg
similarity index 100%
rename from pkg/connector/instill/v0/assets/instill-model.svg
rename to ai/instill/v0/assets/instill-model.svg
diff --git a/pkg/connector/instill/v0/client.go b/ai/instill/v0/client.go
similarity index 100%
rename from pkg/connector/instill/v0/client.go
rename to ai/instill/v0/client.go
diff --git a/pkg/connector/instill/v0/config/definition.json b/ai/instill/v0/config/definition.json
similarity index 96%
rename from pkg/connector/instill/v0/config/definition.json
rename to ai/instill/v0/config/definition.json
index c375661e2..c4654f0a0 100644
--- a/pkg/connector/instill/v0/config/definition.json
+++ b/ai/instill/v0/config/definition.json
@@ -26,6 +26,6 @@
"vendor": "Instill",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/instill/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/ai/instill/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/instill/v0/config/tasks.json b/ai/instill/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/instill/v0/config/tasks.json
rename to ai/instill/v0/config/tasks.json
diff --git a/pkg/connector/instill/v0/image_classification.go b/ai/instill/v0/image_classification.go
similarity index 98%
rename from pkg/connector/instill/v0/image_classification.go
rename to ai/instill/v0/image_classification.go
index 76c5cd728..74df83176 100644
--- a/pkg/connector/instill/v0/image_classification.go
+++ b/ai/instill/v0/image_classification.go
@@ -8,7 +8,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha"
)
diff --git a/pkg/connector/instill/v0/image_to_image.go b/ai/instill/v0/image_to_image.go
similarity index 98%
rename from pkg/connector/instill/v0/image_to_image.go
rename to ai/instill/v0/image_to_image.go
index 33ba1e549..fc36a6a2b 100644
--- a/pkg/connector/instill/v0/image_to_image.go
+++ b/ai/instill/v0/image_to_image.go
@@ -8,7 +8,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha"
)
diff --git a/pkg/connector/instill/v0/instance_segmentation.go b/ai/instill/v0/instance_segmentation.go
similarity index 98%
rename from pkg/connector/instill/v0/instance_segmentation.go
rename to ai/instill/v0/instance_segmentation.go
index 83b17118b..52e04ba5e 100644
--- a/pkg/connector/instill/v0/instance_segmentation.go
+++ b/ai/instill/v0/instance_segmentation.go
@@ -8,7 +8,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha"
)
diff --git a/pkg/connector/instill/v0/keypoint_detection.go b/ai/instill/v0/keypoint_detection.go
similarity index 98%
rename from pkg/connector/instill/v0/keypoint_detection.go
rename to ai/instill/v0/keypoint_detection.go
index 30be7546a..26fda96bd 100644
--- a/pkg/connector/instill/v0/keypoint_detection.go
+++ b/ai/instill/v0/keypoint_detection.go
@@ -8,7 +8,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha"
)
diff --git a/pkg/connector/instill/v0/llm_utils.go b/ai/instill/v0/llm_utils.go
similarity index 98%
rename from pkg/connector/instill/v0/llm_utils.go
rename to ai/instill/v0/llm_utils.go
index afaf7d10b..ea7e4185d 100644
--- a/pkg/connector/instill/v0/llm_utils.go
+++ b/ai/instill/v0/llm_utils.go
@@ -3,7 +3,7 @@ package instill
import (
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha"
)
diff --git a/pkg/connector/instill/v0/main.go b/ai/instill/v0/main.go
similarity index 99%
rename from pkg/connector/instill/v0/main.go
rename to ai/instill/v0/main.go
index 8fd21f205..e21e33178 100644
--- a/pkg/connector/instill/v0/main.go
+++ b/ai/instill/v0/main.go
@@ -13,7 +13,7 @@ import (
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
commonPB "github.com/instill-ai/protogen-go/common/task/v1alpha"
mgmtPB "github.com/instill-ai/protogen-go/core/mgmt/v1beta"
diff --git a/pkg/connector/instill/v0/object_detection.go b/ai/instill/v0/object_detection.go
similarity index 98%
rename from pkg/connector/instill/v0/object_detection.go
rename to ai/instill/v0/object_detection.go
index a7984d8dd..e144878d5 100644
--- a/pkg/connector/instill/v0/object_detection.go
+++ b/ai/instill/v0/object_detection.go
@@ -8,7 +8,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha"
)
diff --git a/pkg/connector/instill/v0/ocr.go b/ai/instill/v0/ocr.go
similarity index 100%
rename from pkg/connector/instill/v0/ocr.go
rename to ai/instill/v0/ocr.go
diff --git a/pkg/connector/instill/v0/semantic_segmentation.go b/ai/instill/v0/semantic_segmentation.go
similarity index 98%
rename from pkg/connector/instill/v0/semantic_segmentation.go
rename to ai/instill/v0/semantic_segmentation.go
index 1239dd66b..5c4f03153 100644
--- a/pkg/connector/instill/v0/semantic_segmentation.go
+++ b/ai/instill/v0/semantic_segmentation.go
@@ -8,7 +8,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha"
)
diff --git a/pkg/connector/instill/v0/text_generation.go b/ai/instill/v0/text_generation.go
similarity index 100%
rename from pkg/connector/instill/v0/text_generation.go
rename to ai/instill/v0/text_generation.go
diff --git a/pkg/connector/instill/v0/text_generation_chat.go b/ai/instill/v0/text_generation_chat.go
similarity index 100%
rename from pkg/connector/instill/v0/text_generation_chat.go
rename to ai/instill/v0/text_generation_chat.go
diff --git a/pkg/connector/instill/v0/text_to_image.go b/ai/instill/v0/text_to_image.go
similarity index 98%
rename from pkg/connector/instill/v0/text_to_image.go
rename to ai/instill/v0/text_to_image.go
index 0d22732e5..b29a47279 100644
--- a/pkg/connector/instill/v0/text_to_image.go
+++ b/ai/instill/v0/text_to_image.go
@@ -8,7 +8,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
modelPB "github.com/instill-ai/protogen-go/model/model/v1alpha"
)
diff --git a/pkg/connector/instill/v0/unspecified.go b/ai/instill/v0/unspecified.go
similarity index 100%
rename from pkg/connector/instill/v0/unspecified.go
rename to ai/instill/v0/unspecified.go
diff --git a/pkg/connector/instill/v0/visual_question_answering.go b/ai/instill/v0/visual_question_answering.go
similarity index 100%
rename from pkg/connector/instill/v0/visual_question_answering.go
rename to ai/instill/v0/visual_question_answering.go
diff --git a/pkg/connector/integration_test.go b/ai/integration_test.go
similarity index 93%
rename from pkg/connector/integration_test.go
rename to ai/integration_test.go
index f1c73186f..da3a3b6d2 100644
--- a/pkg/connector/integration_test.go
+++ b/ai/integration_test.go
@@ -1,7 +1,7 @@
//go:build integration
// +build integration
-package pkg
+package ai
import (
"testing"
@@ -11,8 +11,8 @@ import (
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/openai/v0"
+ "github.com/instill-ai/component/ai/openai/v0"
+ "github.com/instill-ai/component/base"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/openai/v0/README.mdx b/ai/openai/v0/README.mdx
similarity index 98%
rename from pkg/connector/openai/v0/README.mdx
rename to ai/openai/v0/README.mdx
index 19361a4b0..61ab569b8 100644
--- a/pkg/connector/openai/v0/README.mdx
+++ b/ai/openai/v0/README.mdx
@@ -20,7 +20,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/openai/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/openai/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/openai/v0/assets/openai.svg b/ai/openai/v0/assets/openai.svg
similarity index 100%
rename from pkg/connector/openai/v0/assets/openai.svg
rename to ai/openai/v0/assets/openai.svg
diff --git a/pkg/connector/openai/v0/audio_transcriptions.go b/ai/openai/v0/audio_transcriptions.go
similarity index 96%
rename from pkg/connector/openai/v0/audio_transcriptions.go
rename to ai/openai/v0/audio_transcriptions.go
index 391748f9b..fb3ab6dfc 100644
--- a/pkg/connector/openai/v0/audio_transcriptions.go
+++ b/ai/openai/v0/audio_transcriptions.go
@@ -5,7 +5,7 @@ import (
"fmt"
"mime/multipart"
- "github.com/instill-ai/component/pkg/connector/util"
+ "github.com/instill-ai/component/internal/util"
)
const (
diff --git a/pkg/connector/openai/v0/client.go b/ai/openai/v0/client.go
similarity index 95%
rename from pkg/connector/openai/v0/client.go
rename to ai/openai/v0/client.go
index c8b40b2f1..53ef63b1b 100644
--- a/pkg/connector/openai/v0/client.go
+++ b/ai/openai/v0/client.go
@@ -1,7 +1,7 @@
package openai
import (
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/internal/util/httpclient"
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
)
diff --git a/pkg/connector/openai/v0/config/definition.json b/ai/openai/v0/config/definition.json
similarity index 98%
rename from pkg/connector/openai/v0/config/definition.json
rename to ai/openai/v0/config/definition.json
index 577f8f495..a7d3cc28e 100644
--- a/pkg/connector/openai/v0/config/definition.json
+++ b/ai/openai/v0/config/definition.json
@@ -61,6 +61,6 @@
"vendor": "OpenAI",
"vendor_attributes": {},
"version": "0.1.1",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/openai/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/ai/openai/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/openai/v0/config/openai.json b/ai/openai/v0/config/openai.json
similarity index 100%
rename from pkg/connector/openai/v0/config/openai.json
rename to ai/openai/v0/config/openai.json
diff --git a/pkg/connector/openai/v0/config/tasks.json b/ai/openai/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/openai/v0/config/tasks.json
rename to ai/openai/v0/config/tasks.json
diff --git a/pkg/connector/openai/v0/connector_test.go b/ai/openai/v0/connector_test.go
similarity index 98%
rename from pkg/connector/openai/v0/connector_test.go
rename to ai/openai/v0/connector_test.go
index dd917cd15..8f4a7f430 100644
--- a/pkg/connector/openai/v0/connector_test.go
+++ b/ai/openai/v0/connector_test.go
@@ -13,9 +13,9 @@ import (
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
- "github.com/instill-ai/component/pkg/mock"
+ "github.com/instill-ai/component/base"
+ "github.com/instill-ai/component/internal/mock"
+ "github.com/instill-ai/component/internal/util/httpclient"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/openai/v0/list_models.go b/ai/openai/v0/list_models.go
similarity index 100%
rename from pkg/connector/openai/v0/list_models.go
rename to ai/openai/v0/list_models.go
diff --git a/pkg/connector/openai/v0/main.go b/ai/openai/v0/main.go
similarity index 99%
rename from pkg/connector/openai/v0/main.go
rename to ai/openai/v0/main.go
index d0db37d09..b9f276885 100644
--- a/pkg/connector/openai/v0/main.go
+++ b/ai/openai/v0/main.go
@@ -13,7 +13,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/openai/v0/text_embeddings.go b/ai/openai/v0/text_embeddings.go
similarity index 100%
rename from pkg/connector/openai/v0/text_embeddings.go
rename to ai/openai/v0/text_embeddings.go
diff --git a/pkg/connector/openai/v0/text_generation.go b/ai/openai/v0/text_generation.go
similarity index 100%
rename from pkg/connector/openai/v0/text_generation.go
rename to ai/openai/v0/text_generation.go
diff --git a/pkg/connector/openai/v0/text_to_image.go b/ai/openai/v0/text_to_image.go
similarity index 100%
rename from pkg/connector/openai/v0/text_to_image.go
rename to ai/openai/v0/text_to_image.go
diff --git a/pkg/connector/openai/v0/text_to_speech.go b/ai/openai/v0/text_to_speech.go
similarity index 100%
rename from pkg/connector/openai/v0/text_to_speech.go
rename to ai/openai/v0/text_to_speech.go
diff --git a/pkg/connector/stabilityai/v0/README.mdx b/ai/stabilityai/v0/README.mdx
similarity index 98%
rename from pkg/connector/stabilityai/v0/README.mdx
rename to ai/stabilityai/v0/README.mdx
index 37cd25190..d1b5dca62 100644
--- a/pkg/connector/stabilityai/v0/README.mdx
+++ b/ai/stabilityai/v0/README.mdx
@@ -17,7 +17,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/stabilityai/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/stabilityai/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/stabilityai/v0/assets/stability-ai.svg b/ai/stabilityai/v0/assets/stability-ai.svg
similarity index 100%
rename from pkg/connector/stabilityai/v0/assets/stability-ai.svg
rename to ai/stabilityai/v0/assets/stability-ai.svg
diff --git a/pkg/connector/stabilityai/v0/client.go b/ai/stabilityai/v0/client.go
similarity index 94%
rename from pkg/connector/stabilityai/v0/client.go
rename to ai/stabilityai/v0/client.go
index c1c16a03a..e3b69b3bd 100644
--- a/pkg/connector/stabilityai/v0/client.go
+++ b/ai/stabilityai/v0/client.go
@@ -1,7 +1,7 @@
package stabilityai
import (
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/internal/util/httpclient"
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
)
diff --git a/pkg/connector/stabilityai/v0/config/definition.json b/ai/stabilityai/v0/config/definition.json
similarity index 97%
rename from pkg/connector/stabilityai/v0/config/definition.json
rename to ai/stabilityai/v0/config/definition.json
index e2f94a522..4a136d55c 100644
--- a/pkg/connector/stabilityai/v0/config/definition.json
+++ b/ai/stabilityai/v0/config/definition.json
@@ -47,6 +47,6 @@
"vendor": "Stability AI",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/stabilityai/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/ai/stabilityai/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/stabilityai/v0/config/stabilityai.json b/ai/stabilityai/v0/config/stabilityai.json
similarity index 100%
rename from pkg/connector/stabilityai/v0/config/stabilityai.json
rename to ai/stabilityai/v0/config/stabilityai.json
diff --git a/pkg/connector/stabilityai/v0/config/tasks.json b/ai/stabilityai/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/stabilityai/v0/config/tasks.json
rename to ai/stabilityai/v0/config/tasks.json
diff --git a/pkg/connector/stabilityai/v0/connector_test.go b/ai/stabilityai/v0/connector_test.go
similarity index 98%
rename from pkg/connector/stabilityai/v0/connector_test.go
rename to ai/stabilityai/v0/connector_test.go
index f95568f35..f46fa2058 100644
--- a/pkg/connector/stabilityai/v0/connector_test.go
+++ b/ai/stabilityai/v0/connector_test.go
@@ -12,8 +12,8 @@ import (
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/base"
+ "github.com/instill-ai/component/internal/util/httpclient"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/stabilityai/v0/image_to_image.go b/ai/stabilityai/v0/image_to_image.go
similarity index 98%
rename from pkg/connector/stabilityai/v0/image_to_image.go
rename to ai/stabilityai/v0/image_to_image.go
index 98dc3da12..a9679d47b 100644
--- a/pkg/connector/stabilityai/v0/image_to_image.go
+++ b/ai/stabilityai/v0/image_to_image.go
@@ -5,8 +5,8 @@ import (
"fmt"
"mime/multipart"
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/util"
+ "github.com/instill-ai/component/base"
+ "github.com/instill-ai/component/internal/util"
"google.golang.org/protobuf/types/known/structpb"
)
diff --git a/pkg/connector/stabilityai/v0/list_engines.go b/ai/stabilityai/v0/list_engines.go
similarity index 100%
rename from pkg/connector/stabilityai/v0/list_engines.go
rename to ai/stabilityai/v0/list_engines.go
diff --git a/pkg/connector/stabilityai/v0/main.go b/ai/stabilityai/v0/main.go
similarity index 99%
rename from pkg/connector/stabilityai/v0/main.go
rename to ai/stabilityai/v0/main.go
index 778549f3b..881333d5a 100644
--- a/pkg/connector/stabilityai/v0/main.go
+++ b/ai/stabilityai/v0/main.go
@@ -9,7 +9,7 @@ import (
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/stabilityai/v0/text_to_image.go b/ai/stabilityai/v0/text_to_image.go
similarity index 98%
rename from pkg/connector/stabilityai/v0/text_to_image.go
rename to ai/stabilityai/v0/text_to_image.go
index b2b87dfae..020f4a676 100644
--- a/pkg/connector/stabilityai/v0/text_to_image.go
+++ b/ai/stabilityai/v0/text_to_image.go
@@ -3,7 +3,7 @@ package stabilityai
import (
"fmt"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
"google.golang.org/protobuf/types/known/structpb"
)
diff --git a/pkg/connector/googlesearch/v0/README.mdx b/application/googlesearch/v0/README.mdx
similarity index 95%
rename from pkg/connector/googlesearch/v0/README.mdx
rename to application/googlesearch/v0/README.mdx
index e86885123..975a5247e 100644
--- a/pkg/connector/googlesearch/v0/README.mdx
+++ b/application/googlesearch/v0/README.mdx
@@ -16,7 +16,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/googlesearch/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/googlesearch/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/googlesearch/v0/assets/google-search.svg b/application/googlesearch/v0/assets/google-search.svg
similarity index 100%
rename from pkg/connector/googlesearch/v0/assets/google-search.svg
rename to application/googlesearch/v0/assets/google-search.svg
diff --git a/pkg/connector/googlesearch/v0/config/definition.json b/application/googlesearch/v0/config/definition.json
similarity index 98%
rename from pkg/connector/googlesearch/v0/config/definition.json
rename to application/googlesearch/v0/config/definition.json
index 8357116e3..ee9054f6b 100644
--- a/pkg/connector/googlesearch/v0/config/definition.json
+++ b/application/googlesearch/v0/config/definition.json
@@ -54,6 +54,6 @@
"vendor": "Google",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/googlesearch/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/application/googlesearch/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/googlesearch/v0/config/tasks.json b/application/googlesearch/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/googlesearch/v0/config/tasks.json
rename to application/googlesearch/v0/config/tasks.json
diff --git a/pkg/connector/googlesearch/v0/main.go b/application/googlesearch/v0/main.go
similarity index 98%
rename from pkg/connector/googlesearch/v0/main.go
rename to application/googlesearch/v0/main.go
index 3a18d07e3..ab0b95cce 100644
--- a/pkg/connector/googlesearch/v0/main.go
+++ b/application/googlesearch/v0/main.go
@@ -12,7 +12,7 @@ import (
"google.golang.org/api/option"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
const (
diff --git a/pkg/connector/googlesearch/v0/search.go b/application/googlesearch/v0/search.go
similarity index 98%
rename from pkg/connector/googlesearch/v0/search.go
rename to application/googlesearch/v0/search.go
index 630c1d4b2..559c5ebd5 100644
--- a/pkg/connector/googlesearch/v0/search.go
+++ b/application/googlesearch/v0/search.go
@@ -8,7 +8,7 @@ import (
"github.com/PuerkitoBio/goquery"
"google.golang.org/api/customsearch/v1"
- "github.com/instill-ai/component/pkg/connector/util"
+ "github.com/instill-ai/component/internal/util"
)
const (
diff --git a/pkg/connector/numbers/v0/README.mdx b/application/numbers/v0/README.mdx
similarity index 95%
rename from pkg/connector/numbers/v0/README.mdx
rename to application/numbers/v0/README.mdx
index f3be5bffb..2170dc40b 100644
--- a/pkg/connector/numbers/v0/README.mdx
+++ b/application/numbers/v0/README.mdx
@@ -16,7 +16,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/numbers/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/numbers/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/numbers/v0/assets/numbers.svg b/application/numbers/v0/assets/numbers.svg
similarity index 100%
rename from pkg/connector/numbers/v0/assets/numbers.svg
rename to application/numbers/v0/assets/numbers.svg
diff --git a/pkg/connector/numbers/v0/config/definition.json b/application/numbers/v0/config/definition.json
similarity index 98%
rename from pkg/connector/numbers/v0/config/definition.json
rename to application/numbers/v0/config/definition.json
index 4f526778e..b27942bab 100644
--- a/pkg/connector/numbers/v0/config/definition.json
+++ b/application/numbers/v0/config/definition.json
@@ -45,6 +45,6 @@
"vendor": "Numbers Protocol",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/numbers/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/application/numbers/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/numbers/v0/config/tasks.json b/application/numbers/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/numbers/v0/config/tasks.json
rename to application/numbers/v0/config/tasks.json
diff --git a/pkg/connector/numbers/v0/main.go b/application/numbers/v0/main.go
similarity index 99%
rename from pkg/connector/numbers/v0/main.go
rename to application/numbers/v0/main.go
index 14e8bc66f..6e7b316ef 100644
--- a/pkg/connector/numbers/v0/main.go
+++ b/application/numbers/v0/main.go
@@ -18,7 +18,7 @@ import (
"github.com/gofrs/uuid"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
const urlRegisterAsset = "https://api.numbersprotocol.io/api/v3/assets/"
diff --git a/pkg/connector/redis/v0/README.mdx b/application/redis/v0/README.mdx
similarity index 97%
rename from pkg/connector/redis/v0/README.mdx
rename to application/redis/v0/README.mdx
index 126369b44..54862287a 100644
--- a/pkg/connector/redis/v0/README.mdx
+++ b/application/redis/v0/README.mdx
@@ -18,7 +18,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/redis/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/redis/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/redis/v0/assets/redis.svg b/application/redis/v0/assets/redis.svg
similarity index 100%
rename from pkg/connector/redis/v0/assets/redis.svg
rename to application/redis/v0/assets/redis.svg
diff --git a/pkg/connector/redis/v0/chat_history.go b/application/redis/v0/chat_history.go
similarity index 100%
rename from pkg/connector/redis/v0/chat_history.go
rename to application/redis/v0/chat_history.go
diff --git a/pkg/connector/redis/v0/client.go b/application/redis/v0/client.go
similarity index 98%
rename from pkg/connector/redis/v0/client.go
rename to application/redis/v0/client.go
index 95cedf460..355c89e86 100644
--- a/pkg/connector/redis/v0/client.go
+++ b/application/redis/v0/client.go
@@ -5,7 +5,7 @@ import (
"crypto/x509"
"fmt"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
goredis "github.com/redis/go-redis/v9"
"google.golang.org/protobuf/types/known/structpb"
)
diff --git a/pkg/connector/redis/v0/config/definition.json b/application/redis/v0/config/definition.json
similarity index 99%
rename from pkg/connector/redis/v0/config/definition.json
rename to application/redis/v0/config/definition.json
index fd6423276..693ab165a 100644
--- a/pkg/connector/redis/v0/config/definition.json
+++ b/application/redis/v0/config/definition.json
@@ -208,6 +208,6 @@
"vendor": "Redis Labs",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/redis/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/application/redis/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/redis/v0/config/tasks.json b/application/redis/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/redis/v0/config/tasks.json
rename to application/redis/v0/config/tasks.json
diff --git a/pkg/connector/redis/v0/main.go b/application/redis/v0/main.go
similarity index 98%
rename from pkg/connector/redis/v0/main.go
rename to application/redis/v0/main.go
index 494824587..28098d584 100644
--- a/pkg/connector/redis/v0/main.go
+++ b/application/redis/v0/main.go
@@ -9,7 +9,7 @@ import (
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
const (
diff --git a/pkg/connector/restapi/v0/README.mdx b/application/restapi/v0/README.mdx
similarity index 98%
rename from pkg/connector/restapi/v0/README.mdx
rename to application/restapi/v0/README.mdx
index 3457d8c27..96e7cdeb5 100644
--- a/pkg/connector/restapi/v0/README.mdx
+++ b/application/restapi/v0/README.mdx
@@ -22,7 +22,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/restapi/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/restapi/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/restapi/v0/assets/restapi.svg b/application/restapi/v0/assets/restapi.svg
similarity index 100%
rename from pkg/connector/restapi/v0/assets/restapi.svg
rename to application/restapi/v0/assets/restapi.svg
diff --git a/pkg/connector/restapi/v0/auth.go b/application/restapi/v0/auth.go
similarity index 96%
rename from pkg/connector/restapi/v0/auth.go
rename to application/restapi/v0/auth.go
index 7d6ecc8ab..1b67fc659 100644
--- a/pkg/connector/restapi/v0/auth.go
+++ b/application/restapi/v0/auth.go
@@ -3,7 +3,7 @@ package restapi
import (
"fmt"
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/internal/util/httpclient"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/restapi/v0/client.go b/application/restapi/v0/client.go
similarity index 91%
rename from pkg/connector/restapi/v0/client.go
rename to application/restapi/v0/client.go
index 264522021..401829dd2 100644
--- a/pkg/connector/restapi/v0/client.go
+++ b/application/restapi/v0/client.go
@@ -1,7 +1,7 @@
package restapi
import (
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/internal/util/httpclient"
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
)
diff --git a/pkg/connector/restapi/v0/config/definition.json b/application/restapi/v0/config/definition.json
similarity index 99%
rename from pkg/connector/restapi/v0/config/definition.json
rename to application/restapi/v0/config/definition.json
index d3b88850e..d58a6e8db 100644
--- a/pkg/connector/restapi/v0/config/definition.json
+++ b/application/restapi/v0/config/definition.json
@@ -197,6 +197,6 @@
"vendor": "",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/restapi/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/application/restapi/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/restapi/v0/config/tasks.json b/application/restapi/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/restapi/v0/config/tasks.json
rename to application/restapi/v0/config/tasks.json
diff --git a/pkg/connector/restapi/v0/connector_test.go b/application/restapi/v0/connector_test.go
similarity index 97%
rename from pkg/connector/restapi/v0/connector_test.go
rename to application/restapi/v0/connector_test.go
index c14ea9a29..9d3fafe6a 100644
--- a/pkg/connector/restapi/v0/connector_test.go
+++ b/application/restapi/v0/connector_test.go
@@ -13,8 +13,8 @@ import (
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/base"
+ "github.com/instill-ai/component/internal/util/httpclient"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/restapi/v0/main.go b/application/restapi/v0/main.go
similarity index 99%
rename from pkg/connector/restapi/v0/main.go
rename to application/restapi/v0/main.go
index 12886b81e..d6f63f733 100644
--- a/pkg/connector/restapi/v0/main.go
+++ b/application/restapi/v0/main.go
@@ -10,7 +10,7 @@ import (
"net/http"
"sync"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
"github.com/instill-ai/x/errmsg"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/structpb"
diff --git a/pkg/connector/slack/v0/README.mdx b/application/slack/v0/README.mdx
similarity index 95%
rename from pkg/connector/slack/v0/README.mdx
rename to application/slack/v0/README.mdx
index 9f2dca30a..30b4aba4c 100644
--- a/pkg/connector/slack/v0/README.mdx
+++ b/application/slack/v0/README.mdx
@@ -17,7 +17,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/slack/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/slack/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/slack/v0/apiFunctions.go b/application/slack/v0/apiFunctions.go
similarity index 100%
rename from pkg/connector/slack/v0/apiFunctions.go
rename to application/slack/v0/apiFunctions.go
diff --git a/pkg/connector/slack/v0/assets/slack.svg b/application/slack/v0/assets/slack.svg
similarity index 100%
rename from pkg/connector/slack/v0/assets/slack.svg
rename to application/slack/v0/assets/slack.svg
diff --git a/pkg/connector/slack/v0/client.go b/application/slack/v0/client.go
similarity index 100%
rename from pkg/connector/slack/v0/client.go
rename to application/slack/v0/client.go
diff --git a/pkg/connector/slack/v0/config/definition.json b/application/slack/v0/config/definition.json
similarity index 98%
rename from pkg/connector/slack/v0/config/definition.json
rename to application/slack/v0/config/definition.json
index 6f13ecf89..d526309ca 100644
--- a/pkg/connector/slack/v0/config/definition.json
+++ b/application/slack/v0/config/definition.json
@@ -46,6 +46,6 @@
"vendor": "Slack",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/slack/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/application/slack/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/application/slack/v0/config/tasks.json b/application/slack/v0/config/tasks.json
new file mode 100644
index 000000000..e0b0ac627
--- /dev/null
+++ b/application/slack/v0/config/tasks.json
@@ -0,0 +1,239 @@
+{
+ "TASK_READ_MESSAGE": {
+ "instillShortDescription": "Get the latest message since specific date",
+ "input": {
+ "description": "Please input the channel name and the date that we want to start to read",
+ "instillUIOrder": 0,
+ "properties": {
+ "channel_name": {
+ "description": "A channel name display in Slack",
+ "instillAcceptFormats": [
+ "string"
+ ],
+ "instillUIMultiline": true,
+ "instillUIOrder": 0,
+ "instillUpstreamTypes": [
+ "value",
+ "reference",
+ "template"
+ ],
+ "title": "Channel Name",
+ "type": "string"
+ },
+ "start_to_read_date": {
+ "description": "earliest date in all read messages",
+ "instillAcceptFormats": [
+ "string"
+ ],
+ "instillUIMultiline": true,
+ "instillUIOrder": 1,
+ "instillUpstreamTypes": [
+ "value",
+ "reference",
+ "template"
+ ],
+ "title": "Start to read date",
+ "type": "string"
+ },
+ "is_public_channel": {
+ "description": "Whether all members can read the channel",
+ "instillAcceptFormats": [
+ "boolean"
+ ],
+ "instillUIMultiline": true,
+ "instillUIOrder": 2,
+ "instillUpstreamTypes": [
+ "value",
+ "reference",
+ "template"
+ ],
+ "title": "Public channel",
+ "type": "boolean"
+ }
+ },
+ "required": [
+ "channel_name"
+ ],
+ "title": "Input",
+ "type": "object"
+ },
+ "output": {
+ "description": "All messages in Slack channel",
+ "instillUIOrder": 0,
+ "properties": {
+ "conversations": {
+ "description": "An array of conversations with thread messages",
+ "instillUIOrder": 0,
+ "title": "Conversations",
+ "type": "array",
+ "items": {
+ "title": "conversation details",
+ "type": "object",
+ "properties": {
+ "user_id": {
+ "description": "unique id from Slack",
+ "instillFormat": "string",
+ "instillUIOrder": 1,
+ "title": "User UID",
+ "type": "string"
+ },
+ "message": {
+ "description": "message to start a conversation",
+ "instillFormat": "string",
+ "instillUIOrder": 2,
+ "title": "Start Conversation Message",
+ "type": "string"
+ },
+ "start_date": {
+ "description": "when a conversation starts",
+ "instillFormat": "string",
+ "instillUIOrder": 3,
+ "required": [],
+ "title": "Start Date",
+ "type": "string"
+ },
+ "last_date": {
+ "description": "Date of the last message",
+ "instillFormat": "string",
+ "instillUIOrder": 4,
+ "required": [],
+ "title": "Last Date",
+ "type": "string"
+ },
+ "thread_reply_messages": {
+ "description": "replies in a conversation",
+ "instillFormat": "array",
+ "instillUIOrder": 5,
+ "title": "Replied messages",
+ "type": "array",
+ "items": {
+ "title": "relied details",
+ "type": "object",
+ "properties": {
+ "user_id": {
+ "description": "unique id from Slack",
+ "instillFormat": "string",
+ "instillUIOrder": 1,
+ "title": "User UID",
+ "type": "string"
+ },
+ "datetime": {
+ "description": "replied datetime",
+ "instillFormat": "string",
+ "instillUIOrder": 2,
+ "title": "Replied Time",
+ "type": "string"
+ },
+ "message": {
+ "description": "message to reply a conversation",
+ "instillFormat": "string",
+ "instillUIOrder": 3,
+ "title": "Replied Message",
+ "type": "string"
+ }
+ },
+ "required": [
+ "user_id",
+ "datetime",
+ "message"
+ ]
+ }
+ }
+ },
+ "required": [
+ "user_id",
+ "message",
+ "start_date"
+ ]
+ }
+ }
+ },
+ "required": [
+ "conversations"
+ ],
+ "title": "Output",
+ "type": "object"
+ }
+ },
+ "TASK_WRITE_MESSAGE": {
+ "instillShortDescription": "send message to a specific channel",
+ "title": "Send Message",
+ "input": {
+ "description": "Input",
+ "instillUIOrder": 0,
+ "properties": {
+ "channel_name": {
+ "description": "A channel name display in Slack",
+ "instillAcceptFormats": [
+ "string"
+ ],
+ "instillUIMultiline": true,
+ "instillUIOrder": 0,
+ "instillUpstreamTypes": [
+ "value",
+ "reference",
+ "template"
+ ],
+ "title": "Channel Name",
+ "type": "string"
+ },
+ "message": {
+ "description": "message to be sent to the target channel",
+ "instillAcceptFormats": [
+ "string"
+ ],
+ "instillUIMultiline": true,
+ "instillUIOrder": 1,
+ "instillUpstreamTypes": [
+ "value",
+ "reference",
+ "template"
+ ],
+ "title": "Message",
+ "type": "string"
+ },
+ "is_public_channel": {
+ "description": "Whether all members can read the channel",
+ "instillAcceptFormats": [
+ "boolean"
+ ],
+ "instillUIMultiline": true,
+ "instillUIOrder": 2,
+ "instillUpstreamTypes": [
+ "value",
+ "reference",
+ "template"
+ ],
+ "title": "Public channel",
+ "type": "boolean"
+ }
+ },
+ "required": [
+ "channel_name",
+ "message"
+ ],
+ "title": "Input",
+ "type": "object"
+ },
+ "output": {
+ "description": "The greeting sentence",
+ "instillUIOrder": 0,
+ "properties": {
+ "result": {
+ "description": "result for sending message",
+ "instillEditOnNodeFields": [],
+ "instillUIOrder": 0,
+ "required": [],
+ "title": "Result",
+ "type": "string",
+ "instillFormat": "string"
+ }
+ },
+ "required": [
+ "result"
+ ],
+ "title": "Output",
+ "type": "object"
+ }
+ }
+}
diff --git a/pkg/connector/slack/v0/connector_test.go b/application/slack/v0/connector_test.go
similarity index 98%
rename from pkg/connector/slack/v0/connector_test.go
rename to application/slack/v0/connector_test.go
index 8e2e3d48c..76a180517 100644
--- a/pkg/connector/slack/v0/connector_test.go
+++ b/application/slack/v0/connector_test.go
@@ -7,7 +7,7 @@ import (
"time"
qt "github.com/frankban/quicktest"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
)
diff --git a/pkg/connector/slack/v0/main.go b/application/slack/v0/main.go
similarity index 98%
rename from pkg/connector/slack/v0/main.go
rename to application/slack/v0/main.go
index a2085210f..bf8e1f048 100644
--- a/pkg/connector/slack/v0/main.go
+++ b/application/slack/v0/main.go
@@ -9,7 +9,7 @@ import (
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
"github.com/instill-ai/x/errmsg"
"github.com/slack-go/slack"
)
diff --git a/pkg/connector/slack/v0/mockSlack.go b/application/slack/v0/mockSlack.go
similarity index 100%
rename from pkg/connector/slack/v0/mockSlack.go
rename to application/slack/v0/mockSlack.go
diff --git a/pkg/connector/slack/v0/structs.go b/application/slack/v0/structs.go
similarity index 100%
rename from pkg/connector/slack/v0/structs.go
rename to application/slack/v0/structs.go
diff --git a/pkg/connector/slack/v0/taskFunctions.go b/application/slack/v0/taskFunctions.go
similarity index 98%
rename from pkg/connector/slack/v0/taskFunctions.go
rename to application/slack/v0/taskFunctions.go
index 0b79752b0..9fe897dbc 100644
--- a/pkg/connector/slack/v0/taskFunctions.go
+++ b/application/slack/v0/taskFunctions.go
@@ -5,7 +5,7 @@ import (
"sync"
"time"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
"github.com/slack-go/slack"
"google.golang.org/protobuf/types/known/structpb"
)
@@ -17,7 +17,7 @@ func (e *execution) readMessage(in *structpb.Struct) (*structpb.Struct, error) {
if err := base.ConvertFromStructpb(in, ¶ms); err != nil {
return nil, err
}
-
+
targetChannelID, err := loopChannelListAPI(e, params.IsPublicChannel, params.ChannelName)
if err != nil {
@@ -80,7 +80,7 @@ func (e *execution) readMessage(in *structpb.Struct) (*structpb.Struct, error) {
if readTaskResp.Conversations == nil {
readTaskResp.Conversations = []Conversation{}
}
-
+
out, err := base.ConvertToStructpb(readTaskResp)
if err != nil {
return nil, err
diff --git a/pkg/connector/website/v0/README.mdx b/application/website/v0/README.mdx
similarity index 94%
rename from pkg/connector/website/v0/README.mdx
rename to application/website/v0/README.mdx
index ad4bb54ce..af4a3f152 100644
--- a/pkg/connector/website/v0/README.mdx
+++ b/application/website/v0/README.mdx
@@ -16,7 +16,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/website/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/application/website/v0/config/definition.json).
## Supported Tasks
diff --git a/pkg/connector/website/v0/assets/website.svg b/application/website/v0/assets/website.svg
similarity index 100%
rename from pkg/connector/website/v0/assets/website.svg
rename to application/website/v0/assets/website.svg
diff --git a/pkg/connector/website/v0/config/definition.json b/application/website/v0/config/definition.json
similarity index 94%
rename from pkg/connector/website/v0/config/definition.json
rename to application/website/v0/config/definition.json
index 117797b1f..318eaa91a 100644
--- a/pkg/connector/website/v0/config/definition.json
+++ b/application/website/v0/config/definition.json
@@ -15,6 +15,6 @@
"vendor": "",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/website/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/application/website/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/website/v0/config/tasks.json b/application/website/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/website/v0/config/tasks.json
rename to application/website/v0/config/tasks.json
diff --git a/pkg/connector/website/v0/main.go b/application/website/v0/main.go
similarity index 97%
rename from pkg/connector/website/v0/main.go
rename to application/website/v0/main.go
index 9e5191a7d..a141a6fc9 100644
--- a/pkg/connector/website/v0/main.go
+++ b/application/website/v0/main.go
@@ -9,7 +9,7 @@ import (
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
const (
diff --git a/pkg/connector/website/v0/scrape_website.go b/application/website/v0/scrape_website.go
similarity index 98%
rename from pkg/connector/website/v0/scrape_website.go
rename to application/website/v0/scrape_website.go
index 261f3dddb..55efa0a9e 100644
--- a/pkg/connector/website/v0/scrape_website.go
+++ b/application/website/v0/scrape_website.go
@@ -9,7 +9,7 @@ import (
"github.com/PuerkitoBio/goquery"
"github.com/gocolly/colly/v2"
- "github.com/instill-ai/component/pkg/connector/util"
+ "github.com/instill-ai/component/internal/util"
)
type PageInfo struct {
diff --git a/pkg/base/component.go b/base/component.go
similarity index 99%
rename from pkg/base/component.go
rename to base/component.go
index fd36d0262..645f90e17 100644
--- a/pkg/base/component.go
+++ b/base/component.go
@@ -6,7 +6,6 @@ import (
"strings"
"github.com/gofrs/uuid"
- "github.com/instill-ai/component/pkg/jsonref"
"github.com/lestrrat-go/jsref/provider"
"go.uber.org/zap"
"golang.org/x/text/cases"
@@ -15,6 +14,8 @@ import (
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/structpb"
+ "github.com/instill-ai/component/internal/jsonref"
+
pipelinePB "github.com/instill-ai/protogen-go/vdp/pipeline/v1beta"
)
diff --git a/pkg/base/connector.go b/base/connector.go
similarity index 100%
rename from pkg/base/connector.go
rename to base/connector.go
diff --git a/pkg/base/connector_test.go b/base/connector_test.go
similarity index 100%
rename from pkg/base/connector_test.go
rename to base/connector_test.go
diff --git a/pkg/base/execution.go b/base/execution.go
similarity index 100%
rename from pkg/base/execution.go
rename to base/execution.go
diff --git a/pkg/base/formats.go b/base/formats.go
similarity index 100%
rename from pkg/base/formats.go
rename to base/formats.go
diff --git a/pkg/base/openapi.go b/base/openapi.go
similarity index 100%
rename from pkg/base/openapi.go
rename to base/openapi.go
diff --git a/pkg/base/operator.go b/base/operator.go
similarity index 100%
rename from pkg/base/operator.go
rename to base/operator.go
diff --git a/pkg/base/operator_test.go b/base/operator_test.go
similarity index 100%
rename from pkg/base/operator_test.go
rename to base/operator_test.go
diff --git a/pkg/base/testdata/connectorAdditional.json b/base/testdata/connectorAdditional.json
similarity index 100%
rename from pkg/base/testdata/connectorAdditional.json
rename to base/testdata/connectorAdditional.json
diff --git a/pkg/base/testdata/connectorDef.json b/base/testdata/connectorDef.json
similarity index 98%
rename from pkg/base/testdata/connectorDef.json
rename to base/testdata/connectorDef.json
index f2c45c08e..785800b45 100644
--- a/pkg/base/testdata/connectorDef.json
+++ b/base/testdata/connectorDef.json
@@ -1,6 +1,6 @@
{
"version": "1.0.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/base",
+ "source_url": "https://github.com/instill-ai/component/blob/main/base",
"available_tasks": [
"TASK_TEXT_EMBEDDINGS"
],
diff --git a/pkg/base/testdata/connectorTasks.json b/base/testdata/connectorTasks.json
similarity index 100%
rename from pkg/base/testdata/connectorTasks.json
rename to base/testdata/connectorTasks.json
diff --git a/pkg/base/testdata/operatorDef.json b/base/testdata/operatorDef.json
similarity index 96%
rename from pkg/base/testdata/operatorDef.json
rename to base/testdata/operatorDef.json
index 85c9d346e..c0afa3cc6 100644
--- a/pkg/base/testdata/operatorDef.json
+++ b/base/testdata/operatorDef.json
@@ -1,6 +1,6 @@
{
"version": "1.0.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/base",
+ "source_url": "https://github.com/instill-ai/component/blob/main/base",
"available_tasks": [
"TASK_MARSHAL"
],
diff --git a/pkg/base/testdata/operatorTasks.json b/base/testdata/operatorTasks.json
similarity index 100%
rename from pkg/base/testdata/operatorTasks.json
rename to base/testdata/operatorTasks.json
diff --git a/pkg/base/testdata/wantConnectorDefinition.json b/base/testdata/wantConnectorDefinition.json
similarity index 99%
rename from pkg/base/testdata/wantConnectorDefinition.json
rename to base/testdata/wantConnectorDefinition.json
index 3325ce5d5..4511aa950 100644
--- a/pkg/base/testdata/wantConnectorDefinition.json
+++ b/base/testdata/wantConnectorDefinition.json
@@ -206,5 +206,5 @@
}
],
"version": "1.0.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/base"
+ "source_url": "https://github.com/instill-ai/component/blob/main/base"
}
diff --git a/pkg/base/testdata/wantOperatorDefinition.json b/base/testdata/wantOperatorDefinition.json
similarity index 99%
rename from pkg/base/testdata/wantOperatorDefinition.json
rename to base/testdata/wantOperatorDefinition.json
index de3735119..b58e9e5fb 100644
--- a/pkg/base/testdata/wantOperatorDefinition.json
+++ b/base/testdata/wantOperatorDefinition.json
@@ -128,7 +128,7 @@
},
"public": true,
"version": "1.0.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/base",
+ "source_url": "https://github.com/instill-ai/component/blob/main/base",
"tasks": [
{
"name": "TASK_MARSHAL",
diff --git a/pkg/base/usage.go b/base/usage.go
similarity index 100%
rename from pkg/base/usage.go
rename to base/usage.go
diff --git a/component.go b/component.go
new file mode 100644
index 000000000..c5c0c3307
--- /dev/null
+++ b/component.go
@@ -0,0 +1,216 @@
+package component
+
+import (
+ "fmt"
+ "sync"
+
+ "github.com/gofrs/uuid"
+ "go.uber.org/zap"
+ "google.golang.org/protobuf/types/known/structpb"
+
+ "github.com/instill-ai/component/ai/archetypeai/v0"
+ "github.com/instill-ai/component/ai/huggingface/v0"
+ "github.com/instill-ai/component/ai/instill/v0"
+ "github.com/instill-ai/component/ai/openai/v0"
+ "github.com/instill-ai/component/ai/stabilityai/v0"
+ "github.com/instill-ai/component/application/googlesearch/v0"
+ "github.com/instill-ai/component/application/numbers/v0"
+ "github.com/instill-ai/component/application/redis/v0"
+ "github.com/instill-ai/component/application/restapi/v0"
+ "github.com/instill-ai/component/application/slack/v0"
+ "github.com/instill-ai/component/application/website/v0"
+ "github.com/instill-ai/component/base"
+ "github.com/instill-ai/component/data/bigquery/v0"
+ "github.com/instill-ai/component/data/googlecloudstorage/v0"
+ "github.com/instill-ai/component/data/pinecone/v0"
+ "github.com/instill-ai/component/operator/base64/v0"
+ "github.com/instill-ai/component/operator/image/v0"
+ "github.com/instill-ai/component/operator/json/v0"
+ "github.com/instill-ai/component/operator/text/v0"
+
+ pipelinePB "github.com/instill-ai/protogen-go/vdp/pipeline/v1beta"
+)
+
+var (
+ once sync.Once
+ compStore *Store
+)
+
+// Store holds in-memory information about the initialized components.
+type Store struct {
+ operatorUIDs []uuid.UUID
+ operatorUIDMap map[uuid.UUID]*operator
+ operatorIDMap map[string]*operator
+
+ connectorUIDs []uuid.UUID
+ connectorUIDMap map[uuid.UUID]*connector
+ connectorIDMap map[string]*connector
+}
+
+type operator struct {
+ op base.IOperator
+}
+
+type connector struct {
+ con base.IConnector
+}
+
+// ConnectionSecrets contains the global connection secrets of each
+// implemented connector (referenced by ID). Connectors may use these secrets
+// to skip the connector configuration step and have a ready-to-run
+// connection.
+type ConnectionSecrets map[string]map[string]any
+
+// Init initializes the components implemented in this repository and loads
+// their information to memory.
+func Init(
+ logger *zap.Logger,
+ secrets ConnectionSecrets,
+ usageHandlerCreators map[string]base.UsageHandlerCreator,
+) *Store {
+ baseOp := base.Operator{Logger: logger}
+ baseConn := base.Connector{Logger: logger}
+
+ once.Do(func() {
+ compStore = &Store{
+ operatorUIDMap: map[uuid.UUID]*operator{},
+ operatorIDMap: map[string]*operator{},
+ connectorUIDMap: map[uuid.UUID]*connector{},
+ connectorIDMap: map[string]*connector{},
+ }
+ compStore.ImportOperator(base64.Init(baseOp))
+ compStore.ImportOperator(json.Init(baseOp))
+ compStore.ImportOperator(image.Init(baseOp))
+ compStore.ImportOperator(text.Init(baseOp))
+
+ {
+ // StabilityAI
+ conn := stabilityai.Init(baseConn)
+
+ // Secret doesn't allow hyphens
+ conn = conn.WithSecrets(secrets["stabilityai"]).
+ WithUsageHandlerCreator(usageHandlerCreators[conn.GetID()])
+ compStore.ImportConnector(conn)
+ }
+
+ compStore.ImportConnector(instill.Init(baseConn))
+ compStore.ImportConnector(huggingface.Init(baseConn))
+
+ {
+ // OpenAI
+ conn := openai.Init(baseConn)
+ conn = conn.WithSecrets(secrets[conn.GetID()]).
+ WithUsageHandlerCreator(usageHandlerCreators[conn.GetID()])
+ compStore.ImportConnector(conn)
+ }
+
+ compStore.ImportConnector(archetypeai.Init(baseConn))
+ compStore.ImportConnector(numbers.Init(baseConn))
+ compStore.ImportConnector(bigquery.Init(baseConn))
+ compStore.ImportConnector(googlecloudstorage.Init(baseConn))
+ compStore.ImportConnector(googlesearch.Init(baseConn))
+ compStore.ImportConnector(pinecone.Init(baseConn))
+ compStore.ImportConnector(redis.Init(baseConn))
+ compStore.ImportConnector(restapi.Init(baseConn))
+ compStore.ImportConnector(website.Init(baseConn))
+ compStore.ImportConnector(slack.Init(baseConn))
+
+ })
+ return compStore
+}
+
+// ImportOperator loads the operator definitions into memory.
+func (s *Store) ImportOperator(op base.IOperator) {
+ o := &operator{op: op}
+ s.operatorUIDMap[op.GetUID()] = o
+ s.operatorIDMap[op.GetID()] = o
+ s.operatorUIDs = append(s.operatorUIDs, op.GetUID())
+}
+
+// ImportConnector loads the connector definitions into memory.
+func (s *Store) ImportConnector(con base.IConnector) {
+ c := &connector{con: con}
+ s.connectorUIDMap[con.GetUID()] = c
+ s.connectorIDMap[con.GetID()] = c
+ s.connectorUIDs = append(s.connectorUIDs, con.GetUID())
+}
+
+// CreateExecution initializes the execution of a connector given its UID.
+func (s *Store) CreateExecution(defUID uuid.UUID, sysVars map[string]any, connection *structpb.Struct, task string) (*base.ExecutionWrapper, error) {
+ if op, ok := s.operatorUIDMap[defUID]; ok {
+ return op.op.CreateExecution(sysVars, task)
+ }
+ if con, ok := s.connectorUIDMap[defUID]; ok {
+ return con.con.CreateExecution(sysVars, connection, task)
+ }
+ return nil, fmt.Errorf("component definition not found")
+}
+
+// GetOperatorDefinitionByUID returns a operator definition by its UID.
+func (s *Store) GetOperatorDefinitionByUID(defUID uuid.UUID, sysVars map[string]any, component *pipelinePB.OperatorComponent) (*pipelinePB.OperatorDefinition, error) {
+ if op, ok := s.operatorUIDMap[defUID]; ok {
+ return op.op.GetOperatorDefinition(sysVars, component)
+ }
+ return nil, fmt.Errorf("operator definition not found")
+}
+
+// GetOperatorDefinitionByID returns a operator definition by its ID.
+func (s *Store) GetOperatorDefinitionByID(defID string, sysVars map[string]any, component *pipelinePB.OperatorComponent) (*pipelinePB.OperatorDefinition, error) {
+ if op, ok := s.operatorIDMap[defID]; ok {
+ return op.op.GetOperatorDefinition(sysVars, component)
+ }
+ return nil, fmt.Errorf("operator definition not found")
+}
+
+// ListOperatorDefinitions returns all the loaded operator definitions.
+func (s *Store) ListOperatorDefinitions(sysVars map[string]any, returnTombstone bool) []*pipelinePB.OperatorDefinition {
+ defs := []*pipelinePB.OperatorDefinition{}
+ for _, uid := range s.operatorUIDs {
+ op := s.operatorUIDMap[uid]
+ def, err := op.op.GetOperatorDefinition(sysVars, nil)
+ if err == nil {
+ if !def.Tombstone || returnTombstone {
+ defs = append(defs, def)
+ }
+ }
+ }
+ return defs
+}
+
+// GetConnectorDefinitionByUID returns a connector definition by its UID.
+func (s *Store) GetConnectorDefinitionByUID(defUID uuid.UUID, sysVars map[string]any, component *pipelinePB.ConnectorComponent) (*pipelinePB.ConnectorDefinition, error) {
+ if con, ok := s.connectorUIDMap[defUID]; ok {
+ return con.con.GetConnectorDefinition(sysVars, component)
+ }
+ return nil, fmt.Errorf("connector definition not found")
+}
+
+// GetConnectorDefinitionByID returns a connector definition by its ID.
+func (s *Store) GetConnectorDefinitionByID(defID string, sysVars map[string]any, component *pipelinePB.ConnectorComponent) (*pipelinePB.ConnectorDefinition, error) {
+ if con, ok := s.connectorIDMap[defID]; ok {
+ return con.con.GetConnectorDefinition(sysVars, component)
+ }
+ return nil, fmt.Errorf("connector definition not found")
+}
+
+// ListConnectorDefinitions returns all the loaded connector definitions.
+func (s *Store) ListConnectorDefinitions(sysVars map[string]any, returnTombstone bool) []*pipelinePB.ConnectorDefinition {
+ defs := []*pipelinePB.ConnectorDefinition{}
+ for _, uid := range s.connectorUIDs {
+ con := s.connectorUIDMap[uid]
+ def, err := con.con.GetConnectorDefinition(sysVars, nil)
+ if err == nil {
+ if !def.Tombstone || returnTombstone {
+ defs = append(defs, def)
+ }
+ }
+ }
+ return defs
+}
+
+func (s *Store) IsSecretField(defUID uuid.UUID, target string) (bool, error) {
+ if con, ok := s.connectorUIDMap[defUID]; ok {
+ return con.con.IsSecretField(target), nil
+ }
+ return false, fmt.Errorf("connector definition not found")
+}
diff --git a/pkg/connector/bigquery/v0/README.mdx b/data/bigquery/v0/README.mdx
similarity index 93%
rename from pkg/connector/bigquery/v0/README.mdx
rename to data/bigquery/v0/README.mdx
index 155769fb1..72714a03a 100644
--- a/pkg/connector/bigquery/v0/README.mdx
+++ b/data/bigquery/v0/README.mdx
@@ -16,7 +16,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/bigquery/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/bigquery/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/bigquery/v0/assets/bigquery.svg b/data/bigquery/v0/assets/bigquery.svg
similarity index 100%
rename from pkg/connector/bigquery/v0/assets/bigquery.svg
rename to data/bigquery/v0/assets/bigquery.svg
diff --git a/pkg/connector/bigquery/v0/config/definition.json b/data/bigquery/v0/config/definition.json
similarity index 98%
rename from pkg/connector/bigquery/v0/config/definition.json
rename to data/bigquery/v0/config/definition.json
index dd6b4f212..4392259a3 100644
--- a/pkg/connector/bigquery/v0/config/definition.json
+++ b/data/bigquery/v0/config/definition.json
@@ -88,6 +88,6 @@
"vendor": "Google",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/bigquery/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/data/bigquery/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/bigquery/v0/config/tasks.json b/data/bigquery/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/bigquery/v0/config/tasks.json
rename to data/bigquery/v0/config/tasks.json
diff --git a/pkg/connector/bigquery/v0/insert.go b/data/bigquery/v0/insert.go
similarity index 100%
rename from pkg/connector/bigquery/v0/insert.go
rename to data/bigquery/v0/insert.go
diff --git a/pkg/connector/bigquery/v0/main.go b/data/bigquery/v0/main.go
similarity index 98%
rename from pkg/connector/bigquery/v0/main.go
rename to data/bigquery/v0/main.go
index 5d4598cee..d0de96515 100644
--- a/pkg/connector/bigquery/v0/main.go
+++ b/data/bigquery/v0/main.go
@@ -12,7 +12,7 @@ import (
"google.golang.org/api/option"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
const (
diff --git a/pkg/connector/googlecloudstorage/v0/README.mdx b/data/googlecloudstorage/v0/README.mdx
similarity index 94%
rename from pkg/connector/googlecloudstorage/v0/README.mdx
rename to data/googlecloudstorage/v0/README.mdx
index af00b26a2..3e61bcbbf 100644
--- a/pkg/connector/googlecloudstorage/v0/README.mdx
+++ b/data/googlecloudstorage/v0/README.mdx
@@ -16,7 +16,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/googlecloudstorage/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/googlecloudstorage/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/googlecloudstorage/v0/assets/gcs.svg b/data/googlecloudstorage/v0/assets/gcs.svg
similarity index 100%
rename from pkg/connector/googlecloudstorage/v0/assets/gcs.svg
rename to data/googlecloudstorage/v0/assets/gcs.svg
diff --git a/pkg/connector/googlecloudstorage/v0/config/definition.json b/data/googlecloudstorage/v0/config/definition.json
similarity index 97%
rename from pkg/connector/googlecloudstorage/v0/config/definition.json
rename to data/googlecloudstorage/v0/config/definition.json
index 96d6317cb..0a5d06237 100644
--- a/pkg/connector/googlecloudstorage/v0/config/definition.json
+++ b/data/googlecloudstorage/v0/config/definition.json
@@ -61,6 +61,6 @@
"vendor": "Google",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/googlecloudstorage/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/data/googlecloudstorage/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/googlecloudstorage/v0/config/tasks.json b/data/googlecloudstorage/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/googlecloudstorage/v0/config/tasks.json
rename to data/googlecloudstorage/v0/config/tasks.json
diff --git a/pkg/connector/googlecloudstorage/v0/main.go b/data/googlecloudstorage/v0/main.go
similarity index 98%
rename from pkg/connector/googlecloudstorage/v0/main.go
rename to data/googlecloudstorage/v0/main.go
index efed59303..b296a3271 100644
--- a/pkg/connector/googlecloudstorage/v0/main.go
+++ b/data/googlecloudstorage/v0/main.go
@@ -11,7 +11,7 @@ import (
"google.golang.org/api/option"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
const (
diff --git a/pkg/connector/googlecloudstorage/v0/upload.go b/data/googlecloudstorage/v0/upload.go
similarity index 97%
rename from pkg/connector/googlecloudstorage/v0/upload.go
rename to data/googlecloudstorage/v0/upload.go
index 553b24258..680631896 100644
--- a/pkg/connector/googlecloudstorage/v0/upload.go
+++ b/data/googlecloudstorage/v0/upload.go
@@ -8,7 +8,7 @@ import (
"cloud.google.com/go/iam"
"cloud.google.com/go/storage"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
func uploadToGCS(client *storage.Client, bucketName, objectName, data string) error {
diff --git a/pkg/connector/pinecone/v0/README.mdx b/data/pinecone/v0/README.mdx
similarity index 96%
rename from pkg/connector/pinecone/v0/README.mdx
rename to data/pinecone/v0/README.mdx
index 10b173b4f..b7b3a49cc 100644
--- a/pkg/connector/pinecone/v0/README.mdx
+++ b/data/pinecone/v0/README.mdx
@@ -17,7 +17,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/pinecone/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/pinecone/v0/config/definition.json).
## Connection
diff --git a/pkg/connector/pinecone/v0/assets/pinecone.svg b/data/pinecone/v0/assets/pinecone.svg
similarity index 100%
rename from pkg/connector/pinecone/v0/assets/pinecone.svg
rename to data/pinecone/v0/assets/pinecone.svg
diff --git a/pkg/connector/pinecone/v0/config/definition.json b/data/pinecone/v0/config/definition.json
similarity index 98%
rename from pkg/connector/pinecone/v0/config/definition.json
rename to data/pinecone/v0/config/definition.json
index 7146a6322..6baee0ddd 100644
--- a/pkg/connector/pinecone/v0/config/definition.json
+++ b/data/pinecone/v0/config/definition.json
@@ -61,6 +61,6 @@
"vendor": "Pinecone",
"vendor_attributes": {},
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/pinecone/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/data/pinecone/v0",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/connector/pinecone/v0/config/tasks.json b/data/pinecone/v0/config/tasks.json
similarity index 100%
rename from pkg/connector/pinecone/v0/config/tasks.json
rename to data/pinecone/v0/config/tasks.json
diff --git a/pkg/connector/pinecone/v0/connector_test.go b/data/pinecone/v0/connector_test.go
similarity index 98%
rename from pkg/connector/pinecone/v0/connector_test.go
rename to data/pinecone/v0/connector_test.go
index 25ae7a2f2..93030bc9a 100644
--- a/pkg/connector/pinecone/v0/connector_test.go
+++ b/data/pinecone/v0/connector_test.go
@@ -13,8 +13,8 @@ import (
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/base"
+ "github.com/instill-ai/component/internal/util/httpclient"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/connector/pinecone/v0/main.go b/data/pinecone/v0/main.go
similarity index 96%
rename from pkg/connector/pinecone/v0/main.go
rename to data/pinecone/v0/main.go
index b4e44f81d..f20ff5ff7 100644
--- a/pkg/connector/pinecone/v0/main.go
+++ b/data/pinecone/v0/main.go
@@ -9,8 +9,8 @@ import (
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/util/httpclient"
+ "github.com/instill-ai/component/base"
+ "github.com/instill-ai/component/internal/util/httpclient"
)
const (
diff --git a/pkg/connector/pinecone/v0/structs.go b/data/pinecone/v0/structs.go
similarity index 100%
rename from pkg/connector/pinecone/v0/structs.go
rename to data/pinecone/v0/structs.go
diff --git a/pkg/jsonref/interface.go b/internal/jsonref/interface.go
similarity index 100%
rename from pkg/jsonref/interface.go
rename to internal/jsonref/interface.go
diff --git a/pkg/jsonref/options.go b/internal/jsonref/options.go
similarity index 100%
rename from pkg/jsonref/options.go
rename to internal/jsonref/options.go
diff --git a/pkg/jsonref/ref.go b/internal/jsonref/ref.go
similarity index 100%
rename from pkg/jsonref/ref.go
rename to internal/jsonref/ref.go
diff --git a/internal/mock/generator.go b/internal/mock/generator.go
new file mode 100644
index 000000000..0cf10adcb
--- /dev/null
+++ b/internal/mock/generator.go
@@ -0,0 +1,3 @@
+package mock
+
+//go:generate minimock -g -i github.com/instill-ai/component/base.UsageHandler -o ./ -s "_mock.gen.go"
diff --git a/pkg/mock/usage_handler_mock.gen.go b/internal/mock/usage_handler_mock.gen.go
similarity index 100%
rename from pkg/mock/usage_handler_mock.gen.go
rename to internal/mock/usage_handler_mock.gen.go
diff --git a/pkg/connector/util/helper.go b/internal/util/helper.go
similarity index 97%
rename from pkg/connector/util/helper.go
rename to internal/util/helper.go
index a7c79eec1..0e410fbb1 100644
--- a/pkg/connector/util/helper.go
+++ b/internal/util/helper.go
@@ -9,7 +9,7 @@ import (
md "github.com/JohannesKaufmann/html-to-markdown"
"github.com/PuerkitoBio/goquery"
"github.com/h2non/filetype"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
func GetFileExt(fileData []byte) string {
diff --git a/pkg/connector/util/helper_test.go b/internal/util/helper_test.go
similarity index 100%
rename from pkg/connector/util/helper_test.go
rename to internal/util/helper_test.go
diff --git a/pkg/connector/util/httpclient/httpclient.go b/internal/util/httpclient/httpclient.go
similarity index 100%
rename from pkg/connector/util/httpclient/httpclient.go
rename to internal/util/httpclient/httpclient.go
diff --git a/pkg/connector/util/httpclient/httpclient_test.go b/internal/util/httpclient/httpclient_test.go
similarity index 100%
rename from pkg/connector/util/httpclient/httpclient_test.go
rename to internal/util/httpclient/httpclient_test.go
diff --git a/pkg/operator/base64/v0/README.mdx b/operator/base64/v0/README.mdx
similarity index 92%
rename from pkg/operator/base64/v0/README.mdx
rename to operator/base64/v0/README.mdx
index f02364ade..dcafb58b8 100644
--- a/pkg/operator/base64/v0/README.mdx
+++ b/operator/base64/v0/README.mdx
@@ -17,7 +17,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/operator/base64/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/operator/base64/v0/config/definition.json).
## Supported Tasks
diff --git a/pkg/operator/base64/v0/assets/base64.svg b/operator/base64/v0/assets/base64.svg
similarity index 100%
rename from pkg/operator/base64/v0/assets/base64.svg
rename to operator/base64/v0/assets/base64.svg
diff --git a/pkg/operator/base64/v0/config/definition.json b/operator/base64/v0/config/definition.json
similarity index 95%
rename from pkg/operator/base64/v0/config/definition.json
rename to operator/base64/v0/config/definition.json
index 477cb447c..5214f6b6f 100644
--- a/pkg/operator/base64/v0/config/definition.json
+++ b/operator/base64/v0/config/definition.json
@@ -14,7 +14,7 @@
"tombstone": false,
"uid": "3a836447-c211-4134-9cc5-ad45e1cc467e",
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/base64/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/operator/base64/v0",
"description": "Encode or decode a string in Base64 format",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/operator/base64/v0/config/tasks.json b/operator/base64/v0/config/tasks.json
similarity index 100%
rename from pkg/operator/base64/v0/config/tasks.json
rename to operator/base64/v0/config/tasks.json
diff --git a/pkg/operator/base64/v0/encoding_test.go b/operator/base64/v0/encoding_test.go
similarity index 100%
rename from pkg/operator/base64/v0/encoding_test.go
rename to operator/base64/v0/encoding_test.go
diff --git a/pkg/operator/base64/v0/main.go b/operator/base64/v0/main.go
similarity index 98%
rename from pkg/operator/base64/v0/main.go
rename to operator/base64/v0/main.go
index ef46994a8..6ae7c41a6 100644
--- a/pkg/operator/base64/v0/main.go
+++ b/operator/base64/v0/main.go
@@ -12,7 +12,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
const (
diff --git a/pkg/operator/image/v0/README.mdx b/operator/image/v0/README.mdx
similarity index 97%
rename from pkg/operator/image/v0/README.mdx
rename to operator/image/v0/README.mdx
index 6a9194061..1f7834015 100644
--- a/pkg/operator/image/v0/README.mdx
+++ b/operator/image/v0/README.mdx
@@ -21,7 +21,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/operator/image/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/operator/image/v0/config/definition.json).
## Supported Tasks
diff --git a/pkg/operator/image/v0/assets/image.svg b/operator/image/v0/assets/image.svg
similarity index 100%
rename from pkg/operator/image/v0/assets/image.svg
rename to operator/image/v0/assets/image.svg
diff --git a/pkg/operator/image/v0/config/definition.json b/operator/image/v0/config/definition.json
similarity index 96%
rename from pkg/operator/image/v0/config/definition.json
rename to operator/image/v0/config/definition.json
index 17036415d..97460d708 100644
--- a/pkg/operator/image/v0/config/definition.json
+++ b/operator/image/v0/config/definition.json
@@ -17,7 +17,7 @@
"title": "Image",
"uid": "e9eb8fc8-f249-4e11-ad50-5035d79ffc18",
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/image/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/operator/image/v0",
"description": "Manipulate image files",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/operator/image/v0/config/tasks.json b/operator/image/v0/config/tasks.json
similarity index 100%
rename from pkg/operator/image/v0/config/tasks.json
rename to operator/image/v0/config/tasks.json
diff --git a/pkg/operator/image/v0/draw.go b/operator/image/v0/draw.go
similarity index 100%
rename from pkg/operator/image/v0/draw.go
rename to operator/image/v0/draw.go
diff --git a/pkg/operator/image/v0/draw_test.go b/operator/image/v0/draw_test.go
similarity index 100%
rename from pkg/operator/image/v0/draw_test.go
rename to operator/image/v0/draw_test.go
diff --git a/pkg/operator/image/v0/font.go b/operator/image/v0/font.go
similarity index 100%
rename from pkg/operator/image/v0/font.go
rename to operator/image/v0/font.go
diff --git a/pkg/operator/image/v0/main.go b/operator/image/v0/main.go
similarity index 98%
rename from pkg/operator/image/v0/main.go
rename to operator/image/v0/main.go
index e3d23d5d3..44a6db72b 100644
--- a/pkg/operator/image/v0/main.go
+++ b/operator/image/v0/main.go
@@ -16,7 +16,7 @@ import (
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
var (
diff --git a/pkg/operator/image/v0/testdata/cls-dog.json b/operator/image/v0/testdata/cls-dog.json
similarity index 100%
rename from pkg/operator/image/v0/testdata/cls-dog.json
rename to operator/image/v0/testdata/cls-dog.json
diff --git a/pkg/operator/image/v0/testdata/det-coco-1.json b/operator/image/v0/testdata/det-coco-1.json
similarity index 100%
rename from pkg/operator/image/v0/testdata/det-coco-1.json
rename to operator/image/v0/testdata/det-coco-1.json
diff --git a/pkg/operator/image/v0/testdata/det-coco-2.json b/operator/image/v0/testdata/det-coco-2.json
similarity index 100%
rename from pkg/operator/image/v0/testdata/det-coco-2.json
rename to operator/image/v0/testdata/det-coco-2.json
diff --git a/pkg/operator/image/v0/testdata/inst-seg-coco-1.json b/operator/image/v0/testdata/inst-seg-coco-1.json
similarity index 100%
rename from pkg/operator/image/v0/testdata/inst-seg-coco-1.json
rename to operator/image/v0/testdata/inst-seg-coco-1.json
diff --git a/pkg/operator/image/v0/testdata/inst-seg-coco-2.json b/operator/image/v0/testdata/inst-seg-coco-2.json
similarity index 100%
rename from pkg/operator/image/v0/testdata/inst-seg-coco-2.json
rename to operator/image/v0/testdata/inst-seg-coco-2.json
diff --git a/pkg/operator/image/v0/testdata/inst-seg-stomata.json b/operator/image/v0/testdata/inst-seg-stomata.json
similarity index 100%
rename from pkg/operator/image/v0/testdata/inst-seg-stomata.json
rename to operator/image/v0/testdata/inst-seg-stomata.json
diff --git a/pkg/operator/image/v0/testdata/kp-coco-1.json b/operator/image/v0/testdata/kp-coco-1.json
similarity index 100%
rename from pkg/operator/image/v0/testdata/kp-coco-1.json
rename to operator/image/v0/testdata/kp-coco-1.json
diff --git a/pkg/operator/image/v0/testdata/kp-coco-2.json b/operator/image/v0/testdata/kp-coco-2.json
similarity index 100%
rename from pkg/operator/image/v0/testdata/kp-coco-2.json
rename to operator/image/v0/testdata/kp-coco-2.json
diff --git a/pkg/operator/image/v0/testdata/ocr-mm.json b/operator/image/v0/testdata/ocr-mm.json
similarity index 100%
rename from pkg/operator/image/v0/testdata/ocr-mm.json
rename to operator/image/v0/testdata/ocr-mm.json
diff --git a/pkg/operator/image/v0/testdata/sem-seg-cityscape.json b/operator/image/v0/testdata/sem-seg-cityscape.json
similarity index 100%
rename from pkg/operator/image/v0/testdata/sem-seg-cityscape.json
rename to operator/image/v0/testdata/sem-seg-cityscape.json
diff --git a/pkg/operator/json/v0/README.mdx b/operator/json/v0/README.mdx
similarity index 95%
rename from pkg/operator/json/v0/README.mdx
rename to operator/json/v0/README.mdx
index cc61bea6a..924b53db8 100644
--- a/pkg/operator/json/v0/README.mdx
+++ b/operator/json/v0/README.mdx
@@ -18,7 +18,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/operator/json/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/operator/json/v0/config/definition.json).
## Supported Tasks
diff --git a/pkg/operator/json/v0/assets/json.svg b/operator/json/v0/assets/json.svg
similarity index 100%
rename from pkg/operator/json/v0/assets/json.svg
rename to operator/json/v0/assets/json.svg
diff --git a/pkg/operator/json/v0/config/definition.json b/operator/json/v0/config/definition.json
similarity index 95%
rename from pkg/operator/json/v0/config/definition.json
rename to operator/json/v0/config/definition.json
index 2c272fb1b..b20855d7d 100644
--- a/pkg/operator/json/v0/config/definition.json
+++ b/operator/json/v0/config/definition.json
@@ -14,7 +14,7 @@
"title": "JSON",
"uid": "28f53d15-6150-46e6-99aa-f76b70a926c0",
"version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/json/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/operator/json/v0",
"description": "Manipulate and convert JSON objects",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/operator/json/v0/config/tasks.json b/operator/json/v0/config/tasks.json
similarity index 100%
rename from pkg/operator/json/v0/config/tasks.json
rename to operator/json/v0/config/tasks.json
diff --git a/pkg/operator/json/v0/main.go b/operator/json/v0/main.go
similarity index 98%
rename from pkg/operator/json/v0/main.go
rename to operator/json/v0/main.go
index dc122b628..b3e0220cb 100644
--- a/pkg/operator/json/v0/main.go
+++ b/operator/json/v0/main.go
@@ -12,7 +12,7 @@ import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/operator/json/v0/operator_test.go b/operator/json/v0/operator_test.go
similarity index 98%
rename from pkg/operator/json/v0/operator_test.go
rename to operator/json/v0/operator_test.go
index b46333b14..f1ff216b1 100644
--- a/pkg/operator/json/v0/operator_test.go
+++ b/operator/json/v0/operator_test.go
@@ -10,7 +10,7 @@ import (
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
"github.com/instill-ai/x/errmsg"
)
diff --git a/pkg/operator/text/v0/README.mdx b/operator/text/v0/README.mdx
similarity index 95%
rename from pkg/operator/text/v0/README.mdx
rename to operator/text/v0/README.mdx
index d74f5ce18..9119faa3c 100644
--- a/pkg/operator/text/v0/README.mdx
+++ b/operator/text/v0/README.mdx
@@ -17,7 +17,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/operator/text/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/operator/text/v0/config/definition.json).
## Supported Tasks
diff --git a/pkg/operator/text/v0/assets/text.svg b/operator/text/v0/assets/text.svg
similarity index 100%
rename from pkg/operator/text/v0/assets/text.svg
rename to operator/text/v0/assets/text.svg
diff --git a/pkg/operator/text/v0/config/definition.json b/operator/text/v0/config/definition.json
similarity index 95%
rename from pkg/operator/text/v0/config/definition.json
rename to operator/text/v0/config/definition.json
index b4937d6a3..a7fafc0ff 100644
--- a/pkg/operator/text/v0/config/definition.json
+++ b/operator/text/v0/config/definition.json
@@ -13,7 +13,7 @@
"title": "Text",
"uid": "5b7aca5b-1ae3-477f-bf60-d34e1c993c87",
"version": "0.1.1",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/text/v0",
+ "source_url": "https://github.com/instill-ai/component/blob/main/operator/text/v0",
"description": "Extract and manipulate text from different sources",
"release_stage": "RELEASE_STAGE_ALPHA"
}
diff --git a/pkg/operator/text/v0/config/tasks.json b/operator/text/v0/config/tasks.json
similarity index 100%
rename from pkg/operator/text/v0/config/tasks.json
rename to operator/text/v0/config/tasks.json
diff --git a/pkg/operator/text/v0/convert.go b/operator/text/v0/convert.go
similarity index 97%
rename from pkg/operator/text/v0/convert.go
rename to operator/text/v0/convert.go
index d5a33cd13..a6f44eea9 100644
--- a/pkg/operator/text/v0/convert.go
+++ b/operator/text/v0/convert.go
@@ -9,7 +9,7 @@ import (
"code.sajari.com/docconv"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
// ConvertToTextInput defines the input for convert to text task
diff --git a/pkg/operator/text/v0/convert_test.go b/operator/text/v0/convert_test.go
similarity index 100%
rename from pkg/operator/text/v0/convert_test.go
rename to operator/text/v0/convert_test.go
diff --git a/pkg/operator/text/v0/main.go b/operator/text/v0/main.go
similarity index 98%
rename from pkg/operator/text/v0/main.go
rename to operator/text/v0/main.go
index a845418ab..9ff8b38d5 100644
--- a/pkg/operator/text/v0/main.go
+++ b/operator/text/v0/main.go
@@ -10,7 +10,7 @@ import (
"google.golang.org/protobuf/types/known/structpb"
- "github.com/instill-ai/component/pkg/base"
+ "github.com/instill-ai/component/base"
)
const (
diff --git a/pkg/operator/text/v0/split.go b/operator/text/v0/split.go
similarity index 100%
rename from pkg/operator/text/v0/split.go
rename to operator/text/v0/split.go
diff --git a/pkg/operator/text/v0/split_test.go b/operator/text/v0/split_test.go
similarity index 100%
rename from pkg/operator/text/v0/split_test.go
rename to operator/text/v0/split_test.go
diff --git a/pkg/operator/text/v0/testdata/test.docx b/operator/text/v0/testdata/test.docx
similarity index 100%
rename from pkg/operator/text/v0/testdata/test.docx
rename to operator/text/v0/testdata/test.docx
diff --git a/pkg/operator/text/v0/testdata/test.html b/operator/text/v0/testdata/test.html
similarity index 100%
rename from pkg/operator/text/v0/testdata/test.html
rename to operator/text/v0/testdata/test.html
diff --git a/pkg/operator/text/v0/testdata/test.jpg b/operator/text/v0/testdata/test.jpg
similarity index 100%
rename from pkg/operator/text/v0/testdata/test.jpg
rename to operator/text/v0/testdata/test.jpg
diff --git a/pkg/operator/text/v0/testdata/test.odt b/operator/text/v0/testdata/test.odt
similarity index 100%
rename from pkg/operator/text/v0/testdata/test.odt
rename to operator/text/v0/testdata/test.odt
diff --git a/pkg/operator/text/v0/testdata/test.pdf b/operator/text/v0/testdata/test.pdf
similarity index 100%
rename from pkg/operator/text/v0/testdata/test.pdf
rename to operator/text/v0/testdata/test.pdf
diff --git a/pkg/operator/text/v0/testdata/test.png b/operator/text/v0/testdata/test.png
similarity index 100%
rename from pkg/operator/text/v0/testdata/test.png
rename to operator/text/v0/testdata/test.png
diff --git a/pkg/operator/text/v0/testdata/test.rtf b/operator/text/v0/testdata/test.rtf
similarity index 100%
rename from pkg/operator/text/v0/testdata/test.rtf
rename to operator/text/v0/testdata/test.rtf
diff --git a/pkg/operator/text/v0/testdata/test.tif b/operator/text/v0/testdata/test.tif
similarity index 100%
rename from pkg/operator/text/v0/testdata/test.tif
rename to operator/text/v0/testdata/test.tif
diff --git a/pkg/operator/text/v0/testdata/test.txt b/operator/text/v0/testdata/test.txt
similarity index 100%
rename from pkg/operator/text/v0/testdata/test.txt
rename to operator/text/v0/testdata/test.txt
diff --git a/pkg/connector/airbyte/v0/assets/airbyte-destination.svg b/pkg/connector/airbyte/v0/assets/airbyte-destination.svg
deleted file mode 100644
index c19d229cb..000000000
--- a/pkg/connector/airbyte/v0/assets/airbyte-destination.svg
+++ /dev/null
@@ -1,3 +0,0 @@
-
diff --git a/pkg/connector/airbyte/v0/config/definition.json b/pkg/connector/airbyte/v0/config/definition.json
deleted file mode 100644
index b19470cb2..000000000
--- a/pkg/connector/airbyte/v0/config/definition.json
+++ /dev/null
@@ -1,24 +0,0 @@
-{
- "available_tasks": [],
- "custom": false,
- "documentation_url": "https://docs.airbyte.com/integrations/destinations",
- "icon": "Airbyte/airbyte.svg",
- "icon_url": "",
- "id": "airbyte-destination",
- "public": true,
- "version": "0.1.0",
- "release_stage": "RELEASE_STAGE_ALPHA",
- "spec": {
- "connection_specification": {
- "$schema": "http://json-schema.org/draft-07/schema#",
- "title": "Destination",
- "type": "object"
- }
- },
- "title": "Airbyte Destination",
- "tombstone": true,
- "type": "CONNECTOR_TYPE_DATA",
- "uid": "975678a2-5117-48a4-a135-019619dee18e",
- "vendor": "Airbyte",
- "vendor_attributes": {}
-}
diff --git a/pkg/connector/airbyte/v0/config/tasks.json b/pkg/connector/airbyte/v0/config/tasks.json
deleted file mode 100644
index 0967ef424..000000000
--- a/pkg/connector/airbyte/v0/config/tasks.json
+++ /dev/null
@@ -1 +0,0 @@
-{}
diff --git a/pkg/connector/airbyte/v0/main.go b/pkg/connector/airbyte/v0/main.go
deleted file mode 100644
index f57d3baef..000000000
--- a/pkg/connector/airbyte/v0/main.go
+++ /dev/null
@@ -1,54 +0,0 @@
-package airbyte
-
-import (
- "context"
- _ "embed"
- "fmt"
- "sync"
-
- "google.golang.org/protobuf/types/known/structpb"
-
- "github.com/instill-ai/component/pkg/base"
-)
-
-//go:embed config/definition.json
-var definitionJSON []byte
-
-//go:embed config/tasks.json
-var tasksJSON []byte
-
-var once sync.Once
-var con *connector
-
-type connector struct {
- base.Connector
-}
-
-type execution struct {
- base.ConnectorExecution
-}
-
-func Init(bc base.Connector) *connector {
- once.Do(func() {
- con = &connector{Connector: bc}
- err := con.LoadConnectorDefinition(definitionJSON, tasksJSON, nil)
- if err != nil {
- panic(err)
- }
- })
- return con
-}
-
-func (c *connector) CreateExecution(sysVars map[string]any, connection *structpb.Struct, task string) (*base.ExecutionWrapper, error) {
- return &base.ExecutionWrapper{Execution: &execution{
- ConnectorExecution: base.ConnectorExecution{Connector: c, SystemVariables: sysVars, Connection: connection, Task: task},
- }}, nil
-}
-
-func (e *execution) Execute(context.Context, []*structpb.Struct) ([]*structpb.Struct, error) {
- return nil, fmt.Errorf("the Airbyte connector has been removed")
-}
-
-func (c *connector) Test(map[string]any, *structpb.Struct) error {
- return fmt.Errorf("the Airbyte connector has been removed")
-}
diff --git a/pkg/connector/main.go b/pkg/connector/main.go
deleted file mode 100644
index 66de6f92d..000000000
--- a/pkg/connector/main.go
+++ /dev/null
@@ -1,159 +0,0 @@
-package connector
-
-import (
- "fmt"
- "sync"
-
- "github.com/gofrs/uuid"
- "go.uber.org/zap"
- "google.golang.org/protobuf/types/known/structpb"
-
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/connector/airbyte/v0"
- "github.com/instill-ai/component/pkg/connector/archetypeai/v0"
- "github.com/instill-ai/component/pkg/connector/bigquery/v0"
- "github.com/instill-ai/component/pkg/connector/googlecloudstorage/v0"
- "github.com/instill-ai/component/pkg/connector/googlesearch/v0"
- "github.com/instill-ai/component/pkg/connector/huggingface/v0"
- "github.com/instill-ai/component/pkg/connector/instill/v0"
- "github.com/instill-ai/component/pkg/connector/numbers/v0"
- "github.com/instill-ai/component/pkg/connector/openai/v0"
- "github.com/instill-ai/component/pkg/connector/pinecone/v0"
- "github.com/instill-ai/component/pkg/connector/redis/v0"
- "github.com/instill-ai/component/pkg/connector/restapi/v0"
- "github.com/instill-ai/component/pkg/connector/slack/v0"
- "github.com/instill-ai/component/pkg/connector/stabilityai/v0"
- "github.com/instill-ai/component/pkg/connector/website/v0"
-
- pipelinePB "github.com/instill-ai/protogen-go/vdp/pipeline/v1beta"
-)
-
-var (
- once sync.Once
- conStore *Store
-)
-
-// Store holds in-memory information about the initialized connectors.
-type Store struct {
- connectorUIDs []uuid.UUID
- connectorUIDMap map[uuid.UUID]*connector
- connectorIDMap map[string]*connector
-}
-
-type connector struct {
- con base.IConnector
-}
-
-// ConnectionSecrets contains the global connection secrets of each
-// implemented connector (referenced by ID). Connectors may use these secrets
-// to skip the connector configuration step and have a ready-to-run
-// connection.
-type ConnectionSecrets map[string]map[string]any
-
-// Init initializes the different connector components and loads their
-// information to memory.
-func Init(
- logger *zap.Logger,
- secrets ConnectionSecrets,
- usageHandlerCreators map[string]base.UsageHandlerCreator,
-) *Store {
-
- baseConn := base.Connector{Logger: logger}
-
- once.Do(func() {
- conStore = &Store{
- connectorUIDMap: map[uuid.UUID]*connector{},
- connectorIDMap: map[string]*connector{},
- }
-
- {
- // StabilityAI
- conn := stabilityai.Init(baseConn)
-
- // Secret doesn't allow hyphens
- conn = conn.WithSecrets(secrets["stabilityai"]).
- WithUsageHandlerCreator(usageHandlerCreators[conn.GetID()])
- conStore.Import(conn)
- }
-
- conStore.Import(instill.Init(baseConn))
- conStore.Import(huggingface.Init(baseConn))
-
- {
- // OpenAI
- conn := openai.Init(baseConn)
- conn = conn.WithSecrets(secrets[conn.GetID()]).
- WithUsageHandlerCreator(usageHandlerCreators[conn.GetID()])
- conStore.Import(conn)
- }
-
- conStore.Import(archetypeai.Init(baseConn))
- conStore.Import(numbers.Init(baseConn))
- conStore.Import(airbyte.Init(baseConn))
- conStore.Import(bigquery.Init(baseConn))
- conStore.Import(googlecloudstorage.Init(baseConn))
- conStore.Import(googlesearch.Init(baseConn))
- conStore.Import(pinecone.Init(baseConn))
- conStore.Import(redis.Init(baseConn))
- conStore.Import(restapi.Init(baseConn))
- conStore.Import(website.Init(baseConn))
- conStore.Import(slack.Init(baseConn))
-
- })
-
- return conStore
-}
-
-// Import loads the connector definitions into memory.
-func (cs *Store) Import(con base.IConnector) {
- c := &connector{con: con}
- cs.connectorUIDMap[con.GetUID()] = c
- cs.connectorIDMap[con.GetID()] = c
- cs.connectorUIDs = append(cs.connectorUIDs, con.GetUID())
-}
-
-// CreateExecution initializes the execution of a connector given its UID.
-func (cs *Store) CreateExecution(defUID uuid.UUID, sysVars map[string]any, connection *structpb.Struct, task string) (*base.ExecutionWrapper, error) {
- if con, ok := cs.connectorUIDMap[defUID]; ok {
- return con.con.CreateExecution(sysVars, connection, task)
- }
- return nil, fmt.Errorf("connector definition not found")
-}
-
-// GetConnectorDefinitionByUID returns a connector definition by its UID.
-func (cs *Store) GetConnectorDefinitionByUID(defUID uuid.UUID, sysVars map[string]any, component *pipelinePB.ConnectorComponent) (*pipelinePB.ConnectorDefinition, error) {
- if con, ok := cs.connectorUIDMap[defUID]; ok {
- return con.con.GetConnectorDefinition(sysVars, component)
- }
- return nil, fmt.Errorf("connector definition not found")
-}
-
-// GetConnectorDefinitionByID returns a connector definition by its ID.
-func (cs *Store) GetConnectorDefinitionByID(defID string, sysVars map[string]any, component *pipelinePB.ConnectorComponent) (*pipelinePB.ConnectorDefinition, error) {
- if con, ok := cs.connectorIDMap[defID]; ok {
- return con.con.GetConnectorDefinition(sysVars, component)
- }
- return nil, fmt.Errorf("connector definition not found")
-}
-
-// ListConnectorDefinitions returns all the loaded connector definitions.
-func (cs *Store) ListConnectorDefinitions(sysVars map[string]any, returnTombstone bool) []*pipelinePB.ConnectorDefinition {
- defs := []*pipelinePB.ConnectorDefinition{}
- for _, uid := range cs.connectorUIDs {
- con := cs.connectorUIDMap[uid]
- def, err := con.con.GetConnectorDefinition(sysVars, nil)
- if err == nil {
- if !def.Tombstone || returnTombstone {
- defs = append(defs, def)
- }
- }
- }
- return defs
-}
-
-func (cs *Store) IsSecretField(defUID uuid.UUID, target string) (bool, error) {
- if con, ok := cs.connectorUIDMap[defUID]; ok {
- return con.con.IsSecretField(target), nil
- }
- return false, fmt.Errorf("connector definition not found")
-}
diff --git a/pkg/connector/slack/v0/config/tasks.json b/pkg/connector/slack/v0/config/tasks.json
deleted file mode 100644
index bbfbc9323..000000000
--- a/pkg/connector/slack/v0/config/tasks.json
+++ /dev/null
@@ -1,239 +0,0 @@
-{
- "TASK_READ_MESSAGE": {
- "instillShortDescription": "Get the latest message since specific date",
- "input": {
- "description": "Please input the channel name and the date that we want to start to read",
- "instillUIOrder": 0,
- "properties": {
- "channel_name": {
- "description": "A channel name display in Slack",
- "instillAcceptFormats": [
- "string"
- ],
- "instillUIMultiline": true,
- "instillUIOrder": 0,
- "instillUpstreamTypes": [
- "value",
- "reference",
- "template"
- ],
- "title": "Channel Name",
- "type": "string"
- },
- "start_to_read_date": {
- "description": "earliest date in all read messages",
- "instillAcceptFormats": [
- "string"
- ],
- "instillUIMultiline": true,
- "instillUIOrder": 1,
- "instillUpstreamTypes": [
- "value",
- "reference",
- "template"
- ],
- "title": "Start to read date",
- "type": "string"
- },
- "is_public_channel": {
- "description": "Whether all members can read the channel",
- "instillAcceptFormats": [
- "boolean"
- ],
- "instillUIMultiline": true,
- "instillUIOrder": 2,
- "instillUpstreamTypes": [
- "value",
- "reference",
- "template"
- ],
- "title": "Public channel",
- "type": "boolean"
- }
- },
- "required": [
- "channel_name"
- ],
- "title": "Input",
- "type": "object"
- },
- "output": {
- "description": "All messages in Slack channel",
- "instillUIOrder": 0,
- "properties": {
- "conversations": {
- "description": "An array of conversations with thread messages",
- "instillUIOrder": 0,
- "title": "Conversations",
- "type": "array",
- "items": {
- "title": "conversation details",
- "type": "object",
- "properties": {
- "user_id": {
- "description": "unique id from Slack",
- "instillFormat": "string",
- "instillUIOrder": 1,
- "title": "User UID",
- "type": "string"
- },
- "message": {
- "description": "message to start a conversation",
- "instillFormat": "string",
- "instillUIOrder": 2,
- "title": "Start Conversation Message",
- "type": "string"
- },
- "start_date": {
- "description": "when a conversation starts",
- "instillFormat": "string",
- "instillUIOrder": 3,
- "required": [],
- "title": "Start Date",
- "type": "string"
- },
- "last_date": {
- "description": "Date of the last message",
- "instillFormat": "string",
- "instillUIOrder": 4,
- "required": [],
- "title": "Last Date",
- "type": "string"
- },
- "thread_reply_messages": {
- "description": "replies in a conversation",
- "instillFormat": "array",
- "instillUIOrder": 5,
- "title": "Replied messages",
- "type": "array",
- "items": {
- "title": "relied details",
- "type": "object",
- "properties": {
- "user_id": {
- "description": "unique id from Slack",
- "instillFormat": "string",
- "instillUIOrder": 1,
- "title": "User UID",
- "type": "string"
- },
- "datetime": {
- "description": "replied datetime",
- "instillFormat": "string",
- "instillUIOrder": 2,
- "title": "Replied Time",
- "type": "string"
- },
- "message": {
- "description": "message to reply a conversation",
- "instillFormat": "string",
- "instillUIOrder": 3,
- "title": "Replied Message",
- "type": "string"
- }
- },
- "required": [
- "user_id",
- "datetime",
- "message"
- ]
- }
- }
- },
- "required": [
- "user_id",
- "message",
- "start_date"
- ]
- }
- }
- },
- "required": [
- "conversations"
- ],
- "title": "Output",
- "type": "object"
- }
- },
- "TASK_WRITE_MESSAGE": {
- "instillShortDescription": "send message to a specific channel",
- "title": "Send Message",
- "input": {
- "description": "Input",
- "instillUIOrder": 0,
- "properties": {
- "channel_name": {
- "description": "A channel name display in Slack",
- "instillAcceptFormats": [
- "string"
- ],
- "instillUIMultiline": true,
- "instillUIOrder": 0,
- "instillUpstreamTypes": [
- "value",
- "reference",
- "template"
- ],
- "title": "Channel Name",
- "type": "string"
- },
- "message": {
- "description": "message to be sent to the target channel",
- "instillAcceptFormats": [
- "string"
- ],
- "instillUIMultiline": true,
- "instillUIOrder": 1,
- "instillUpstreamTypes": [
- "value",
- "reference",
- "template"
- ],
- "title": "Message",
- "type": "string"
- },
- "is_public_channel": {
- "description": "Whether all members can read the channel",
- "instillAcceptFormats": [
- "boolean"
- ],
- "instillUIMultiline": true,
- "instillUIOrder": 2,
- "instillUpstreamTypes": [
- "value",
- "reference",
- "template"
- ],
- "title": "Public channel",
- "type": "boolean"
- }
- },
- "required": [
- "channel_name",
- "message"
- ],
- "title": "Input",
- "type": "object"
- },
- "output": {
- "description": "The greeting sentence",
- "instillUIOrder": 0,
- "properties": {
- "result": {
- "description": "result for sending message",
- "instillEditOnNodeFields": [],
- "instillUIOrder": 0,
- "required": [],
- "title": "Result",
- "type": "string",
- "instillFormat": "string"
- }
- },
- "required": [
- "result"
- ],
- "title": "Output",
- "type": "object"
- }
- }
- }
\ No newline at end of file
diff --git a/pkg/mock/generator.go b/pkg/mock/generator.go
deleted file mode 100644
index f64b8dfe9..000000000
--- a/pkg/mock/generator.go
+++ /dev/null
@@ -1,3 +0,0 @@
-package mock
-
-//go:generate minimock -g -i github.com/instill-ai/component/pkg/base.UsageHandler -o ./ -s "_mock.gen.go"
diff --git a/pkg/operator/end/v0/assets/end.svg b/pkg/operator/end/v0/assets/end.svg
deleted file mode 100644
index b62a05707..000000000
--- a/pkg/operator/end/v0/assets/end.svg
+++ /dev/null
@@ -1,3 +0,0 @@
-
diff --git a/pkg/operator/end/v0/config/definition.json b/pkg/operator/end/v0/config/definition.json
deleted file mode 100644
index bdb368e74..000000000
--- a/pkg/operator/end/v0/config/definition.json
+++ /dev/null
@@ -1,19 +0,0 @@
-{
- "available_tasks": [
- "TASK_END"
- ],
- "custom": false,
- "documentation_url": "https://www.instill.tech/docs/latest/vdp/operators/op-end",
- "icon": "assets/end.svg",
- "icon_url": "",
- "id": "end",
- "public": true,
- "spec": {},
- "title": "End",
- "tombstone": true,
- "uid": "4f39c8bc-8617-495d-80de-80d0f5397516",
- "version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/end/v0",
- "description": "Create an output interface in a pipeline when triggered synchronously",
- "release_stage": "RELEASE_STAGE_ALPHA"
-}
diff --git a/pkg/operator/end/v0/config/tasks.json b/pkg/operator/end/v0/config/tasks.json
deleted file mode 100644
index 45149f505..000000000
--- a/pkg/operator/end/v0/config/tasks.json
+++ /dev/null
@@ -1,55 +0,0 @@
-{
- "TASK_END": {
- "input": {
- "additionalProperties": false,
- "description": "Input",
- "instillEditOnNodeFields": [],
- "instillUIOrder": 0,
- "patternProperties": {
- "^[a-z_][-a-z_0-9]{0,31}$": {
- "description": "Arbitrary data",
- "instillAcceptFormats": [
- "*"
- ],
- "instillUIOrder": 0,
- "instillUpstreamTypes": [
- "value",
- "reference",
- "template"
- ],
- "title": "Key"
- }
- },
- "required": [],
- "title": "Input",
- "type": "object"
- },
- "metadata": {
- "additionalProperties": false,
- "patternProperties": {
- "^[a-z_][-a-z_0-9]{0,31}$": {
- "properties": {
- "description": {
- "type": "string"
- },
- "title": {
- "type": "string"
- }
- },
- "title": "Metadata Item",
- "type": "object"
- }
- },
- "title": "Metadata",
- "type": "object"
- },
- "output": {
- "description": "Output",
- "instillEditOnNodeFields": [],
- "instillUIOrder": 0,
- "required": [],
- "title": "Output",
- "type": "object"
- }
- }
-}
diff --git a/pkg/operator/end/v0/main.go b/pkg/operator/end/v0/main.go
deleted file mode 100644
index 939ce87ab..000000000
--- a/pkg/operator/end/v0/main.go
+++ /dev/null
@@ -1,54 +0,0 @@
-package end
-
-import (
- "context"
- _ "embed"
- "fmt"
- "sync"
-
- "google.golang.org/protobuf/types/known/structpb"
-
- "github.com/instill-ai/component/pkg/base"
-)
-
-//go:embed config/definition.json
-var definitionJSON []byte
-
-//go:embed config/tasks.json
-var tasksJSON []byte
-
-var once sync.Once
-var op *operator
-
-type operator struct {
- base.Operator
-}
-
-type execution struct {
- base.OperatorExecution
-}
-
-func Init(bo base.Operator) *operator {
- once.Do(func() {
- op = &operator{Operator: bo}
- err := op.LoadOperatorDefinition(definitionJSON, tasksJSON, nil)
- if err != nil {
- panic(err)
- }
- })
- return op
-}
-
-func (o *operator) CreateExecution(sysVars map[string]any, task string) (*base.ExecutionWrapper, error) {
- return &base.ExecutionWrapper{Execution: &execution{
- OperatorExecution: base.OperatorExecution{Operator: o, SystemVariables: sysVars, Task: task},
- }}, nil
-}
-
-func (e *execution) Execute(_ context.Context, inputs []*structpb.Struct) ([]*structpb.Struct, error) {
- return nil, fmt.Errorf("the Airbyte operator has been removed")
-}
-
-func (o *operator) Test(sysVars map[string]any, connection *structpb.Struct) error {
- return fmt.Errorf("the Airbyte operator has been removed")
-}
diff --git a/pkg/operator/main.go b/pkg/operator/main.go
deleted file mode 100644
index 20777c262..000000000
--- a/pkg/operator/main.go
+++ /dev/null
@@ -1,103 +0,0 @@
-package operator
-
-import (
- "fmt"
- "sync"
-
- "github.com/gofrs/uuid"
- "go.uber.org/zap"
-
- "github.com/instill-ai/component/pkg/base"
- "github.com/instill-ai/component/pkg/operator/base64/v0"
- "github.com/instill-ai/component/pkg/operator/end/v0"
- "github.com/instill-ai/component/pkg/operator/image/v0"
- "github.com/instill-ai/component/pkg/operator/json/v0"
- "github.com/instill-ai/component/pkg/operator/start/v0"
- "github.com/instill-ai/component/pkg/operator/text/v0"
-
- pipelinePB "github.com/instill-ai/protogen-go/vdp/pipeline/v1beta"
-)
-
-var (
- once sync.Once
- opStore *Store
-)
-
-// Store holds in-memory information about the initialized operators.
-type Store struct {
- operatorUIDs []uuid.UUID
- operatorUIDMap map[uuid.UUID]*operator
- operatorIDMap map[string]*operator
-}
-
-type operator struct {
- op base.IOperator
-}
-
-// Init initializes the different operator components and loads their
-// information to memory.
-func Init(logger *zap.Logger) *Store {
- baseOp := base.Operator{Logger: logger}
-
- once.Do(func() {
- opStore = &Store{
- operatorUIDMap: map[uuid.UUID]*operator{},
- operatorIDMap: map[string]*operator{},
- }
- opStore.Import(start.Init(baseOp)) // deprecated
- opStore.Import(end.Init(baseOp)) // deprecated
- opStore.Import(base64.Init(baseOp))
- opStore.Import(json.Init(baseOp))
- opStore.Import(image.Init(baseOp))
- opStore.Import(text.Init(baseOp))
-
- })
- return opStore
-}
-
-// Import loads the operator definitions into memory.
-func (os *Store) Import(op base.IOperator) {
- o := &operator{op: op}
- os.operatorUIDMap[op.GetUID()] = o
- os.operatorIDMap[op.GetID()] = o
- os.operatorUIDs = append(os.operatorUIDs, op.GetUID())
-}
-
-// CreateExecution initializes the execution of a operator given its UID.
-func (os *Store) CreateExecution(defUID uuid.UUID, sysVars map[string]any, task string) (*base.ExecutionWrapper, error) {
- if op, ok := os.operatorUIDMap[defUID]; ok {
- return op.op.CreateExecution(sysVars, task)
- }
- return nil, fmt.Errorf("operator definition not found")
-}
-
-// GetOperatorDefinitionByUID returns a operator definition by its UID.
-func (os *Store) GetOperatorDefinitionByUID(defUID uuid.UUID, sysVars map[string]any, component *pipelinePB.OperatorComponent) (*pipelinePB.OperatorDefinition, error) {
- if op, ok := os.operatorUIDMap[defUID]; ok {
- return op.op.GetOperatorDefinition(sysVars, component)
- }
- return nil, fmt.Errorf("operator definition not found")
-}
-
-// GetOperatorDefinitionByID returns a operator definition by its ID.
-func (os *Store) GetOperatorDefinitionByID(defID string, sysVars map[string]any, component *pipelinePB.OperatorComponent) (*pipelinePB.OperatorDefinition, error) {
- if op, ok := os.operatorIDMap[defID]; ok {
- return op.op.GetOperatorDefinition(sysVars, component)
- }
- return nil, fmt.Errorf("operator definition not found")
-}
-
-// ListOperatorDefinitions returns all the loaded operator definitions.
-func (os *Store) ListOperatorDefinitions(sysVars map[string]any, returnTombstone bool) []*pipelinePB.OperatorDefinition {
- defs := []*pipelinePB.OperatorDefinition{}
- for _, uid := range os.operatorUIDs {
- op := os.operatorUIDMap[uid]
- def, err := op.op.GetOperatorDefinition(sysVars, nil)
- if err == nil {
- if !def.Tombstone || returnTombstone {
- defs = append(defs, def)
- }
- }
- }
- return defs
-}
diff --git a/pkg/operator/start/v0/assets/start.svg b/pkg/operator/start/v0/assets/start.svg
deleted file mode 100644
index 418a3093c..000000000
--- a/pkg/operator/start/v0/assets/start.svg
+++ /dev/null
@@ -1,3 +0,0 @@
-
diff --git a/pkg/operator/start/v0/config/definition.json b/pkg/operator/start/v0/config/definition.json
deleted file mode 100644
index 3e64b86d6..000000000
--- a/pkg/operator/start/v0/config/definition.json
+++ /dev/null
@@ -1,19 +0,0 @@
-{
- "available_tasks": [
- "TASK_START"
- ],
- "custom": false,
- "documentation_url": "https://www.instill.tech/docs/latest/vdp/operators/op-start",
- "icon": "assets/start.svg",
- "icon_url": "",
- "id": "start",
- "public": true,
- "spec": {},
- "title": "Start",
- "tombstone": true,
- "uid": "2ac8be70-0f7a-4b61-a33d-098b8acfa6f3",
- "version": "0.1.0",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/start/v0",
- "description": "Define data formats customized for their AI-first applications",
- "release_stage": "RELEASE_STAGE_ALPHA"
-}
diff --git a/pkg/operator/start/v0/config/tasks.json b/pkg/operator/start/v0/config/tasks.json
deleted file mode 100644
index 25b0aee71..000000000
--- a/pkg/operator/start/v0/config/tasks.json
+++ /dev/null
@@ -1,64 +0,0 @@
-{
- "TASK_START": {
- "input": {
- "description": "Input",
- "instillEditOnNodeFields": [],
- "instillUIOrder": 0,
- "required": [],
- "title": "Input",
- "type": "object"
- },
- "metadata": {
- "additionalProperties": false,
- "patternProperties": {
- "^[a-z_][-a-z_0-9]{0,31}$": {
- "properties": {
- "description": {
- "type": "string"
- },
- "instillFormat": {
- "type": "string"
- },
- "items": {
- "properties": {
- "instillFormat": {
- "type": "string"
- },
- "type": {
- "type": "string"
- }
- },
- "required": [
- "type",
- "instillFormat"
- ],
- "type": "object"
- },
- "title": {
- "type": "string"
- },
- "type": {
- "type": "string"
- }
- },
- "required": [
- "type",
- "instillFormat"
- ],
- "title": "Metadata Item",
- "type": "object"
- }
- },
- "title": "Metadata",
- "type": "object"
- },
- "output": {
- "description": "Output",
- "instillEditOnNodeFields": [],
- "instillUIOrder": 0,
- "required": [],
- "title": "Output",
- "type": "object"
- }
- }
-}
diff --git a/pkg/operator/start/v0/main.go b/pkg/operator/start/v0/main.go
deleted file mode 100644
index 669968d2c..000000000
--- a/pkg/operator/start/v0/main.go
+++ /dev/null
@@ -1,54 +0,0 @@
-package start
-
-import (
- "context"
- _ "embed"
- "fmt"
- "sync"
-
- "google.golang.org/protobuf/types/known/structpb"
-
- "github.com/instill-ai/component/pkg/base"
-)
-
-//go:embed config/definition.json
-var definitionJSON []byte
-
-//go:embed config/tasks.json
-var tasksJSON []byte
-
-var once sync.Once
-var op *operator
-
-type operator struct {
- base.Operator
-}
-
-type execution struct {
- base.OperatorExecution
-}
-
-func Init(bo base.Operator) *operator {
- once.Do(func() {
- op = &operator{Operator: bo}
- err := op.LoadOperatorDefinition(definitionJSON, tasksJSON, nil)
- if err != nil {
- panic(err)
- }
- })
- return op
-}
-
-func (o *operator) CreateExecution(sysVars map[string]any, task string) (*base.ExecutionWrapper, error) {
- return &base.ExecutionWrapper{Execution: &execution{
- OperatorExecution: base.OperatorExecution{Operator: o, SystemVariables: sysVars, Task: task},
- }}, nil
-}
-
-func (e *execution) Execute(_ context.Context, inputs []*structpb.Struct) ([]*structpb.Struct, error) {
- return nil, fmt.Errorf("the Airbyte operator has been removed")
-}
-
-func (o *operator) Test(sysVars map[string]any, connection *structpb.Struct) error {
- return fmt.Errorf("the Airbyte operator has been removed")
-}
diff --git a/tools/compogen/README.md b/tools/compogen/README.md
index 598bb51b7..3c74bd034 100644
--- a/tools/compogen/README.md
+++ b/tools/compogen/README.md
@@ -55,7 +55,7 @@ Certain optional fields modify the document behaviour:
## TODO
-- Support `oneOf` schemas for resource properties, present in, e.g., the [REST API](https://github.com/instill-ai/component/blob/main/pkg/connector/restapi/v0/config/definition.json#L26) connectors.
+- Support `oneOf` schemas for resource properties, present in, e.g., the [REST API](https://github.com/instill-ai/component/blob/main/application/restapi/v0/config/definition.json#L26) connectors.
- We might leverage some Go implementation of JSON schema. Some candidates:
- [santhosh-tekuri/jsonschema](https://pkg.go.dev/github.com/santhosh-tekuri/jsonschema/v5#Schema)
- [omissis/go-jsonschema](https://github.com/omissis/go-jsonschema/blob/934012d/pkg/schemas/model.go#L107)
diff --git a/tools/compogen/cmd/testdata/readme-connector.txt b/tools/compogen/cmd/testdata/readme-connector.txt
index 4851f3b73..c0f4bee78 100644
--- a/tools/compogen/cmd/testdata/readme-connector.txt
+++ b/tools/compogen/cmd/testdata/readme-connector.txt
@@ -46,7 +46,7 @@ cmp pkg/dummy/README.mdx want-readme.mdx
}
},
"release_stage": "RELEASE_STAGE_COMING_SOON",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/connector/dummy/v0"
+ "source_url": "https://github.com/instill-ai/component/blob/main/data/dummy/v0"
}
-- tasks.json --
{
@@ -96,7 +96,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/dummy/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/dummy/v0/config/definition.json).
## Connection
diff --git a/tools/compogen/cmd/testdata/readme-operator.txt b/tools/compogen/cmd/testdata/readme-operator.txt
index 3950643c9..05e1258c6 100644
--- a/tools/compogen/cmd/testdata/readme-operator.txt
+++ b/tools/compogen/cmd/testdata/readme-operator.txt
@@ -34,7 +34,7 @@ cmp pkg/dummy/README.mdx want-readme.mdx
"title": "Dummy",
"description": "Perform an action",
"release_stage": "RELEASE_STAGE_BETA",
- "source_url": "https://github.com/instill-ai/component/blob/main/pkg/operator/dummy/v0"
+ "source_url": "https://github.com/instill-ai/component/blob/main/operator/dummy/v0"
}
-- tasks.json --
{
@@ -169,7 +169,7 @@ It can carry out the following tasks:
## Configuration
-The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/operator/dummy/v0/config/definition.json).
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/operator/dummy/v0/config/definition.json).
## Supported Tasks
diff --git a/tools/compogen/pkg/gen/readme.go b/tools/compogen/pkg/gen/readme.go
index d20afc935..b81287e65 100644
--- a/tools/compogen/pkg/gen/readme.go
+++ b/tools/compogen/pkg/gen/readme.go
@@ -16,7 +16,7 @@ import (
"github.com/go-playground/validator/v10"
"github.com/russross/blackfriday/v2"
- component "github.com/instill-ai/component/pkg/base"
+ componentbase "github.com/instill-ai/component/base"
)
const (
@@ -53,7 +53,7 @@ func (g *READMEGenerator) parseDefinition(configDir string) (d definition, err e
return d, err
}
- renderedDefinitionJSON, err := component.RenderJSON(definitionJSON, nil)
+ renderedDefinitionJSON, err := componentbase.RenderJSON(definitionJSON, nil)
if err != nil {
return d, err
}
@@ -94,7 +94,7 @@ func (g *READMEGenerator) parseTasks(configDir string) (map[string]task, error)
}
- renderedTasksJSON, err := component.RenderJSON(tasksJSON, additionalJSONs)
+ renderedTasksJSON, err := componentbase.RenderJSON(tasksJSON, additionalJSONs)
if err != nil {
return nil, err
}
@@ -237,7 +237,7 @@ func parseREADMETasks(availableTasks []string, tasks map[string]task) ([]readmeT
}
if rt.Title = t.Title; rt.Title == "" {
- rt.Title = component.TaskIDToTitle(at)
+ rt.Title = componentbase.TaskIDToTitle(at)
}
readmeTasks[i] = rt