Merge branch 'instill-ai:main' into main

instill-ai · Nov 6, 2024 · 43cfb20 · 43cfb20
2 parents 2e181fe + 7e5d3de
commit 43cfb20
Show file tree

Hide file tree

Showing 356 changed files with 30,863 additions and 8,510 deletions.
diff --git a/.env b/.env
@@ -12,6 +12,12 @@ PUBLIC_SERVICE_PORT=8081
 DOCKER_BUILDKIT=1
 COMPOSE_DOCKER_CLI_BUILD=1
 
+# ONNX_MODEL_FOLDER_PATH specifies the directory where ONNX models are stored.
+# These models are loaded dynamically at runtime. The path is set relative to
+# the project root, allowing for consistent model loading across different
+# deployment environments.
+ONNX_MODEL_FOLDER_PATH=${PWD}/pkg/component/resources/onnx
+
 # test
 
 # TEST_DBHOST and TEST_DBNAME are used to initialize a separate database for

diff --git a/.env.component b/.env.component
@@ -0,0 +1,25 @@
+# Component global secrets
+
+# Provide your API key for the AI vendors so that you can set the components up
+# with default credentials.
+CFG_COMPONENT_SECRETS_OPENAI_APIKEY=
+CFG_COMPONENT_SECRETS_STABILITYAI_APIKEY=
+CFG_COMPONENT_SECRETS_ANTHROPIC_APIKEY=
+CFG_COMPONENT_SECRETS_COHERE_APIKEY=
+CFG_COMPONENT_SECRETS_MISTRALAI_APIKEY=
+CFG_COMPONENT_SECRETS_GROQ_APIKEY=
+CFG_COMPONENT_SECRETS_FIREWORKSAI_APIKEY=
+
+# Numbers Protocol API key.
+CFG_COMPONENT_SECRETS_NUMBERS_XAPIKEY=
+
+# OAuth secrets. When these are filled, the specified component will support
+# OAuth integrations.
+CFG_COMPONENT_SECRETS_GOOGLEDRIVE_OAUTHCLIENTID=
+CFG_COMPONENT_SECRETS_GOOGLEDRIVE_OAUTHCLIENTSECRET=
+CFG_COMPONENT_SECRETS_SLACK_OAUTHCLIENTID=
+CFG_COMPONENT_SECRETS_SLACK_OAUTHCLIENTSECRET=
+# Dummy values are added here to check OAuth connection creation on integration
+# tests.
+CFG_COMPONENT_SECRETS_GITHUB_OAUTHCLIENTID=dummy-id
+CFG_COMPONENT_SECRETS_GITHUB_OAUTHCLIENTSECRET=dummy-secret
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -44,12 +44,26 @@ $ make latest PROFILE=exclude-pipeline # launch all the dependent services excep
 ```sh
 $ cd $MY_WORKSPACE
 $ git clone https://github.com/instill-ai/pipeline-backend && cd pipeline-backend
-$ make build && make dev
+$ make build-dev && make dev
 ```
 
 Now, you have the Go project set up in the container, in which you can compile
 and run the binaries together with the integration test in each container shell.
 
+##### Injecting component secrets
+
+Some components can be configured with global secrets. This has several
+applications:
+
+- By accepting a global API key, some components have a default setup. When
+  the `setup` block is omitted in the recipe, this API key will be used.
+- In order to connect to 3rd party vendors via OAuth, the application
+  client ID and secret must be injected.
+
+You can set the values of these global secrets in
+[`.env.component`](./.env.component) before running the Docker container in
+order to add a global configuration to your components.
+
 #### Run the server and the Temporal worker
 
 ```sh
@@ -71,11 +85,9 @@ will create and migrate a test database to keep these queries isolated from the
 main DB. You can set the database host and name by overriding the `TEST_DBHOST`
 and `TEST_DBNAME` values.
 
-Certain tests depend on the [`docconv`](https://github.com/sajari/docconv)
-package and aren't run by default. You can trigger them by adding the `OCR=true`
-flag to the coverage command. Make sure to install the [package
-dependencies](https://github.com/sajari/docconv?tab=readme-ov-file#dependencies)
-first.
+Certain tests depend on external packages and aren't run by default:
+- For [`docconv`](https://github.com/sajari/docconv) tests, add `OCR=true` flag and install its [dependencies](https://github.com/sajari/docconv?tab=readme-ov-file#dependencies).
+- For [`onnxruntime`](https://github.com/microsoft/onnxruntime) tests, add `ONNX=true` flag. Follow the [guideline](#set-up-onnx-runtime) to set up ONNX Runtime (Linux only).
 
 #### Run the integration tests
 
@@ -91,18 +103,28 @@ At the end of the tests, some SQL queries are run to clean up the data.
 `DB_HOST` points to the database host so the SQL connection can be established.
 If empty, tests will try to connect to `localhost:5432`.
 
-#### Stop the dev container
-
-```bash
-$ make stop
-```
-
 #### Remove the dev container
 
 ```bash
 $ make rm
 ```
 
+### Set up ONNX Runtime (Linux only)
+
+1. Download the latest [ONNX Runtime release](https://github.com/microsoft/onnxruntime/releases) for your system.
+
+2. Install ONNX Runtime:
+   ```bash
+   sudo mkdir -p /usr/local/onnxruntime
+   sudo tar -xzf onnxruntime-*-*-*.tgz -C /usr/local/onnxruntime --strip-components=1
+   export ONNXRUNTIME_ROOT_PATH=/usr/local/onnxruntime
+   export LD_RUN_PATH=$ONNXRUNTIME_ROOT_PATH/lib
+   export LIBRARY_PATH=$ONNXRUNTIME_ROOT_PATH/lib
+   export C_INCLUDE_PATH=$ONNXRUNTIME_ROOT_PATH/include
+   ```
+
+**Note:** If you don't have sudo access, extract to a user-writeable location (e.g., `~/onnxruntime`), set `ONNXRUNTIME_ROOT_PATH` accordingly, and adjust the environment variables as shown above. No need to create symlinks in this case.
+
 ## Codebase contribution
 
 ### Pre-commit hooks

diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml
@@ -22,8 +22,6 @@ jobs:
         ports:
           - 5432:5432
     steps:
-      - name: Install operator dependencies
-        run: sudo apt-get update -y && sudo apt-get install poppler-utils wv unrtf tidy tesseract-ocr libtesseract-dev -y
 
       - uses: actions/checkout@v3
         with:
@@ -40,10 +38,23 @@ jobs:
 
       - uses: actions/checkout@v3
 
+      - name: Install onnxruntime library and headers
+        run: |
+          export ONNXRUNTIME_ROOT_PATH=$GITHUB_WORKSPACE/onnxruntime
+          LATEST_VERSION=$(wget -qO- https://api.github.com/repos/microsoft/onnxruntime/releases/latest | jq -r .tag_name)
+          ONNX_ARCH=$([ "$TARGETARCH" = "arm64" ] && echo "aarch64" || echo "x64")
+          wget https://github.com/microsoft/onnxruntime/releases/download/${LATEST_VERSION}/onnxruntime-linux-${ONNX_ARCH}-${LATEST_VERSION#v}.tgz
+          tar -xzf onnxruntime-linux-${ONNX_ARCH}-${LATEST_VERSION#v}.tgz
+          mv onnxruntime-linux-${ONNX_ARCH}-${LATEST_VERSION#v} ${ONNXRUNTIME_ROOT_PATH}
+          rm onnxruntime-linux-${ONNX_ARCH}-${LATEST_VERSION#v}.tgz
+          echo "LD_RUN_PATH=${ONNXRUNTIME_ROOT_PATH}/lib" >> $GITHUB_ENV
+          echo "LIBRARY_PATH=${ONNXRUNTIME_ROOT_PATH}/lib" >> $GITHUB_ENV
+          echo "C_INCLUDE_PATH=${ONNXRUNTIME_ROOT_PATH}/include" >> $GITHUB_ENV
+
       - name: Generate coverage report
         run: |
-          go mod tidy
-          make coverage DBTEST=true OCR=true
+          make build-dev
+          make coverage DBTEST=true OCR=true ONNX=true
 
       - name: Upload coverage report
         uses: codecov/codecov-action@v2

diff --git a/.github/workflows/golangci-lint.yml b/.github/workflows/golangci-lint.yml
@@ -21,8 +21,24 @@ jobs:
         with:
           go-version: ${{ env.GOLANG_VERSION }}
           cache: false
+      - name: Install sorx
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libsoxr-dev
+      - name: Install onnxruntime library and headers
+        run: |
+          export ONNXRUNTIME_ROOT_PATH=$GITHUB_WORKSPACE/onnxruntime
+          LATEST_VERSION=$(wget -qO- https://api.github.com/repos/microsoft/onnxruntime/releases/latest | jq -r .tag_name)
+          ONNX_ARCH=$([ "$TARGETARCH" = "arm64" ] && echo "aarch64" || echo "x64")
+          wget https://github.com/microsoft/onnxruntime/releases/download/${LATEST_VERSION}/onnxruntime-linux-${ONNX_ARCH}-${LATEST_VERSION#v}.tgz
+          tar -xzf onnxruntime-linux-${ONNX_ARCH}-${LATEST_VERSION#v}.tgz
+          mv onnxruntime-linux-${ONNX_ARCH}-${LATEST_VERSION#v} ${ONNXRUNTIME_ROOT_PATH}
+          rm onnxruntime-linux-${ONNX_ARCH}-${LATEST_VERSION#v}.tgz
+          echo "C_INCLUDE_PATH=${ONNXRUNTIME_ROOT_PATH}/include" >> $GITHUB_ENV
+          echo "LD_RUN_PATH=${ONNXRUNTIME_ROOT_PATH}/lib" >> $GITHUB_ENV
+          echo "LIBRARY_PATH=${ONNXRUNTIME_ROOT_PATH}/lib" >> $GITHUB_ENV
       - name: golangci-lint
         uses: golangci/golangci-lint-action@v6
         with:
-          version: v1.59
-          args: --timeout=10m
+          version: v1.61
+          args: --timeout=10m --build-tags onnx
diff --git a/.github/workflows/integration-test.yml b/.github/workflows/integration-test.yml
@@ -78,7 +78,10 @@ jobs:
           envFile: .env
 
       - name: Launch Instill Core (latest)
+        # CFG_COMPONENT_SECRETS_GITHUB* variables are injected to test OAuth connection
+        # creation
         run: |
+          sed -i 's/\(\w\+GITHUB\w\+\)=/\1=foo/' .env.component
           COMPOSE_PROFILES=all \
           EDITION=local-ce:test \
           RAY_LATEST_TAG=latest \
@@ -95,7 +98,7 @@ jobs:
       - name: Install k6
         run: |
           go install go.k6.io/xk6/cmd/xk6@v${{ env.XK6_VERSION }}
-          xk6 build v${{ env.K6_VERSION }} --with github.com/grafana/xk6-sql && sudo cp k6 /usr/bin
+          xk6 build v${{ env.K6_VERSION }} --with github.com/grafana/xk6-sql@v0.4.1 && sudo cp k6 /usr/bin
 
       - name: Checkout (pipeline)
         uses: actions/checkout@v3

diff --git a/.golangci.yaml b/.golangci.yaml
@@ -5,3 +5,5 @@ linters:
 linters-settings:
   stylecheck:
     checks: [ "all" ]
+  staticcheck:
+    checks: [ "all", "-SA1019" ]
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -11,7 +11,18 @@ repos:
   - repo: https://github.com/dnephin/pre-commit-golang
     rev: v0.5.1
     hooks:
+      # To make the golangci-lint hook work with ONNX:
+      # 1. Install ONNX runtime locally:
+      #    - Mac: `brew install onnxruntime`
+      #    - Linux: Follow https://github.com/streamer45/silero-vad-go
+      # 2. For Mac OS with Homebrew:
+      #    a. Get install path: `brew info onnxruntime`
+      #    b. Set environment variables:
+      #       export LIBRARY_PATH=/opt/homebrew/Cellar/onnxruntime/1.17.1/lib
+      #       export C_INCLUDE_PATH=/opt/homebrew/Cellar/onnxruntime/1.17.1/include/onnxruntime
+      #    c. Run: sudo update_dyld_shared_cache
       - id: golangci-lint
+        args: ["--build-tags", "onnx"]
       - id: go-mod-tidy
   - repo: https://github.com/pinglin/conventional-pre-commit
     rev: v1.1.0

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,90 @@
 # Changelog
 
+## [0.46.0-beta](https://github.com/instill-ai/pipeline-backend/compare/v0.45.2-beta...v0.46.0-beta) (2024-11-05)
+
+
+### Features
+
+* add `condition` field support for iterator ([#803](https://github.com/instill-ai/pipeline-backend/issues/803)) ([04b1252](https://github.com/instill-ai/pipeline-backend/commit/04b12529eb5a1fe5c4bc13fe89828fbdc403bc54))
+* add markdowns per pages ([#792](https://github.com/instill-ai/pipeline-backend/issues/792)) ([3ee428e](https://github.com/instill-ai/pipeline-backend/commit/3ee428ef7be4a4c0f2e2e22bfe8b795b89333a0c))
+* add resolution field ([#808](https://github.com/instill-ai/pipeline-backend/issues/808)) ([f15f6bf](https://github.com/instill-ai/pipeline-backend/commit/f15f6bf5de177f23b529a2b87c1809b9c10f4265))
+* add task sync ([#793](https://github.com/instill-ai/pipeline-backend/issues/793)) ([41a1eeb](https://github.com/instill-ai/pipeline-backend/commit/41a1eeb47e7eadcf17c6db7141418aef41affb19))
+* **component,audio:** add TASK_DETECT_ACTIVITY and TASK_SEGMENT ([#762](https://github.com/instill-ai/pipeline-backend/issues/762)) ([9e92a31](https://github.com/instill-ai/pipeline-backend/commit/9e92a311e76a4e9d937f372fa3f203d9b84ee22e))
+* **component,http:** refactor `restapi` component to `http` component ([#797](https://github.com/instill-ai/pipeline-backend/issues/797)) ([c2b1862](https://github.com/instill-ai/pipeline-backend/commit/c2b18620a38e5f69cfc4b30e61f371a8824d9cd1))
+* **component:** add error handling for missing conversation ([#806](https://github.com/instill-ai/pipeline-backend/issues/806)) ([54cc616](https://github.com/instill-ai/pipeline-backend/commit/54cc616726c2b6dd5c6ddaf3b677443de73e71c0))
+* **component:** inject global secrets as environment variables ([#786](https://github.com/instill-ai/pipeline-backend/issues/786)) ([8d842a6](https://github.com/instill-ai/pipeline-backend/commit/8d842a668c96e4557db9a63421d94f4c352be17b))
+* convert time type to string ([#809](https://github.com/instill-ai/pipeline-backend/issues/809)) ([7de8465](https://github.com/instill-ai/pipeline-backend/commit/7de84651b4019e69806c1b0ea461084a37d6fbd5))
+* **googledrive:** add the tasks for google drive ([#725](https://github.com/instill-ai/pipeline-backend/issues/725)) ([b6fe968](https://github.com/instill-ai/pipeline-backend/commit/b6fe9686c688f9c3b7de5942510e2818d893ade6))
+* **integration:** identify supported OAuth integrations through global secrets ([#791](https://github.com/instill-ai/pipeline-backend/issues/791)) ([5a96453](https://github.com/instill-ai/pipeline-backend/commit/5a964532e6cbc35eb6778626f8a5161c29dffc15))
+* **minio:** import updated minio package and add tag on file upload ([#779](https://github.com/instill-ai/pipeline-backend/issues/779)) ([ef86318](https://github.com/instill-ai/pipeline-backend/commit/ef863189a3c1617e7a47e89ba1632863f7e122ec))
+* revamp Instill Format ([#774](https://github.com/instill-ai/pipeline-backend/issues/774)) ([24153e2](https://github.com/instill-ai/pipeline-backend/commit/24153e2c57ba4ce508059a0bd1af8528b07b5ed3))
+* support `length` attribute for array data ([#810](https://github.com/instill-ai/pipeline-backend/issues/810)) ([fb4f4f7](https://github.com/instill-ai/pipeline-backend/commit/fb4f4f73064327be025999a4f9fbf31b8ac2e230))
+* **web:** refactor the web operator ([#772](https://github.com/instill-ai/pipeline-backend/issues/772)) ([ae4e3c2](https://github.com/instill-ai/pipeline-backend/commit/ae4e3c2d21951d666bbf0b0ae4c384d18446e41f))
+
+
+### Bug Fixes
+
+* **component,image:** fix missing show score draw ([#801](https://github.com/instill-ai/pipeline-backend/issues/801)) ([a405bf7](https://github.com/instill-ai/pipeline-backend/commit/a405bf70070222108a8500ae946dbf653c1ca1a0))
+* fix bug not to return error if there is no app or conversation ([#816](https://github.com/instill-ai/pipeline-backend/issues/816)) ([a946cfd](https://github.com/instill-ai/pipeline-backend/commit/a946cfd22ede65f608d79efcbb2d52e14fadc692))
+* fix iterator upstream check ([#794](https://github.com/instill-ai/pipeline-backend/issues/794)) ([671971f](https://github.com/instill-ai/pipeline-backend/commit/671971f5e1ed69f87f76ec75b2a0d96db3637e62))
+* **run:** add metadata retention handler ([#800](https://github.com/instill-ai/pipeline-backend/issues/800)) ([25ec0c2](https://github.com/instill-ai/pipeline-backend/commit/25ec0c227521cebbe79e873be9e859649341079d))
+* **run:** add namespace id in response ([#811](https://github.com/instill-ai/pipeline-backend/issues/811)) ([8d29ffb](https://github.com/instill-ai/pipeline-backend/commit/8d29ffbc813545833f9694b78bef07a31b428a22))
+* **run:** rename pipeline run columns and fix tests ([#776](https://github.com/instill-ai/pipeline-backend/issues/776)) ([98f1e00](https://github.com/instill-ai/pipeline-backend/commit/98f1e001f683c8b5ae72b553e3217f0459ac3ef8))
+* **slack:** correct link to OAuth config in documentation ([#805](https://github.com/instill-ai/pipeline-backend/issues/805)) ([aa0752d](https://github.com/instill-ai/pipeline-backend/commit/aa0752dca35ff49960c35d6082e295d3a11b16d6))
+
+## [0.45.2-beta](https://github.com/instill-ai/pipeline-backend/compare/v0.45.1-beta...v0.45.2-beta) (2024-10-29)
+
+
+### Features
+
+* **collection:** add task split ([#780](https://github.com/instill-ai/pipeline-backend/issues/780)) ([1719e48](https://github.com/instill-ai/pipeline-backend/commit/1719e48fbea205ed8a8ac84c6d6c262fd89ab86e))
+
+
+### Miscellaneous Chores
+
+* release v0.45.2-beta ([777a362](https://github.com/instill-ai/pipeline-backend/commit/777a36212312797e32beedae1d9b967c3b8f04e4))
+
+## [0.45.1-beta](https://github.com/instill-ai/pipeline-backend/compare/v0.45.0-beta...v0.45.1-beta) (2024-10-25)
+
+
+### Features
+
+* **component, video:** implemented fractional fps value - fps as number, and added some debugging flags for ffmpeg ([#765](https://github.com/instill-ai/pipeline-backend/issues/765)) ([8a93442](https://github.com/instill-ai/pipeline-backend/commit/8a934420a01c986181b1660d1c13178ff7e79105))
+* **component:** add support for streaming in the Anthropic and Mistral components ([#781](https://github.com/instill-ai/pipeline-backend/issues/781)) ([66f15fe](https://github.com/instill-ai/pipeline-backend/commit/66f15fe935183bb79610cd397d259b95bd253a17))
+
+
+### Miscellaneous Chores
+
+* release v0.45.1-beta ([60a58bd](https://github.com/instill-ai/pipeline-backend/commit/60a58bde6c163e50dfa53c522187441ec098d043))
+
+## [0.45.0-beta](https://github.com/instill-ai/pipeline-backend/compare/v0.44.0-beta...v0.45.0-beta) (2024-10-23)
+
+
+### Features
+
+* **run:** add requester id in list pipeline run response ([#770](https://github.com/instill-ai/pipeline-backend/issues/770)) ([a89a03d](https://github.com/instill-ai/pipeline-backend/commit/a89a03d2a9f4961831da88a25d7a2f6dd3bb3f73))
+
+## [0.44.0-beta](https://github.com/instill-ai/pipeline-backend/compare/v0.43.2-beta...v0.44.0-beta) (2024-10-22)
+
+
+### Features
+
+* **collection:** add concat ([#748](https://github.com/instill-ai/pipeline-backend/issues/748)) ([04d1467](https://github.com/instill-ai/pipeline-backend/commit/04d1467cef580ee423651ebed86a7a5ef6e64b64))
+* **compogen:** improve Title Case capitalization ([#757](https://github.com/instill-ai/pipeline-backend/issues/757)) ([f956ead](https://github.com/instill-ai/pipeline-backend/commit/f956ead548168e8cd2e1a20a99cf6bb0d03a057f))
+* **component:** update documentation URL to component ID ([#749](https://github.com/instill-ai/pipeline-backend/issues/749)) ([d4083c2](https://github.com/instill-ai/pipeline-backend/commit/d4083c262801f302ad705c830db2907c1273560a))
+* **instillmodel:** implement instill model embedding ([#727](https://github.com/instill-ai/pipeline-backend/issues/727)) ([17d88bc](https://github.com/instill-ai/pipeline-backend/commit/17d88bc46dbcf118dcf3e181c6886de6f309b29e))
+* **run:** run logging data list by requester API ([#730](https://github.com/instill-ai/pipeline-backend/issues/730)) ([e1e844b](https://github.com/instill-ai/pipeline-backend/commit/e1e844b4f6db226a29ba2937f0f05005a5899d49))
+* **slack:** enable OAuth 2.0 integration ([#758](https://github.com/instill-ai/pipeline-backend/issues/758)) ([8043dc5](https://github.com/instill-ai/pipeline-backend/commit/8043dc5b1d564cecd4f5227ad57b926189ab243f))
+* standardize the tag naming convention ([#767](https://github.com/instill-ai/pipeline-backend/issues/767)) ([fd0500f](https://github.com/instill-ai/pipeline-backend/commit/fd0500f56f91c006f82a3201f435c032215a4513))
+* **web:** refactor web operator ([#753](https://github.com/instill-ai/pipeline-backend/issues/753)) ([700805f](https://github.com/instill-ai/pipeline-backend/commit/700805f492652482c07472a9965ae06e1384c86c))
+
+
+### Bug Fixes
+
+* **groq:** use credential-supported model in example ([#752](https://github.com/instill-ai/pipeline-backend/issues/752)) ([fc90435](https://github.com/instill-ai/pipeline-backend/commit/fc904350c978dd5c57fbe24aaf32be8945f3b9f5))
+* **run:** not return minio error in list pipeline run ([#744](https://github.com/instill-ai/pipeline-backend/issues/744)) ([4d0afa1](https://github.com/instill-ai/pipeline-backend/commit/4d0afa16baa0ddcfe29371d8d3106eb7d066574f))
+* **whatsapp:** fix dir name ([#763](https://github.com/instill-ai/pipeline-backend/issues/763)) ([029aef9](https://github.com/instill-ai/pipeline-backend/commit/029aef91459feca4c1111372ba386d9272a1870c))
+
 ## [0.43.2-beta](https://github.com/instill-ai/pipeline-backend/compare/v0.43.1-beta...v0.43.2-beta) (2024-10-16)