Plugin/merge main 0217 (#13800)

Signed-off-by: yihong0618 <[email protected]> Signed-off-by: -LAN- <[email protected]> Signed-off-by: xhe <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: kurokobo <[email protected]> Co-authored-by: Hiroshi Fujita <[email protected]> Co-authored-by: NFish <[email protected]> Co-authored-by: Gen Sato <[email protected]> Co-authored-by: eux <[email protected]> Co-authored-by: huangzhuo1949 <[email protected]> Co-authored-by: huangzhuo <[email protected]> Co-authored-by: lotsik <[email protected]> Co-authored-by: crazywoola <[email protected]> Co-authored-by: Wu Tianwei <[email protected]> Co-authored-by: nite-knite <[email protected]> Co-authored-by: Jyong <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: gakkiyomi <[email protected]> Co-authored-by: CN-P5 <[email protected]> Co-authored-by: CN-P5 <[email protected]> Co-authored-by: Chuehnone <[email protected]> Co-authored-by: yihong <[email protected]> Co-authored-by: Kevin9703 <[email protected]> Co-authored-by: -LAN- <[email protected]> Co-authored-by: Boris Feld <[email protected]> Co-authored-by: mbo <[email protected]> Co-authored-by: mabo <[email protected]> Co-authored-by: Warren Chen <[email protected]> Co-authored-by: KVOJJJin <[email protected]> Co-authored-by: JzoNgKVO <[email protected]> Co-authored-by: jiandanfeng <[email protected]> Co-authored-by: zhu-an <[email protected]> Co-authored-by: zhaoqingyu.1075 <[email protected]> Co-authored-by: 海狸大師 <[email protected]> Co-authored-by: Xu Song <[email protected]> Co-authored-by: rayshaw001 <[email protected]> Co-authored-by: Ding Jiatong <[email protected]> Co-authored-by: Bowen Liang <[email protected]> Co-authored-by: JasonVV <[email protected]> Co-authored-by: le0zh <[email protected]> Co-authored-by: zhuxinliang <[email protected]> Co-authored-by: k-zaku <[email protected]> Co-authored-by: Joel <[email protected]> Co-authored-by: luckylhb90 <[email protected]> Co-authored-by: hobo.l <[email protected]> Co-authored-by: jiangbo721 <[email protected]> Co-authored-by: 刘江波 <[email protected]> Co-authored-by: Shun Miyazawa <[email protected]> Co-authored-by: EricPan <[email protected]> Co-authored-by: crazywoola <[email protected]> Co-authored-by: zxhlyh <[email protected]> Co-authored-by: sino <[email protected]> Co-authored-by: Jhvcc <[email protected]> Co-authored-by: lowell <[email protected]> Co-authored-by: Ademílson Tonato <[email protected]> Co-authored-by: Ademílson Tonato <[email protected]> Co-authored-by: IWAI, Masaharu <[email protected]> Co-authored-by: Yueh-Po Peng (Yabi) <[email protected]> Co-authored-by: 非法操作 <[email protected]> Co-authored-by: Jason <[email protected]> Co-authored-by: Xin Zhang <[email protected]> Co-authored-by: yjc980121 <[email protected]> Co-authored-by: heyszt <[email protected]> Co-authored-by: Abdullah AlOsaimi <[email protected]> Co-authored-by: Abdullah AlOsaimi <[email protected]> Co-authored-by: Yingchun Lai <[email protected]> Co-authored-by: Hash Brown <[email protected]> Co-authored-by: zuodongxu <[email protected]> Co-authored-by: Masashi Tomooka <[email protected]> Co-authored-by: aplio <[email protected]> Co-authored-by: Obada Khalili <[email protected]> Co-authored-by: Nam Vu <[email protected]> Co-authored-by: Kei YAMAZAKI <[email protected]> Co-authored-by: TechnoHouse <[email protected]> Co-authored-by: Riddhimaan-Senapati <[email protected]> Co-authored-by: MaFee921 <[email protected]> Co-authored-by: te-chan <[email protected]> Co-authored-by: HQidea <[email protected]> Co-authored-by: Joshbly <[email protected]> Co-authored-by: xhe <[email protected]> Co-authored-by: weiwenyan-dev <[email protected]> Co-authored-by: ex_wenyan.wei <[email protected]> Co-authored-by: engchina <[email protected]> Co-authored-by: engchina <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: 呆萌闷油瓶 <[email protected]> Co-authored-by: Kemal <[email protected]> Co-authored-by: Lazy_Frog <[email protected]> Co-authored-by: Novice Lee <[email protected]> Co-authored-by: Yi Xiao <[email protected]> Co-authored-by: Steven sun <[email protected]> Co-authored-by: steven <[email protected]> Co-authored-by: Kalo Chin <[email protected]> Co-authored-by: Katy Tao <[email protected]> Co-authored-by: depy <[email protected]> Co-authored-by: 胡春东 <[email protected]> Co-authored-by: Junjie.M <[email protected]> Co-authored-by: MuYu <[email protected]> Co-authored-by: Naoki Takashima <[email protected]> Co-authored-by: Summer-Gu <[email protected]> Co-authored-by: Fei He <[email protected]> Co-authored-by: ybalbert001 <[email protected]> Co-authored-by: Yuanbo Li <[email protected]> Co-authored-by: douxc <[email protected]> Co-authored-by: liuzhenghua <[email protected]> Co-authored-by: Novice <[email protected]> Co-authored-by: Wu Jiayang <[email protected]> Co-authored-by: Your Name <[email protected]> Co-authored-by: kimjion <[email protected]> Co-authored-by: AugNSo <[email protected]> Co-authored-by: llinvokerl <[email protected]> Co-authored-by: liusurong.lsr <[email protected]> Co-authored-by: Vasu Negi <[email protected]> Co-authored-by: Hundredwz <[email protected]> Co-authored-by: Xiyuan Chen <[email protected]>
langgenius · Feb 17, 2025 · f16abd4 · f16abd4
1 parent 9153b6d
commit f16abd4
Show file tree

Hide file tree

Showing 114 changed files with 1,056 additions and 553 deletions.
diff --git a/.github/workflows/api-tests.yml b/.github/workflows/api-tests.yml
@@ -27,6 +27,9 @@ jobs:
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Setup Poetry and Python ${{ matrix.python-version }}
         uses: ./.github/actions/setup-poetry

diff --git a/.github/workflows/build-push.yml b/.github/workflows/build-push.yml
@@ -80,10 +80,12 @@ jobs:
           cache-to: type=gha,mode=max,scope=${{ matrix.service_name }}
 
       - name: Export digest
+        env:
+          DIGEST: ${{ steps.build.outputs.digest }}
         run: |
           mkdir -p /tmp/digests
-          digest="${{ steps.build.outputs.digest }}"
-          touch "/tmp/digests/${digest#sha256:}"
+          sanitized_digest=${DIGEST#sha256:}
+          touch "/tmp/digests/${sanitized_digest}"
 
       - name: Upload digest
         uses: actions/upload-artifact@v4
@@ -133,10 +135,15 @@ jobs:
 
       - name: Create manifest list and push
         working-directory: /tmp/digests
+        env:
+          IMAGE_NAME: ${{ env[matrix.image_name_env] }}
         run: |
           docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
-            $(printf '${{ env[matrix.image_name_env] }}@sha256:%s ' *)
+            $(printf "$IMAGE_NAME@sha256:%s " *)
 
       - name: Inspect image
+        env:
+          IMAGE_NAME: ${{ env[matrix.image_name_env] }}
+          IMAGE_VERSION: ${{ steps.meta.outputs.version }}
         run: |
-          docker buildx imagetools inspect ${{ env[matrix.image_name_env] }}:${{ steps.meta.outputs.version }}
+          docker buildx imagetools inspect "$IMAGE_NAME:$IMAGE_VERSION"
diff --git a/.github/workflows/db-migration-test.yml b/.github/workflows/db-migration-test.yml
@@ -20,6 +20,9 @@ jobs:
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Setup Poetry and Python
         uses: ./.github/actions/setup-poetry

diff --git a/.github/workflows/expose_service_ports.sh b/.github/workflows/expose_service_ports.sh
@@ -9,6 +9,6 @@ yq eval '.services["pgvecto-rs"].ports += ["5431:5432"]' -i docker/docker-compos
 yq eval '.services["elasticsearch"].ports += ["9200:9200"]' -i docker/docker-compose.yaml
 yq eval '.services.couchbase-server.ports += ["8091-8096:8091-8096"]' -i docker/docker-compose.yaml
 yq eval '.services.couchbase-server.ports += ["11210:11210"]' -i docker/docker-compose.yaml
-yq eval '.services.tidb.ports += ["4000:4000"]' -i docker/docker-compose.yaml
+yq eval '.services.tidb.ports += ["4000:4000"]' -i docker/tidb/docker-compose.yaml
 
 echo "Ports exposed for sandbox, weaviate, tidb, qdrant, chroma, milvus, pgvector, pgvecto-rs, elasticsearch, couchbase"
diff --git a/.github/workflows/style.yml b/.github/workflows/style.yml
@@ -18,6 +18,9 @@ jobs:
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Check changed files
         id: changed-files
@@ -60,6 +63,9 @@ jobs:
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Check changed files
         id: changed-files
@@ -96,6 +102,9 @@ jobs:
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Check changed files
         id: changed-files
@@ -124,6 +133,9 @@ jobs:
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Check changed files
         id: changed-files

diff --git a/.github/workflows/tool-test-sdks.yaml b/.github/workflows/tool-test-sdks.yaml
@@ -26,6 +26,9 @@ jobs:
 
     steps:
       - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Use Node.js ${{ matrix.node-version }}
         uses: actions/setup-node@v4

diff --git a/.github/workflows/translate-i18n-base-on-english.yml b/.github/workflows/translate-i18n-base-on-english.yml
@@ -16,6 +16,7 @@ jobs:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 2 # last 2 commits
+          persist-credentials: false
 
       - name: Check for file changes in i18n/en-US
         id: check_files

diff --git a/.github/workflows/vdb-tests.yml b/.github/workflows/vdb-tests.yml
@@ -28,6 +28,9 @@ jobs:
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Setup Poetry and Python ${{ matrix.python-version }}
         uses: ./.github/actions/setup-poetry
@@ -51,7 +54,15 @@ jobs:
       - name: Expose Service Ports
         run: sh .github/workflows/expose_service_ports.sh
 
-      - name: Set up Vector Stores (TiDB, Weaviate, Qdrant, PGVector, Milvus, PgVecto-RS, Chroma, MyScale, ElasticSearch, Couchbase)
+      - name: Set up Vector Store (TiDB)
+        uses: hoverkraft-tech/[email protected]
+        with:
+          compose-file: docker/tidb/docker-compose.yaml
+          services: |
+            tidb
+            tiflash
+
+      - name: Set up Vector Stores (Weaviate, Qdrant, PGVector, Milvus, PgVecto-RS, Chroma, MyScale, ElasticSearch, Couchbase)
         uses: hoverkraft-tech/[email protected]
         with:
           compose-file: |
@@ -67,7 +78,9 @@ jobs:
             pgvector
             chroma
             elasticsearch
-            tidb
+
+      - name: Check TiDB Ready
+        run: poetry run -P api python api/tests/integration_tests/vdb/tidb_vector/check_tiflash_ready.py
 
       - name: Test Vector Stores
         run: poetry run -P api bash dev/pytest/pytest_vdb.sh
diff --git a/.github/workflows/web-tests.yml b/.github/workflows/web-tests.yml
@@ -22,6 +22,9 @@ jobs:
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Check changed files
         id: changed-files

diff --git a/.gitignore b/.gitignore
@@ -163,6 +163,7 @@ docker/volumes/db/data/*
 docker/volumes/redis/data/*
 docker/volumes/weaviate/*
 docker/volumes/qdrant/*
+docker/tidb/volumes/*
 docker/volumes/etcd/*
 docker/volumes/minio/*
 docker/volumes/milvus/*

diff --git a/README.md b/README.md
@@ -108,6 +108,72 @@ Please refer to our [FAQ](https://docs.dify.ai/getting-started/install-self-host
 **7. Backend-as-a-Service**: 
   All of Dify's offerings come with corresponding APIs, so you could effortlessly integrate Dify into your own business logic.
 
+## Feature Comparison
+<table style="width: 100%;">
+  <tr>
+    <th align="center">Feature</th>
+    <th align="center">Dify.AI</th>
+    <th align="center">LangChain</th>
+    <th align="center">Flowise</th>
+    <th align="center">OpenAI Assistants API</th>
+  </tr>
+  <tr>
+    <td align="center">Programming Approach</td>
+    <td align="center">API + App-oriented</td>
+    <td align="center">Python Code</td>
+    <td align="center">App-oriented</td>
+    <td align="center">API-oriented</td>
+  </tr>
+  <tr>
+    <td align="center">Supported LLMs</td>
+    <td align="center">Rich Variety</td>
+    <td align="center">Rich Variety</td>
+    <td align="center">Rich Variety</td>
+    <td align="center">OpenAI-only</td>
+  </tr>
+  <tr>
+    <td align="center">RAG Engine</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+  </tr>
+  <tr>
+    <td align="center">Agent</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">✅</td>
+  </tr>
+  <tr>
+    <td align="center">Workflow</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+  </tr>
+  <tr>
+    <td align="center">Observability</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">❌</td>
+  </tr>
+  <tr>
+    <td align="center">Enterprise Feature (SSO/Access control)</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">❌</td>
+    <td align="center">❌</td>
+  </tr>
+  <tr>
+    <td align="center">Local Deployment</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+  </tr>
+</table>
 
 ## Using Dify
 

diff --git a/README_KL.md b/README_KL.md
@@ -87,9 +87,7 @@ Dify is an open-source LLM app development platform. Its intuitive interface com
 
 ## Feature Comparison
 <table style="width: 100%;">
-  <tr
-
->
+  <tr>
     <th align="center">Feature</th>
     <th align="center">Dify.AI</th>
     <th align="center">LangChain</th>

diff --git a/README_SI.md b/README_SI.md
@@ -106,6 +106,73 @@ Prosimo, glejte naša pogosta vprašanja [FAQ](https://docs.dify.ai/getting-star
 **7. Backend-as-a-Service**: 
   AVse ponudbe Difyja so opremljene z ustreznimi API-ji, tako da lahko Dify brez težav integrirate v svojo poslovno logiko.
 
+## Primerjava Funkcij
+
+<table style="width: 100%;">
+  <tr>
+    <th align="center">Funkcija</th>
+    <th align="center">Dify.AI</th>
+    <th align="center">LangChain</th>
+    <th align="center">Flowise</th>
+    <th align="center">OpenAI Assistants API</th>
+  </tr>
+  <tr>
+    <td align="center">Programski pristop</td>
+    <td align="center">API + usmerjeno v aplikacije</td>
+    <td align="center">Python koda</td>
+    <td align="center">Usmerjeno v aplikacije</td>
+    <td align="center">Usmerjeno v API</td>
+  </tr>
+  <tr>
+    <td align="center">Podprti LLM-ji</td>
+    <td align="center">Bogata izbira</td>
+    <td align="center">Bogata izbira</td>
+    <td align="center">Bogata izbira</td>
+    <td align="center">Samo OpenAI</td>
+  </tr>
+  <tr>
+    <td align="center">RAG pogon</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+  </tr>
+  <tr>
+    <td align="center">Agent</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">✅</td>
+  </tr>
+  <tr>
+    <td align="center">Potek dela</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+  </tr>
+  <tr>
+    <td align="center">Spremljanje</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">❌</td>
+  </tr>
+  <tr>
+    <td align="center">Funkcija za podjetja (SSO/nadzor dostopa)</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">❌</td>
+    <td align="center">❌</td>
+  </tr>
+  <tr>
+    <td align="center">Lokalna namestitev</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+  </tr>
+</table>
 
 ## Uporaba Dify
 
@@ -187,4 +254,4 @@ Zaradi zaščite vaše zasebnosti se izogibajte objavljanju varnostnih vprašanj
 
 ## Licenca
 
-To skladišče je na voljo pod [odprtokodno licenco Dify](LICENSE) , ki je v bistvu Apache 2.0 z nekaj dodatnimi omejitvami.
+To skladišče je na voljo pod [odprtokodno licenco Dify](LICENSE) , ki je v bistvu Apache 2.0 z nekaj dodatnimi omejitvami.
diff --git a/api/Dockerfile b/api/Dockerfile
@@ -55,9 +55,11 @@ RUN \
         # basic environment
         curl nodejs libgmp-dev libmpfr-dev libmpc-dev \
         # For Security
-        # expat libldap-2.5-0 perl libsqlite3-0 zlib1g \
+        expat libldap-2.5-0 perl libsqlite3-0 zlib1g \
         # install a chinese font to support the use of tools like matplotlib
         fonts-noto-cjk \
+        # install a package to improve the accuracy of guessing mime type and file extension
+        media-types \
         # install libmagic to support the use of python-magic guess MIMETYPE
         libmagic1 \
     && apt-get autoremove -y \

diff --git a/api/README.md b/api/README.md
@@ -37,7 +37,13 @@
 
 4. Create environment.
 
-   Dify API service uses [Poetry](https://python-poetry.org/docs/) to manage dependencies. You can execute `poetry shell` to activate the environment.
+   Dify API service uses [Poetry](https://python-poetry.org/docs/) to manage dependencies. First, you need to add the poetry shell plugin, if you don't have it already, in order to run in a virtual environment. [Note: Poetry shell is no longer a native command so you need to install the poetry plugin beforehand]
+
+   ```bash
+   poetry self add poetry-plugin-shell
+   ```
+
+   Then, You can execute `poetry shell` to activate the environment.
 
 5. Install dependencies
 

diff --git a/api/configs/feature/__init__.py b/api/configs/feature/__init__.py
@@ -373,8 +373,8 @@ def WEB_API_CORS_ALLOW_ORIGINS(self) -> list[str]:
     )
 
     RESPECT_XFORWARD_HEADERS_ENABLED: bool = Field(
-        description="Enable or disable the X-Forwarded-For Proxy Fix middleware from Werkzeug"
-        " to respect X-* headers to redirect clients",
+        description="Enable handling of X-Forwarded-For, X-Forwarded-Proto, and X-Forwarded-Port headers"
+        " when the app is behind a single trusted reverse proxy.",
         default=False,
     )
 

diff --git a/api/constants/__init__.py b/api/constants/__init__.py
@@ -15,7 +15,7 @@
 
 if dify_config.ETL_TYPE == "Unstructured":
     DOCUMENT_EXTENSIONS = ["txt", "markdown", "md", "mdx", "pdf", "html", "htm", "xlsx", "xls"]
-    DOCUMENT_EXTENSIONS.extend(("docx", "csv", "eml", "msg", "pptx", "xml", "epub"))
+    DOCUMENT_EXTENSIONS.extend(("doc", "docx", "csv", "eml", "msg", "pptx", "xml", "epub"))
     if dify_config.UNSTRUCTURED_API_URL:
         DOCUMENT_EXTENSIONS.append("ppt")
     DOCUMENT_EXTENSIONS.extend([ext.upper() for ext in DOCUMENT_EXTENSIONS])

diff --git a/api/controllers/console/datasets/datasets.py b/api/controllers/console/datasets/datasets.py
@@ -623,7 +623,6 @@ def get(self):
         match vector_type:
             case (
                 VectorType.RELYT
-                | VectorType.PGVECTOR
                 | VectorType.TIDB_VECTOR
                 | VectorType.CHROMA
                 | VectorType.TENCENT