Integrations page

.vscode/markdown.code-snippets

@@ -251,4 +251,33 @@
 		],
 		"description": "For marking features as being added in a certain version"
 	},
+
+	"Integrations Tabs code": {
+		"prefix": "integrations-tabs-code",
+        "scope": "markdown",
+		"body": [
+			"<!-- Delete these imports if already imported in the file -->",
+			"import Tabs from '@theme/Tabs';",
+			"import TabItem from '@theme/TabItem';",
+			"",
+			"<Tabs groupId=\"languages\">",
+			"<TabItem value=\"py\" label=\"Python Client v4\">",
+			"",
+			"```python",
+			"# Python v4 example goes here",
+			"# note there is an empty line before this code snippet,",
+			"# without the empty line the code won't render",
+			"```",
+			"</TabItem>",
+			"<TabItem value=\"py3\" label=\"Python Client v3\">",
+			"",
+			"```python",
+			"# Python v3 example goes here",
+			"```",
+			"",
+			"</TabItem>",
+			"</Tabs>",
+		],
+		"description": "Adds the tabs section for code examples"
+	},
 }

developers/integrations/cloud-hyperscalers/aws/index.md

@@ -0,0 +1,19 @@
+---
+title: Amazon Web Services
+sidebar_position: 1
+---
+
+Launch a Weaviate cluster from the Amazon Web Services (AWS) marketplace. AWS supports model provider integrations through SageMaker and Bedrock.
+
+## AWS and Weaviate
+Weaviate integrates with [AWS](https://aws.amazon.com/) infrastructure and services like [SageMaker](https://aws.amazon.com/sagemaker/) and [Bedrock](https://aws.amazon.com/bedrock/).
+
+* [Deploy Weaviate from AWS Marketplace](/developers/weaviate/installation/aws-marketplace)
+* [Run embedding and generative models on SageMaker and Bedrock](/developers/weaviate/model-providers/aws)
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| RAG with Cohere models on Amazon Bedrock and Weaviate | The example use case generates targeted advertisements for vacation stay listings based on a target audience. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/cloud-hyperscalers/aws/RAG_Cohere_Weaviate_v4_client.ipynb)

developers/integrations/cloud-hyperscalers/google/index.md

@@ -0,0 +1,22 @@
+---
+title: Google Cloud Platform
+sidebar_position: 2
+---
+
+Launch a Weaviate cluster from the Google Cloud Platform (GCP) marketplace. Weaviate integrates with Google AI Studio and Google Vertex AI.
+
+## GCP and Weaviate 
+Weaviate integrates with GCP infrastructure and services like Google [AI Studio](https://ai.google.dev/aistudio) and [Vertex AI](https://cloud.google.com/vertex-ai?hl=en).
+
+* [Deploy Weaviate on GCP Marketplace](/developers/weaviate/installation/gc-marketplace)
+* [Run embedding and generative models on Vertex AI and AI Studio](/developers/weaviate/model-providers/google)
+
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Build a multimodal application using Gemini Flash | This notebook shows you how to use Weaviate and Gemini Flash to build a multimodal application. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/cloud-hyperscalers/google/gemini/multimodal-and-gemini-flash/NY-Roadshow-Gemini.ipynb) |
+| BigQuery and Weaviate | Sync data between BigQuery and Weaviate using DSPy. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/cloud-hyperscalers/google/bigquery/BigQuery-Weaviate-DSPy-RAG.ipynb) |
+| Semantic Search with Gemini Ultra | This notebook shows you how to use Weaviate and Gemini Ultra. |[Notebook](https://github.com/weaviate/recipes/blob/main/integrations/cloud-hyperscalers/google/gemini/gemini-ultra/gemini-ultra-weaviate.ipynb) |

developers/integrations/cloud-hyperscalers/index.md

@@ -0,0 +1,10 @@
+---
+title: Cloud Hyperscalers
+sidebar_position: 1
+---
+
+Cloud hyperscalers offer a variety of services and infrastructure for large-scale computing and storage.
+
+Learn about how Weaviate integrates with these hyperscalers: 
+* [Amazon Web Services](/developers/integrations/cloud-hyperscalers/aws)
+* [Google Cloud Platform](/developers/integrations/cloud-hyperscalers/google)

developers/integrations/compute-infrastructure/index.md

@@ -0,0 +1,11 @@
+---
+title: Compute Infrastructure
+sidebar_position: 2
+image: og/integrations/home.jpg
+---
+
+Compute Infrastructure solutions provide managed platforms for computationally intensive workloads. Use these platforms to develop, deploy, and scale your application.
+
+Learn about how Weaviate integrates with these solutions:
+* [Modal](/developers/integrations/compute-infrastructure/modal)
+* [Replicate](/developers/integrations/compute-infrastructure/replicate)

developers/integrations/compute-infrastructure/modal/index.md

@@ -0,0 +1,20 @@
+---
+title: Modal
+sidebar_position: 1
+---
+
+[Modal](https://modal.com/) provides a serverless platform that has on-demand access to GPUs and a custom high-performance container runtime. Use Modal to easily deploy and automatically scale high-performance applications.
+
+## Modal and Weaviate
+Weaviate leverages Modal's serverless infrastructure for fast embedding generation and for fast generative model calls.
+
+To dynamically scale your application based on workload demands, [host the Weaviate client](https://modal.com/docs/examples/vector-analogies-wikipedia#deploy-a-serverless-read-only-weaviate-client-with-modal) on Modal.
+
+
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Embed and Search Text at Scale with Modal and Weaviate | Build a full application that discovers analogies between Wikipedia articles. Combine serverless infrastructure from Modal with the search and storage capabilities of Weaviate. | [Blog post](/blog/modal-and-weaviate#modal-serverless-infrastructure-for-gpus-and-more), [Notebook](https://github.com/weaviate/recipes/tree/main/integrations/compute-infrastructure/modal), [Modal Guide](https://modal.com/docs/examples/vector-analogies-wikipedia#deploy-a-serverless-read-only-weaviate-client-with-modal)|

developers/integrations/compute-infrastructure/replicate/index.md

@@ -0,0 +1,16 @@
+---
+title: Replicate
+sidebar_position: 2
+---
+
+[Replicate](https://replicate.com/) is a platform that allows users to run machine learning models through a cloud API. They host many open-source models, including embedding and language models. Users can run or fine-tune the models to meet their application needs.
+
+## Replicate and Weaviate
+To use models on Replicate, you need to use [LlamaIndex](https://docs.llamaindex.ai/en/stable/api_reference/llms/replicate/) or [LangChain](https://python.langchain.com/v0.2/docs/integrations/llms/replicate/) and connect it to your Weaviate vector store.
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+Run Llama 2 on Replicate | Build a LlamaIndex query engine using Replicate, Weaviate, and Llama 2 as the generative model. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/compute-infrastructure/replicate-llama2/notebook.ipynb) |

developers/integrations/data-platforms/confluent-cloud/index.md

@@ -0,0 +1,32 @@
+---
+title: Confluent Cloud
+sidebar_position: 1
+image: og/integrations/home.jpg
+---
+
+Confluent Cloud is a fully managed Apache Kafka service that offers real-time data streaming with seamless integration across major cloud providers, high performance, and robust security features. Learn more at [Confluent Cloud](https://www.confluent.io/confluent-cloud/).
+
+## Confluent and Weaviate
+You can stream data from Confluent Cloud to Weaviate using the [Weaviate Confluent Connector](https://github.com/weaviate/confluent-connector). For setup and usage details, refer to the connector's [README](https://github.com/weaviate/confluent-connector/blob/main/README.md).
+
+
+## Our Resources 
+The resources are broken into two categories: 
+1. [**Hands on Learning**](#hands-on-learning): Build your technical understanding with end-to-end tutorials.
+
+2. [**Read and Listen**](#read-and-listen): Develop your conceptual understanding of these technologies.
+
+### Hands on Learning
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| PySpark Notebook | Learn how to use PySpark | [Notebook](https://github.com/weaviate/confluent-connector/blob/main/notebooks/01_demo_pyspark.ipynb) |
+| Confluent-Weaviate Connector with Embedded | This notebook shows you how to use the confluent-weaviate connector with Weaviate Embedded. | [Notebook](https://github.com/weaviate/confluent-connector/blob/main/notebooks/02_demo_confluent_weaviate.ipynb) |
+| Confluent-Weaviate Connector with Weaviate Cloud | This notebook shows you how to use the confluent-weaviate connector with Weaviate Cloud. | [Notebook](https://github.com/weaviate/confluent-connector/blob/main/notebooks/03_demo_confluent_wcs.ipynb) |
+| Confluent-Weaviate Connector with Weaviate Cloud and Databricks | Learn how to integrate the confluent-weaviate connector with Weaviate Cloud and Databricks. | [Notebook](https://github.com/weaviate/confluent-connector/blob/main/notebooks/04_demo_confluent_databricks.ipynb) |
+
+
+### Read and Listen
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Make Real-Time AI a Reality with Weaviate + Confluent | Learn how to build an application using Weaviate and Confluent. | [Blog](/blog/confluent-and-weaviate) |

developers/integrations/data-platforms/context-data/index.md

@@ -0,0 +1,24 @@
+---
+title: Context Data
+sidebar_position: 2
+image: og/integrations/home.jpg
+---
+
+VectorETL by [Context Data](https://contextdata.ai/) is a modular no-code Python framework designed to help AI and Data Engineers to:
+
+* Quickly extract data from multiple data sources (databases, cloud storage, and local files)
+* Embed using major models (including OpenAI, Cohere, and Google Gemini)
+* Write to vector databases 
+
+## Context Data and Weaviate
+Weaviate is a [target connection](https://context-data.gitbook.io/context-data-1/adding-target-connections#add-a-weaviate-target-connection) in Context Data. You will need to create a Weaviate cluster and input the URL and authentication credentials when prompted. 
+
+
+## Our Resources 
+[**Hands on Learning**](#hands-on-learning): Build your technical understanding with end-to-end tutorials.
+
+### Hands on Learning
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| VectorETL into Weaviate | Three examples showing you how to ingest data from Google Cloud Storage, Postgress, and S3 into Weaviate. | [Notebook](https://github.com/weaviate/recipes/tree/main/integrations/data-platforms/context-data) |

developers/integrations/data-platforms/firecrawl/index.md

@@ -0,0 +1,19 @@
+---
+title: Firecrawl
+sidebar_position: 3
+image: og/integrations/home.jpg
+---
+
+[Firecrawl](https://www.firecrawl.dev/) is an AI first web scraping tool that allows you to easily crawl and extract clean, structured data from websites. It is an API service that converts a URL into clean markdown or structured data. 
+
+## Firecrawl and Weaviate 
+Firecrawl handles the complexities of web scraping like proxies, caching, rate limits, and dynamic content, delivering markdown or JSON output that is ready to be ingested into vector databases like Weaviate.
+
+## Our Resources 
+[**Hands on Learning**](#hands-on-learning): Build your technical understanding with end-to-end tutorials.
+
+### Hands on Learning
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Firecrawl to Weaviate | This notebook will show you how to scrape webpages using Firecrawl and load it into Weaviate. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/data-platforms/web-search/firecrawl/firecrawl-to-weaviate.ipynb)

developers/integrations/data-platforms/index.md

@@ -0,0 +1,14 @@
+---
+title: Data Platforms
+sidebar_position: 3
+image: og/integrations/home.jpg
+---
+
+Data Platforms offer robust solutions for managing, processing, and analyzing large volumes of data. These platforms provide tools and services that facilitate seamless data ingestion directly into Weaviate. 
+
+Learn about how Weaviate integrates with these solutions:
+* [Confluent Cloud](/developers/integrations/data-platforms/confluent-cloud)
+* [Context Data](/developers/integrations/data-platforms/context-data/)
+* [Spark](/developers/integrations/data-platforms/spark)
+* [Unstructured](/developers/integrations/data-platforms/unstructured)
+* [Firecrawl](/developers/integrations/data-platforms/firecrawl/)

developers/integrations/data-platforms/spark/index.md

@@ -0,0 +1,31 @@
+---
+title: Spark
+sidebar_position: 4
+image: og/integrations/home.jpg
+---
+
+[Apache Spark](https://spark.apache.org/docs/latest/api/python/index.html) (or the Python API, PySpark) is an open-source data processing framework used for real-time, large-scale data processing. 
+
+## Spark and Weaviate
+The Spark connector enables you to easily ingest data from Spark data structures into Weaviate. 
+
+You can learn more about the Weaviate connector for Spark in [this repository](https://github.com/weaviate/spark-connector).
+
+## Our Resources 
+The resources are broken into two categories: 
+1. [**Hands on Learning**](#hands-on-learning): Build your technical understanding with end-to-end tutorials.
+
+2. [**Read and Listen**](#read-and-listen): Develop your conceptual understanding of these technologies.
+
+### Hands on Learning
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Weaviate Tutorial | Learn how to ingest data into Weaviate with Spark. | [Tutorial](/developers/weaviate/tutorials/spark-connector)
+| Using the Spark Connector for Weaviate | Learn how to take data from a Spark dataframe and feed it into Weaviate. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/data-platforms/spark/spark-connector.ipynb) |
+
+### Read and Listen 
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| The Sphere Dataset in Weaviate | Learn how to import and query the Sphere dataset in Weaviate. | [Blog](/blog/sphere-dataset-in-weaviate) |
+| The Details Behind the Sphere Dataset in Weaviate | The details on how we ingested ~1 billion article snippets into Weaviate. | [Blog](/blog/details-behind-the-sphere-dataset-in-weaviate) |

developers/integrations/data-platforms/unstructured/index.md

@@ -0,0 +1,32 @@
+---
+title: Unstructured
+sidebar_position: 5
+image: og/integrations/home.jpg
+---
+
+[Unstructured](https://unstructured.io/) offers a platform and tools for ingesting and processing unstructured data for building retrieval augmented generation (RAG) applications.
+
+Unstructured has two offerings:
+1. [Unstructured Platform](https://docs.unstructured.io/platform/overview): No-code user interface
+2. [Serverless API](https://docs.unstructured.io/api-reference/api-services/overview): Run scripts or code to call the Unstructured Ingest CLI
+
+## Unstructured and Weaviate
+You can ingest and process data from a variety of sources into your Weaviate cluster. Weaviate is a destination connector in the [Platform](https://docs.unstructured.io/platform/platform-destination-connectors/weaviate) and [API](https://docs.unstructured.io/api-reference/ingest/destination-connector/weaviate). 
+
+
+## Our Resources 
+The resources are broken into two categories: 
+1. [**Hands on Learning**](#hands-on-learning): Build your technical understanding with end-to-end tutorials.
+
+2. [**Read and Listen**](#read-and-listen): Develop your conceptual understanding of these technologies.
+
+### Hands on Learning
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Ingest Data from S3 into Weaviate | Learn how to use Unstructured's API to grab data from an S3 bucket and load it into Weaviate | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/data-platforms/unstructured/unstructured_weaviate.ipynb)
+
+### Read and Listen 
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Ingesting PDFs into Weaviate | Learn how to load and transform PDF documents into Weaviate. | [Blog](/blog/ingesting-pdfs-into-weaviate) |

developers/integrations/index.md

@@ -0,0 +1,37 @@
+---
+title: Integrations
+sidebar_position: 0
+image: og/integrations/home.jpg
+---
+
+Weaviate's integration ecosystem enables developers to build various applications leveraging Weaviate and another technology.
+
+All the notebooks and code examples are on [Weaviate Recipes](https://github.com/weaviate/recipes)!
+
+<img
+    src={require('./ecosystem.png').default}
+    alt="alt"
+    style={{ maxWidth: "70%", display: "block", marginLeft: "auto", marginRight: "auto"}}
+/>
+
+
+## About the Categories
+The ecosystem is divided into these categories:
+
+* **Cloud Hyperscalers** - Large-scale computing and storage
+* **Compute Infrastructure** - Run and scale containerized applications
+* **Data Platforms** - Data ingestion and web scraping 
+* **LLM Frameworks** - Build generative AI applications
+* **Observability and Evaluation** - Monitor and analyze generative AI workflows
+
+
+
+## List of Companies
+
+| Company Category | Companies |
+|------------------|-----------|
+| Cloud Hyperscalers | [Google](/developers/integrations/cloud-hyperscalers/google), [AWS](/developers/integrations/cloud-hyperscalers/aws), Azure|
+| Compute Infrastructure | [Modal](/developers/integrations/compute-infrastructure/modal), [Replicate](/developers/integrations/compute-infrastructure/replicate) |
+| Data Platforms |[Confluent Cloud](/developers/integrations/data-platforms/confluent-cloud), [Firecrawl](/developers/integrations/data-platforms/firecrawl), [Spark](/developers/integrations/data-platforms/spark), [Unstructured](/developers/integrations/data-platforms/unstructured) |
+| LLM Frameworks | [Composio](/developers/integrations/llm-frameworks/composio/), [DSPy](/developers/integrations/llm-frameworks/dspy/), [LangChain](/developers/integrations/llm-frameworks/langchain/), [LlamaIndex](/developers/integrations/llm-frameworks/llamaindex/), [Semantic Kernel](/developers/integrations/llm-frameworks/semantic-kernel/) |
+| Operations | [Arize](/developers/integrations/operations/arize/), [Langtrace](/developers/integrations/operations/langtrace/), [LangWatch](/developers/integrations/operations/langwatch/), [Nomic](/developers/integrations/operations/nomic/), [Ragas](/developers/integrations/operations/ragas/), [Weights & Biases](/developers/integrations/operations/wandb/) |

developers/integrations/llm-frameworks/composio/index.md

@@ -0,0 +1,26 @@
+---
+title: Composio
+sidebar_position: 1
+image: og/integrations/home.jpg
+---
+
+[Composio](https://docs.composio.dev/introduction/intro/overview) uses function calling for managing and integrating tools with language models and AI agents.  
+
+## Composio and Weaviate 
+With Weaviate's retrieval, you can make the agent more personalized and context-aware. 
+
+The integration is supported through our LangChain vector store. You need to have a running Weaviate instance and create the vector store with: 
+```python
+WeaviateVectorStore.from_documents( )
+```
+
+You can learn more about how to create a vector store [here](https://python.langchain.com/v0.2/docs/integrations/vectorstores/weaviate/#step-1-data-import).
+
+## Our Resources 
+[**Hands on Learning**](#hands-on-learning): Build your technical understanding with end-to-end tutorials.
+
+### Hands on Learning
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Gmail Agent | Integrate Composio's Gmail tool with Weaviate to create an agent that will respond to new messages. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/function-calling/composio/agent.ipynb) |