Integrations page

.vscode/markdown.code-snippets

@@ -251,4 +251,33 @@
 		],
 		"description": "For marking features as being added in a certain version"
 	},
+
+	"Integrations Tabs code": {
+		"prefix": "integrations-tabs-code",
+        "scope": "markdown",
+		"body": [
+			"<!-- Delete these imports if already imported in the file -->",
+			"import Tabs from '@theme/Tabs';",
+			"import TabItem from '@theme/TabItem';",
+			"",
+			"<Tabs groupId=\"languages\">",
+			"<TabItem value=\"py\" label=\"Python Client v4\">",
+			"",
+			"```python",
+			"# Python v4 example goes here",
+			"# note there is an empty line before this code snippet,",
+			"# without the empty line the code won't render",
+			"```",
+			"</TabItem>",
+			"<TabItem value=\"py3\" label=\"Python Client v3\">",
+			"",
+			"```python",
+			"# Python v3 example goes here",
+			"```",
+			"",
+			"</TabItem>",
+			"</Tabs>",
+		],
+		"description": "Adds the tabs section for code examples"
+	},
 }

developers/integrations/cloud-hyperscalers/aws/index.md

@@ -0,0 +1,19 @@
+---
+title: Amazon Web Services
+sidebar_position: 1
+---
+
+Launch a Weaviate cluster from the Amazon Web Services (AWS) marketplace. AWS supports model provider integrations through SageMaker and Bedrock.
+
+## AWS and Weaviate
+Weaviate integrates with [AWS](https://aws.amazon.com/) infrastructure and services like [SageMaker](https://aws.amazon.com/sagemaker/) and [Bedrock](https://aws.amazon.com/bedrock/).
+
+* [Deploy Weaviate from AWS Marketplace](/developers/weaviate/installation/aws-marketplace)
+* [Run embedding and generative models on SageMaker and Bedrock](/developers/weaviate/model-providers/aws)
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| RAG with Cohere models on Amazon Bedrock and Weaviate | The example use case generates targeted advertisements for vacation stay listings based on a target audience. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/cloud-hyperscalers/aws/RAG_Cohere_Weaviate_v4_client.ipynb)

developers/integrations/cloud-hyperscalers/google/index.md

@@ -0,0 +1,22 @@
+---
+title: Google Cloud Platform
+sidebar_position: 2
+---
+
+Launch a Weaviate cluster from the Google Cloud Platform (GCP) marketplace. Weaviate integrates with Google AI Studio and Google Vertex AI.
+
+## GCP and Weaviate 
+Weaviate integrates with GCP infrastructure and services like Google [AI Studio](https://ai.google.dev/aistudio) and [Vertex AI](https://cloud.google.com/vertex-ai?hl=en).
+
+* [Deploy Weaviate on GCP Marketplace](/developers/weaviate/installation/gc-marketplace)
+* [Run embedding and generative models on Vertex AI and AI Studio](/developers/weaviate/model-providers/google)
+
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Build a multimodal application using Gemini Flash | This notebook shows you how to use Weaviate and Gemini Flash to build a multimodal application. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/cloud-hyperscalers/google/gemini/multimodal-and-gemini-flash/NY-Roadshow-Gemini.ipynb) |
+| BigQuery and Weaviate | Sync data between BigQuery and Weaviate using DSPy. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/cloud-hyperscalers/google/bigquery/BigQuery-Weaviate-DSPy-RAG.ipynb) |
+| Semantic Search with Gemini Ultra | This notebook shows you how to use Weaviate and Gemini Ultra. |[Notebook](https://github.com/weaviate/recipes/blob/main/integrations/cloud-hyperscalers/google/gemini/gemini-ultra/gemini-ultra-weaviate.ipynb) |

developers/integrations/cloud-hyperscalers/index.md

@@ -0,0 +1,10 @@
+---
+title: Cloud Hyperscalers
+sidebar_position: 1
+---
+
+Cloud hyperscalers offer a variety of services and infrastructure for large-scale computing and storage.
+
+Learn about how Weaviate integrates with these hyperscalers: 
+* [Amazon Web Services](/developers/integrations/cloud-hyperscalers/aws)
+* [Google Cloud Platform](/developers/integrations/cloud-hyperscalers/google)

developers/integrations/compute-infrastructure/index.md

@@ -0,0 +1,11 @@
+---
+title: Compute Infrastructure
+sidebar_position: 2
+image: og/integrations/home.jpg
+---
+
+Compute Infrastructure solutions provide managed platforms for computationally intensive workloads. Use these platforms to develop, deploy, and scale your application.
+
+Learn about how Weaviate integrates with these solutions:
+* [Modal](/developers/integrations/compute-infrastructure/modal)
+* [Replicate](/developers/integrations/compute-infrastructure/replicate)

developers/integrations/compute-infrastructure/modal/index.md

@@ -0,0 +1,20 @@
+---
+title: Modal
+sidebar_position: 1
+---
+
+[Modal](https://modal.com/) provides a serverless platform that has on-demand access to GPUs and a custom high-performance container runtime. Use Modal to easily deploy and automatically scale high-performance applications.
+
+## Modal and Weaviate
+Weaviate leverages Modal's serverless infrastructure for fast embedding generation and for fast generative model calls.
+
+To dynamically scale your application based on workload demands, [host the Weaviate client](https://modal.com/docs/examples/vector-analogies-wikipedia#deploy-a-serverless-read-only-weaviate-client-with-modal) on Modal.
+
+
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Embed and Search Text at Scale with Modal and Weaviate | Build a full application that discovers analogies between Wikipedia articles. Combine serverless infrastructure from Modal with the search and storage capabilities of Weaviate. | [Blog post](/blog/modal-and-weaviate#modal-serverless-infrastructure-for-gpus-and-more), [Notebook](https://github.com/weaviate/recipes/tree/main/integrations/compute-infrastructure/modal), [Modal Guide](https://modal.com/docs/examples/vector-analogies-wikipedia#deploy-a-serverless-read-only-weaviate-client-with-modal)|

developers/integrations/compute-infrastructure/replicate/index.md

@@ -0,0 +1,16 @@
+---
+title: Replicate
+sidebar_position: 2
+---
+
+[Replicate](https://replicate.com/) is a platform that allows users to run machine learning models through a cloud API. They host many open-source models, including embedding and language models. Users can run or fine-tune the models to meet their application needs.
+
+## Replicate and Weaviate
+To use models on Replicate, you need to use [LlamaIndex](https://docs.llamaindex.ai/en/stable/api_reference/llms/replicate/) or [LangChain](https://python.langchain.com/v0.2/docs/integrations/llms/replicate/) and connect it to your Weaviate vector store.
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+Run Llama 2 on Replicate | Build a LlamaIndex query engine using Replicate, Weaviate, and Llama 2 as the generative model. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/compute-infrastructure/replicate-llama2/notebook.ipynb) |

developers/integrations/data-pipeline/firecrawl/index.md

@@ -0,0 +1,15 @@
+---
+title: Firecrawl
+sidebar_position: 2
+---
+
+[content]
+
+## Firecrawl and Weaviate
+[content]
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |

developers/integrations/data-pipeline/index.md

@@ -0,0 +1,13 @@
+---
+title: Data Platforms
+sidebar_position: 3
+image: og/integrations/home.jpg
+---
+
+Data platforms 
+
+Learn about how Weaviate integrates with these data platforms: 
+* [Confluent Cloud](/developers/integrations/data-platforms/confluent-cloud)
+* [Firecrawl](/developers/integrations/data-platforms/firecrawl)
+* [Spark](/developers/integrations/data-platforms/spark)
+* [Unstructured](/developers/integrations/data-platforms/unstructured)

developers/integrations/data-pipeline/spark/index.md

@@ -0,0 +1,16 @@
+---
+title: Spark
+sidebar_position: 3
+---
+
+Home page for Spark
+
+## Spark and Weaviate
+[content]
+
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |

developers/integrations/data-pipeline/unstructured/index.md

@@ -0,0 +1,15 @@
+---
+title: Unstructured
+sidebar_position: 4
+---
+
+[content]
+
+## Unstructured and Weaviate
+[content]
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |

developers/integrations/data-platforms/confluent-cloud/index.md

@@ -0,0 +1,15 @@
+---
+title: Confluent Cloud
+sidebar_position: 1
+---
+[content]
+
+## Confluent Cloud and Weaviate
+[content]
+
+
+## Our Resources 
+**Hands on Learning**: Build your technical understanding with end-to-end tutorials.
+
+| Topic | Description | Resource | 
+| --- | --- | --- |

developers/integrations/index.md

@@ -0,0 +1,37 @@
+---
+title: Integrations
+sidebar_position: 0
+image: og/integrations/home.jpg
+---
+
+Weaviate's integration ecosystem enables developers to build various applications leveraging Weaviate and another technology.
+
+All the notebooks and code examples are on [Weaviate Recipes](https://github.com/weaviate/recipes)!
+
+<img
+    src={require('./ecosystem.png').default}
+    alt="alt"
+    style={{ maxWidth: "70%", display: "block", marginLeft: "auto", marginRight: "auto"}}
+/>
+
+
+## About the Categories
+The ecosystem is divided into these categories:
+
+* **Cloud Hyperscalers** - Large-scale computing and storage
+* **Compute Infrastructure** - Run and scale containerized applications
+* **Data Platforms** - Data ingestion and web scraping 
+* **LLM Frameworks** - Build generative AI applications
+* **Observability and Evaluation** - Monitor and analyze generative AI workflows
+
+
+
+## List of Companies
+
+| Company Category | Companies |
+|------------------|-----------|
+| Cloud Hyperscalers | [Google](/developers/integrations/cloud-hyperscalers/google), [AWS](/developers/integrations/cloud-hyperscalers/aws), Azure|
+| Compute Infrastructure | [Modal](/developers/integrations/compute-infrastructure/modal), [Replicate](/developers/integrations/compute-infrastructure/replicate) |
+| Data Platforms |[Confluent Cloud](/developers/integrations/data-platforms/confluent-cloud), [Firecrawl](/developers/integrations/data-platforms/firecrawl), [Spark](/developers/integrations/data-platforms/spark), [Unstructured](/developers/integrations/data-platforms/unstructured) |
+| LLM Frameworks | DSPy, LangChain, LlamaIndex, Semantic Kernel |
+| Observability and Evaluation | Arize, Langtrace, LangWatch, Nomic, Ragas, Weights & Biases |

developers/integrations/llm-frameworks/dspy/dspy.md

@@ -0,0 +1,65 @@
+---
+title: DSPy
+sidebar_position: 1
+image: og/docs/more-resources.jpg
+---
+[DSPy](https://github.com/stanfordnlp/dspy) is a framework for programming language models created by Stanford NLP. DSPy introduces two key concepts: **programming model** and **optimizers**.
+
+- **Programing model**: The programming model lets you define a series of components that make a language model request. Components include input and output fields, task descriptions, and calls to a vector database like Weaviate.
+
+- **Optimizers**: Optimizers compile your DSPy program to tune the language model prompt and/or the weights.
+
+## DSPy and Weaviate
+
+Weaviate is integrated with DSPy through the retriever module! You only need to connect to your Weaviate cluster through [WCD](https://console.weaviate.cloud/) or localhost, and pass in your collection:
+
+```python
+weaviate_client = weaviate.Client("http://localhost:8080") # or pass in your WCD cluster url
+
+retriever_module = WeaviateRM("WeaviateBlogChunk", # collection name
+                    weaviate_client=weaviate_client)
+```
+
+## Our Resources 
+Here are a few resources on using DSPy from the Weaviate team!
+
+The resources are broken into two categories: 
+1. [**Hands on Learning**](#hands-on-learning): Build your technical understanding with end-to-end tutorials.
+
+2. [**Read and Listen**](#read-and-listen): Develop your conceptual understanding of these technologies.
+
+
+### Hands on Learning 
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| Getting Started with RAG in DSPy | Learn about the four components needed to build a DSPy program: Installation, settings, datasets with `dspy.Example`, LLM metrics, DSPy programming model, and optimization. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/1.Getting-Started-with-RAG-in-DSPy.ipynb), [Video](https://youtu.be/CEuUG4Umfxs?si=4Gp8gR9glmoMJNaU) |
+| DSPy + Weaviate for the Next Generation of LLM Apps | Build a 4-layer DSPy program for generating blog posts from queries. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/2.Writing-Blog-Posts-with-DSPy.ipynb), [Video](https://youtu.be/ickqCzFxWj0?si=AxCbD9tq2cbAH6bB)|
+| RAG with Persona | Build a compound AI system with DSPy, Cohere, and Weaviate, where you'll add a persona to the language model. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/fullstack-recipes/RAGwithPersona/4.RAG-with-Persona.ipynb), [Post](https://twitter.com/ecardenas300/status/1765444492348243976)|
+| Adding Depth to RAG Programs | Enhancing DSPy programs by integrating unique input-output examples and multiple LLMs. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/3.Adding-Depth-to-RAG-Programs.ipynb), [Video](https://youtu.be/0c7Ksd6BG88?si=YUF2wm1ncUTkSuPQ) |
+| Hurricane: Writing Blog Posts with Generative Feedback Loops | Introduction to Hurricane, a web app for demonstrating generative feedback loops with blog posts. | [Notebook](https://github.com/weaviate-tutorials/Hurricane), [Blog](/blog/hurricane-generative-feedback-loops) |
+| Structured Outputs with DSPy | The three methods for structuring outputs in DSPy programs. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/4.Structured-Outputs-with-DSPy.ipynb), [Video](https://youtu.be/tVw3CwrN5-8?si=P7fWeXzQ7p-2SFYF) |
+| Building RAG with Command R+ from Cohere, DSPy, and Weaviate | Overview of Command R+ with a quick RAG demo in DSPy. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/llms/Command-R-Plus.ipynbb), [Video](https://youtu.be/6dgXALb_5Ag?si=nSX2AnmpbUau_2JF) |
+| Advanced Optimizers in DSPy | Dive into optimizing DSPy programs with various techniques. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/5.Advanced-Optimizers.ipynb) |
+| Llama 3 RAG Demo with DSPy Optimization, Ollama, and Weaviate | Integrating Llama3 with DSPy and optimizing prompts with MIPRO. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/llms/Llama3.ipynb), [Video](https://youtu.be/1h3_h8t3L14?si=G4d-aY5Ynpv8ckea)|
+| BigQuery and Weaviate orchestrated with DSPy | Build an end-to-end RAG pipeline that uses BigQuery and Weaviate using DSPy. | [Notebook](https://github.com/weaviate/recipes/blob/main/integrations/cloud-hyperscalers/google/bigquery/BigQuery-Weaviate-DSPy-RAG.ipynb)|
+
+
+
+### Read and Listen
+
+| Topic | Description | Resource | 
+| --- | --- | --- |
+| DSPy and ColBERT with Omar Khattab! - Weaviate Podcast #85 | Omar Khattab joins the Weaviate podcast to discuss DSPy and ColBERT. | [Video](https://www.youtube.com/watch?v=CDung1LnLbY) |
+| DSPy Explained| The core concepts of DSPy and walks through the introduction notebooks showing how to compile a simple retrieve-then-read RAG program and Multi-Hop RAG Program. | [Video](https://youtu.be/41EfOY0Ldkc?si=sFieUeHc9rXRn6uk)|
+| XMC.dspy with Karel D'Oosterlinck - Weaviate Podcast #87 | Karel D'Oosterlinck joins the Weaviate podcast to discuss IReRa (Infer-Retrieve-Rank). | [Video](https://youtu.be/_ye26_8XPcs?si=ZBodgHbOcaq2Kwky)
+| Intro to DSPy: Goodbye Prompting, Hello Programming | Overview of DSPy and how it solves the fragility problem in LLM-based applications. | [Blog](https://towardsdatascience.com/intro-to-dspy-goodbye-prompting-hello-programming-4ca1c6ce3eb9)|
+| Fine-Tuning Cohere’s Reranker | Generate synthetic data with DSPy to fine-tune Cohere’s reranker model. |[Blog](/blog/fine-tuning-coheres-reranker)|
+| Your Language Model Deserves Better Prompting | Overview of the DSPy optimizers for prompt tuning. | [Blog](/blog/dspy-optimizers)|
+
+
+## Questions and feedback
+
+import DocsFeedback from '/_includes/docs-feedback.mdx';
+
+<DocsFeedback/>

developers/integrations/llm-frameworks/index.md

@@ -0,0 +1,11 @@
+---
+title: LLM Frameworks
+sidebar_position: 4
+image: og/integrations/home.jpg
+---
+
+Add content here
+
+
+
+Add content here

developers/integrations/llm-frameworks/langchain/langchain.md

@@ -0,0 +1,4 @@
+---
+title: LangChain
+sidebar_position: 4
+---

developers/integrations/llm-frameworks/llamaindex/LlamaIndex.md

@@ -0,0 +1,7 @@
+---
+title: LlamaIndex
+sidebar_position: 2
+image: og/docs/llamaindex.jpg
+---
+
+Integrations page - welcome message

developers/integrations/llm-frameworks/semantic-kernel/semantic-kernel.md

@@ -0,0 +1,4 @@
+---
+title: Semantic Kernel
+sidebar_position: 3
+---

developers/integrations/observability-and-evaluation/arize/index.md

@@ -0,0 +1,7 @@
+---
+title: Arize
+sidebar_position: 1
+image: og/integrations/home.jpg
+---
+
+Home page for Arize

developers/integrations/observability-and-evaluation/index.md

@@ -0,0 +1,7 @@
+---
+title: Observability and Evaluation
+sidebar_position: 5
+image: og/integrations/home.jpg
+---
+
+Home page for Observability and Evaluation

developers/integrations/observability-and-evaluation/langtrace/index.md

@@ -0,0 +1,7 @@
+---
+title: Langtrace
+sidebar_position: 2
+image: og/integrations/home.jpg
+---
+
+Home page for Langtrace

developers/integrations/observability-and-evaluation/nomic/index.md

@@ -0,0 +1,7 @@
+---
+title: Nomic
+sidebar_position: 3
+image: og/integrations/home.jpg
+---
+
+Home page for Nomic