Initial commit

.github/workflows/static.yml

@@ -34,7 +34,7 @@ jobs:
 
       - name: Install dependencies
         run: |
-          pip install -r build_requirements.txt
+          pip install -r dev/build_requirements.txt
       # Build the example book
       - name: Build book
         run: |

LICENSE.md

@@ -0,0 +1,24 @@
+# DB license
+Definitions. 
+
+Agreement: The agreement between Databricks, Inc., and you governing the use of the Databricks Services, as that term is defined in the Master Cloud Services Agreement (MCSA) located at www.databricks.com/legal/mcsa.
+
+Licensed Materials: The source code, object code, data, and/or other works to which this license applies.
+
+Scope of Use. You may not use the Licensed Materials except in connection with your use of the Databricks Services pursuant to the Agreement. Your use of the Licensed Materials must comply at all times with any restrictions applicable to the Databricks Services, generally, and must be used in accordance with any applicable documentation. You may view, use, copy, modify, publish, and/or distribute the Licensed Materials solely for the purposes of using the Licensed Materials within or connecting to the Databricks Services. If you do not agree to these terms, you may not view, use, copy, modify, publish, and/or distribute the Licensed Materials.
+
+Redistribution. You may redistribute and sublicense the Licensed Materials so long as all use is in compliance with these terms. In addition:
+
+You must give any other recipients a copy of this License;
+You must cause any modified files to carry prominent notices stating that you changed the files;
+You must retain, in any derivative works that you distribute, all copyright, patent, trademark, and attribution notices, excluding those notices that do not pertain to any part of the derivative works; and
+If a "NOTICE" text file is provided as part of its distribution, then any derivative works that you distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the derivative works.
+You may add your own copyright statement to your modifications and may provide additional license terms and conditions for use, reproduction, or distribution of your modifications, or for any such derivative works as a whole, provided your use, reproduction, and distribution of the Licensed Materials otherwise complies with the conditions stated in this License.
+
+Termination. This license terminates automatically upon your breach of these terms or upon the termination of your Agreement. Additionally, Databricks may terminate this license at any time on notice. Upon termination, you must permanently delete the Licensed Materials and all copies thereof.
+
+DISCLAIMER; LIMITATION OF LIABILITY.
+
+THE LICENSED MATERIALS ARE PROVIDED “AS-IS” AND WITH ALL FAULTS. DATABRICKS, ON BEHALF OF ITSELF AND ITS LICENSORS, SPECIFICALLY DISCLAIMS ALL WARRANTIES RELATING TO THE LICENSED MATERIALS, EXPRESS AND IMPLIED, INCLUDING, WITHOUT LIMITATION, IMPLIED WARRANTIES, CONDITIONS AND OTHER TERMS OF MERCHANTABILITY, SATISFACTORY QUALITY OR FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. DATABRICKS AND ITS LICENSORS TOTAL AGGREGATE LIABILITY RELATING TO OR ARISING OUT OF YOUR USE OF OR DATABRICKS’ PROVISIONING OF THE LICENSED MATERIALS SHALL BE LIMITED TO ONE THOUSAND ($1,000) DOLLARS.  IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE LICENSED MATERIALS OR THE USE OR OTHER DEALINGS IN THE LICENSED MATERIALS.
+
+For the latest text of this license: https://www.databricks.com/legal/db-license

NOTICE.md

@@ -0,0 +1,93 @@
+Copyright (2024) Databricks, Inc.
+
+This Software includes software developed at Databricks (https://www.databricks.com/) and its use is subject to the included LICENSE file.
+—--
+
+This Software contains code from the following open source projects, licensed under the Apache 2.0 license:
+
+databricks/databricks-sdk-java - https://github.com/databricks/databricks-sdk-java
+Copyright (2023) Databricks, Inc.
+License - https://github.com/databricks/databricks-sdk-java/blob/main/LICENSE
+Notice - Copyright (2023) Databricks, Inc.
+
+mlflow/mlflow - https://github.com/mlflow/mlflow
+Copyright mlflow authors
+License - https://github.com/mlflow/mlflow/blob/master/LICENSE.txt
+—--
+
+This Software contains code from the following open source projects, licensed under the MIT license:
+
+langchain-ai/langchain - https://github.com/langchain-ai/langchain
+Copyright (c) LangChain, Inc.
+License - https://github.com/langchain-ai/langchain/blob/master/LICENSE
+—--
+
+This Software can optionally be used with code from the following open source projects, which are subject to their corresponding license:
+
+huggingface/transformers - https://github.com/huggingface/transformers
+Copyright transformers authors
+License - https://github.com/huggingface/transformers/blob/main/LICENSE
+
+tesseract-ocr/tesseract - https://github.com/tesseract-ocr/tesseract
+Copyright tesseract authors - https://github.com/tesseract-ocr/tesseract/blob/main/AUTHORS
+License - https://github.com/tesseract-ocr/tesseract/blob/main/LICENSE
+
+nltk/nltk - https://github.com/nltk/nltk
+Copyright nltk authors - https://github.com/nltk/nltk/blob/develop/AUTHORS.md
+License - https://github.com/nltk/nltk/blob/develop/LICENSE.txt
+
+PrithivirajDamodaran/FlashRank - https://github.com/PrithivirajDamodaran/FlashRank
+Copyright FlashRank authors
+License - https://github.com/PrithivirajDamodaran/FlashRank/blob/main/LICENSE
+
+Azure/azure-sdk-for-python/sdk/documentintelligence/azure-ai-documentintelligence - https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/documentintelligence/azure-ai-documentintelligence/azure/ai/documentintelligence
+Copyright (c) Microsoft Corporation.
+License - https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/documentintelligence/azure-ai-documentintelligence/LICENSE
+
+openai/tiktoken - https://github.com/openai/tiktoken
+Copyright (c) 2022 OpenAI, Shantanu Jain
+License - https://github.com/openai/tiktoken/blob/main/LICENSE
+
+matthewwithanm/python-markdownify - https://github.com/matthewwithanm/python-markdownify
+Copyright 2012-2018 Matthew Tretter
+License - https://github.com/matthewwithanm/python-markdownify/blob/develop/LICENSE
+
+Unstructured-IO/unstructured-python-client - https://github.com/Unstructured-IO/unstructured-python-client
+Copyright (c) 2023 Unstructured-IO
+License - https://github.com/Unstructured-IO/unstructured-python-client/blob/main/LICENSE.md
+
+Unstructured-IO/unstructured-inference - https://github.com/Unstructured-IO/unstructured-inference
+Copyright unstructured-IO authors
+License - https://github.com/Unstructured-IO/unstructured-inference/blob/main/LICENSE
+
+euske/pdfminer - https://github.com/euske/pdfminer
+Copyright (c) 2004-2019  Yusuke Shinyama
+License - https://github.com/euske/pdfminer/blob/master/LICENSE
+
+JessicaTegner/pypandoc - https://github.com/JessicaTegner/pypandoc
+Copyright (c) 2011 Juho Vepsäläinen
+License - https://github.com/JessicaTegner/pypandoc/blob/master/LICENSE
+
+pytorch/pytorch - https://github.com/pytorch/pytorch
+Copyright pytorch authors - https://github.com/pytorch/pytorch/blob/main/LICENSE
+License - https://github.com/pytorch/pytorch/blob/main/LICENSE
+Notice - https://github.com/pytorch/pytorch/blob/main/NOTICE
+
+py-pdf/pypdf - https://github.com/py-pdf/pypdf
+Copyright (c) 2006-2008, Mathieu Fenniak
+Some contributions copyright (c) 2007, Ashish Kulkarni <[email protected]>
+Some contributions copyright (c) 2014, Steve Witham <[email protected]> All rights reserved.
+License - https://github.com/py-pdf/pypdf/blob/main/LICENSE
+
+scikit-learn/scikit-learn - https://github.com/scikit-learn/scikit-learn
+Copyright (c) 2007-2024 The scikit-learn developers.
+All rights reserved.
+License - https://github.com/scikit-learn/scikit-learn?tab=BSD-3-Clause-1-ov-file#
+
+pymupdf/PyMuPDF - https://github.com/pymupdf/PyMuPDF
+Copyright PyMuPDF authors
+License - https://github.com/pymupdf/PyMuPDF?tab=AGPL-3.0-1-ov-file#
+
+jgm/pandoc - https://github.com/jgm/pandoc
+Copyright (C) 2006-2024 John MacFarlane <jgm at berkeley dot edu>
+License - https://github.com/jgm/pandoc/blob/main/COPYRIGHT

README.md

@@ -1 +1,2 @@
-# genai-cookbook
+# Databricks Generative AI Cookbook
+

dev/README.md

@@ -0,0 +1,13 @@
+# Databricks Mosaic Generative AI Cookbook
+
+To start working on this book:
+- clone the repo; `cd cookbook`
+- use your preferred approach to starting a new python environment
+- in that environment, `pip install jupyter-book`
+- build and preview the site with `jupyter-book build --all genai_cookbook`
+
+The homepage is at `genai_cookbook/index.md`
+
+The content pages are in `genai_cookbook/nbs/`
+
+Jupyter book is fairly flexible and offers a lot of different options for formatting, cross-referencing, adding formatted callouts, etc. Read more at the [Jupyter Book docs](https://jupyterbook.org/en/stable/intro.html).

dev/build_requirements.txt

@@ -0,0 +1 @@
+jupyter-book

dev/dev_requirements.txt

@@ -0,0 +1,2 @@
+jupyter-book
+livereload

dev/server.py

@@ -0,0 +1,14 @@
+from livereload import Server, shell
+import os
+
+# Define the subfolder you want to serve
+subfolder = '../genai_cookbook/_build/html/'
+
+# Create a new server instance
+server = Server()
+
+# Watch the subfolder (and subdirectories) for changes to HTML, CSS, and JavaScript files
+server.watch(subfolder, shell('touch reload.txt'), delay=1)
+
+# Serve the specified subfolder on port 8000
+server.serve(root=subfolder, port=8000)

dev/watch.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+fswatch -o --exclude "../genai_cookbook/_build" ../genai_cookbook | xargs -n1 -I{} jupyter-book build --all ../genai_cookbook

genai_cookbook/_config.yml

@@ -0,0 +1,30 @@
+# Book settings
+# Learn more at https://jupyterbook.org/customize/config.html
+
+title: Databricks Generative AI Cookbook
+author: The Databricks GenAI Community
+logo: logo2.png
+
+# Force re-execution of notebooks on each build.
+# See https://jupyterbook.org/content/execute.html
+execute:
+  execute_notebooks: 'off'
+
+# Information about where the book exists on the web
+repository:
+  url: https://github.com/databricks-genai-cookbook/cookbook/
+  path_to_book: ./genai_cookbook  # Optional path to your book, relative to the repository root
+  branch: main  # Which branch of the repository should be used when creating links (optional)
+
+# Add GitHub buttons to your book
+# See https://jupyterbook.org/customize/config.html#add-a-link-to-your-repository
+html:
+  favicon: images/index/favicon.ico
+  use_issues_button: true
+  use_repository_button: true
+  home_page_in_navbar: false
+  google_analytics_id: G-6BZ4NTBHVJ
+  use_multitoc_numbering: false
+sphinx:
+  config:
+    html_show_copyright: false

genai_cookbook/_toc.yml

@@ -0,0 +1,59 @@
+# Table of contents
+# Learn more at https://jupyterbook.org/customize/toc.html
+
+format: jb-book
+root: index
+parts:
+- caption: "Overview"
+  chapters:
+    - file: index-2
+- caption: "Learn"
+  numbered: true
+  chapters:
+  - file: nbs/1-introduction-to-rag
+  - file: nbs/2-fundamentals-unstructured
+    sections:
+    - file: nbs/2-fundamentals-unstructured-data-pipeline
+    - file: nbs/2-fundamentals-unstructured-chain
+    - file: nbs/2-fundamentals-unstructured-eval
+    - file: nbs/2-fundamentals-unstructured-llmops
+  - file: nbs/3-deep-dive
+    sections:
+    - file: nbs/3-deep-dive-data-pipeline
+    - file: nbs/3-deep-dive-chain
+  - file: nbs/4-evaluation
+    sections:
+    - file: nbs/4-evaluation-eval-sets
+    - file: nbs/4-evaluation-metrics
+    - file: nbs/4-evaluation-infra
+  - file: nbs/5-rag-development-workflow
+- caption: "Implement"
+  numbered: false
+  chapters:
+  - file: nbs/5-hands-on-requirements
+  - file: nbs/6-implement-overview
+# - caption: "Build: POC"
+#   numbered: true
+#   chapters:
+  - file: nbs/5-hands-on-build-poc
+# - caption: "Build: Evaluate RAG POC's quality"
+#   numbered: true
+#   chapters:
+  - file: nbs/5-hands-on-curate-eval-set
+  - file: nbs/5-hands-on-evaluate-poc
+# - caption: "Build: Debug & iterate on RAG quality"  
+#   numbered: true
+#   chapters:
+  # - file: nbs/5-hands-on-improve-quality
+  #   sections:
+  - file: nbs/5-hands-on-improve-quality-step-1
+    sections:
+    - file: nbs/5-hands-on-improve-quality-step-1-retrieval
+    - file: nbs/5-hands-on-improve-quality-step-1-generation
+  - file: nbs/5-hands-on-improve-quality-step-2
+    sections:
+    - file: nbs/5-hands-on-improve-quality-step-2-data-pipeline
+  - file: nbs/5-hands-on-deploy-and-monitor
+# - caption: "Deploy & monitor a RAG app"
+#   chapters:
+

genai_cookbook/index-2.md

@@ -0,0 +1,80 @@
+---
+title: Databricks Generative AI Cookbook
+---
+
+# Databricks Generative AI Cookbook
+
+**TLDR;** this cookbook and its sample code will take you from initial POC to high-quality production-ready application using [Mosaic AI Quality Lab](https://docs.databricks.com/generative-ai/agent-evaluation/index.html) and [Mosaic AI Agent Framework](https://docs.databricks.com/generative-ai/retrieval-augmented-generation.html) on the Databricks platform.
+
+The Databricks Generative AI Cookbook is a definitive how-to guide for building *high-quality* generative AI applications. *High-quality* applications are applications that:
+1. **Accurate:** provide correct responses
+2. **Safe:** do not deliver harmful or insecure responses
+3. **Governed:** respect data permissions & access controls and track lineage
+
+Developed in partnership with Mosaic AI's research team, this cookbook lays out Databricks best-practice development workflow for building high-quality RAG apps: *evaluation driven development.* It outlines the most relevant knobs & approaches that can increase RAG application quality and provides a comprehensive repository of sample code implementing those techniques. 
+
+```{important}
+- Only have 10 minutes and want to see a demo of Mosaic AI Agent Framework & Quality lab?  Start [here](https://DBDEMO).
+- Want to hop into code and deploy a RAG POC using your data?  Start [here](./nbs/6-implement-overview.md).
+- Don't have any data, but want to deploy a sample RAG application?  Start here.
+```
+
+```{image} images/index/dbxquality.png
+:align: center
+```
+
+<br/>
+
+
+```{image} images/5-hands-on/review_app2.gif
+:align: center
+```
+
+<br/>
+
+This cookbook is intended for use with the Databricks platform.  Specifically:
+- [Mosaic AI Agent Framework](https://docs.databricks.com/generative-ai/retrieval-augmented-generation.html) which provides a fast developer workflow with enterprise-ready LLMops & governance
+- [Mosaic AI Quality Lab](https://docs.databricks.com/generative-ai/agent-evaluation/index.html) which provides reliable, quality measurement using proprietary AI-assisted LLM judges to measure quality metrics that are powered by human feedback collected through an intuitive web-based chat UI
+
+
+# Retrieval-augmented generation (RAG)
+
+> This first release focuses on retrieval-augmented generation (RAG).  Future releases will include the other popular generative AI techniques: agents & function calling, prompt engineering, fine tuning, and pre-training.
+
+The RAG cookbook is divided into 2 sections:
+1. [**Learn:**](#learn) Understand the required components of a production-ready, high-quality RAG application
+2. [**Implement:**](#implement) Use our sample code to follow an evaluation-driven workflow for delivering a high-quality RAG application
+
+## Code-based quick starts
+
+| Time required | Outcome | Link |
+|------ | ---- | ---- |
+| 🕧 <br/> 10 minutes | Sample RAG app deployed to web-based chat app that collects feedback | [RAG Demo]((https://DBDEMO)) |
+| 🕧🕧🕧 <br/>60 minutes | POC RAG app with *your data* deployed to a chat UI that can collect feedback from your business stakeholders | [Build & deploy a POC](./nbs/5-hands-on-build-poc.md)|
+| 🕧🕧 <br/>30 minutes | Comprehensive quality/cost/latency evaluation of your POC app | - [Evaluate your POC](./nbs/5-hands-on-evaluate-poc.md) <br/> - [Identify the root causes of quality issues](./nbs/5-hands-on-improve-quality-step-1.md) |
+
+
+
+## Table of contents
+<!--
+**Table of contents**
+1. [RAG overview](./nbs/1-introduction-to-rag): Understand how RAG works at a high-level
+2. [RAG fundamentals](./nbs/2-fundamentals-unstructured): Understand the key components in a RAG app
+3. [RAG quality knobs](./nbs/3-deep-dive): Understand the knobs Databricks recommends tuning improve RAG app quality 
+4. [RAG quality evaluation deep dive](./nbs/4-evaluation): Understand how RAG evaluation works, including creating evaluation sets, the quality metrics that matter, and required developer tooling
+5. [Evaluation-driven development](nbs/5-rag-development-workflow.md): Understand Databricks recommended development workflow for building, testing, and deploying a high-quality RAG application: evaluation-driven development-->
+
+```{tableofcontents}
+```
+<!--
+#### Implement
+
+**Table of contents**
+
+
+1. [Gather Requirements](./nbs/5-hands-on-requirements.md): Requirements you must discover from stakeholders before building a RAG app
+2. [Deploy POC to Collect Stakeholder Feedback](./nbs/5-hands-on-build-poc.md): Launch a proof of concept (POC) to gather feedback from stakeholders and understand baseline quality
+3. [Evaluate POC’s Quality](./nbs/5-hands-on-evaluate-poc.md): Assess the quality of your POC to identify areas for improvement
+4. [Root Cause & Iteratively Fix Quality Issues](./nbs/5-hands-on-improve-quality.md): Diagnose the root causes of any quality issues and apply iterative fixes to improve the app's quality
+5. [Deploy & Monitor](./nbs/5-hands-on-deploy-and-monitor.md): Deploy the finalized RAG app to production and continuously monitor its performance to ensure sustained quality.
+-->