Merge pull request #4 from databricks/dl/updates

Miscellaneous updates

.gitignore

@@ -1 +1,4 @@
-genai_cookbook/_build
+genai_cookbook/_build
+env
+.env
+.DS_STORE

README.md

@@ -1,2 +1,16 @@
-# Databricks Generative AI Cookbook
+# Retrieval Augmented Generation
 
+Please visit http://ai-cookbook.io for the accompanying documentation for this repo.
+
+This repo provides [learning materials](https://ai-cookbook.io/) and [production-ready code](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code) to build a **high-quality RAG application** using Databricks. The [Mosaic Generative AI Cookbook](https://ai-cookbook.io/) provides:
+  - A conceptual overview and deep dive into various Generative AI design patterns, such as Prompt Engineering, Agents, RAG, and Fine Tuning
+  - An overview of Evaluation-Driven development
+  - The theory of every parameter/knob that impacts quality
+  - How to root cause quality issues and detemermine which knobs are relevant to experiment with for your use case
+  - Best practices for how to experiment with each knob
+
+The [provided code](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code) is intended for use with the Databricks platform.  Specifically:
+- [Mosaic AI Agent Framework]() which provides a fast developer workflow with enterprise-ready LLMops & governance
+- [Mosaic AI Agent Evaluation]() which provides reliable, quality measurement using proprietary AI-assisted LLM judges to measure quality metrics that are powered by human feedback collected through an intuitive web-based chat UI
+
+![Alt text](rag_app_sample_code/dbxquality.png)

genai_cookbook/_config.yml

@@ -2,7 +2,7 @@
 # Learn more at https://jupyterbook.org/customize/config.html
 
 title: Databricks Generative AI Cookbook
-author: The Databricks GenAI Community
+author: Databricks
 logo: logo2.png
 
 # Force re-execution of notebooks on each build.
@@ -12,7 +12,7 @@ execute:
 
 # Information about where the book exists on the web
 repository:
-  url: https://github.com/databricks-genai-cookbook/cookbook/
+  url: https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code
   path_to_book: ./genai_cookbook  # Optional path to your book, relative to the repository root
   branch: main  # Which branch of the repository should be used when creating links (optional)
 

genai_cookbook/index-2.md

@@ -42,8 +42,8 @@ This cookbook is intended for use with the Databricks platform.  Specifically:
 > This first release focuses on retrieval-augmented generation (RAG).  Future releases will include the other popular generative AI techniques: agents & function calling, prompt engineering, fine tuning, and pre-training.
 
 The RAG cookbook is divided into 2 sections:
-1. [**Learn:**](#learn) Understand the required components of a production-ready, high-quality RAG application
-2. [**Implement:**](#implement) Use our sample code to follow an evaluation-driven workflow for delivering a high-quality RAG application
+1. **Learn:** Understand the required components of a production-ready, high-quality RAG application
+2. **Implement:** Use our sample code to follow an evaluation-driven workflow for delivering a high-quality RAG application
 
 ## Code-based quick starts
 

genai_cookbook/index.md

@@ -42,8 +42,8 @@ This cookbook is intended for use with the Databricks platform.  Specifically:
 > This first release focuses on retrieval-augmented generation (RAG).  Future releases will include the other popular generative AI techniques: agents & function calling, prompt engineering, fine tuning, and pre-training.
 
 The RAG cookbook is divided into 2 sections:
-1. [**Learn:**](#learn) Understand the required components of a production-ready, high-quality RAG application
-2. [**Implement:**](#implement) Use our sample code to follow an evaluation-driven workflow for delivering a high-quality RAG application
+1. **Learn:** Understand the required components of a production-ready, high-quality RAG application
+2. **Implement:** Use our sample code to follow an evaluation-driven workflow for delivering a high-quality RAG application
 
 ## Code-based quick starts
 

genai_cookbook/nbs/2-fundamentals-unstructured-data-pipeline.md

@@ -13,7 +13,7 @@ During data preparation, the RAG application's data pipeline takes raw unstructu
 
 In the remainder of this section, we describe the process of preparing unstructured data for retrieval using *semantic search*. Semantic search understands the contextual meaning and intent of a user query to provide more relevant search results.
 
-Semantic search is one of several approaches that can be taken when implementing the retrieval component of a RAG application over unstructured data. We cover alternate retrieval strategies in the [retrieval knobs section](/nbs/3-deep-dive).
+Semantic search is one of several approaches that can be taken when implementing the retrieval component of a RAG application over unstructured data. We cover alternate retrieval strategies in the [retrieval knobs section](/nbs/3-deep-dive-chain.md#retrieval).
 
 
 

genai_cookbook/nbs/5-hands-on-build-poc.md

@@ -11,6 +11,11 @@
 1. Completed [start here](./6-implement-overview.md) steps
 2. Data from your [requirements](/nbs/5-hands-on-requirements.md#requirements-questions) is available in your [Lakehouse](https://www.databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html) inside a Unity Catalog [volume](https://docs.databricks.com/en/connect/unity-catalog/volumes.html) <!-- or [Delta Table](https://docs.databricks.com/en/delta/index.html)-->
 
+```{admonition} [Code Repository](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code)
+:class: tip
+You can find all of the sample code referenced throughout this section [here](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code).
+```
+
 **Expected outcome**
 
 At the end of this step, you will have deployed the [Agent Evaluation Review App](https://docs.databricks.com/generative-ai/agent-evaluation/human-evaluation.html) which allows your stakeholders to test and provide feedback on your POC. Detailed logs from your stakeholder's usage and their feedback will flow to Delta Tables in your Lakehouse.
@@ -57,32 +62,36 @@ By default, the POC uses the open source models available on [Mosaic AI Foundati
 
 
 
-1. **Open the POC code folder within `01_POC_app` based on your type of data:**
+1. **Open the POC code folder within [`A_POC_app`](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code/A_POC_app) based on your type of data:**
 
    <br/>
 
    | File type                  | Source                 | POC application folder |
    |----------------------------|------------------------|------------------------|
-   | PDF files                  | UC Volume              |   `pdf_uc_volume`                     |
-   | Powerpoint files           | UC Volume              |        `pptx_uc_volume`                |
-   | DOCX files                 | UC Volume              |        `docx_uc_volume`                |
-   | JSON files w/ text/markdown/HTML content & metadata | UC Volume  |              `json_uc_volume`          |  
+   | PDF files                  | UC Volume              |   [`pdf_uc_volume`](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code/A_POC_app/pdf_uc_volume)                     |
+   | Powerpoint files           | UC Volume              |        [`pptx_uc_volume`](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code/A_POC_app/pptx_uc_volume)                |
+   | DOCX files                 | UC Volume              |        [`docx_uc_volume`](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code/A_POC_app/docx_uc_volume)                |
+   | JSON files w/ text/markdown/HTML content & metadata | UC Volume  |              [`json_uc_volume`](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code/A_POC_app/html_uc_volume)          |  
    <!--| HTML content               | Delta Table            |                        |
    | Markdown or regular text   | Delta Table            |                        | -->
 
-   If your data doesn't meet one of the above requirements, you can customize the parsing function (`parser_udf`) within `02_poc_data_pipeline` to work with your file types.
+   If your data doesn't meet one of the above requirements, you can customize the parsing function (`parser_udf`) within `02_poc_data_pipeline` in the above POC directories to work with your file types.
 
    Inside the POC folder, you will see the following notebooks:
 
 ```{image} ../images/5-hands-on/6_img.png
 :align: center
 ```
 
+```{tip}
+The notebooks referenced below are relative to the specific POC you've chosen. For example, if you see a reference to `00_config` and you've chosen `pdf_uc_volume`, you'll find the relevant `00_config` notebook at [`A_POC_app/pdf_uc_volume/00_config`](https://github.com/databricks/genai-cookbook/blob/main/rag_app_sample_code/A_POC_app/pdf_uc_volume/00_config.py).
+```
+
 <br/>
 
 2. **Optionally, review the default parameters**
 
-   Open the `00_config` Notebook to view the POC's applications default parameters for the data pipeline and RAG chain.
+   Open the `00_config` Notebook within the POC directory you chose above to view the POC's applications default parameters for the data pipeline and RAG chain.
 
 
    ```{note}
@@ -91,11 +100,11 @@ By default, the POC uses the open source models available on [Mosaic AI Foundati
 
 3. **Validate the configuration**
 
-   Run the `00_validate_config` to check that your configuration is valid and all resources are available. You will see an `rag_chain_config.yaml` file appear in your directory - we will use this in step 4 to deploy the application.
+   Run the `01_validate_config` to check that your configuration is valid and all resources are available. You will see an `rag_chain_config.yaml` file appear in your directory - we will use this in step 4 to deploy the application.
 
 4. **Run the data pipeline**
 
-   The POC data pipeline is a Databricks Notebook based on Apache Spark.  Open the `02_poc_data_pipeline` Notebook and press Run All to execute the pipeline. The pipeline will:
+   The POC data pipeline is a Databricks Notebook based on Apache Spark. Open the `02_poc_data_pipeline` Notebook and press Run All to execute the pipeline. The pipeline will:
 
    1. Load the raw documents from the UC Volume
    2. Parse each document, saving the results to a Delta Table
@@ -129,7 +138,7 @@ By default, the POC uses the open source models available on [Mosaic AI Foundati
 
    The default POC chain is a multi-turn conversation RAG chain built using LangChain.
 
-   ```{note}
+   ```{tip}
    The POC Chain uses MLflow code-based logging. To understand more about code-based logging, visit the [docs](https://docs.databricks.com/generative-ai/create-log-agent.html#code-based-vs-serialization-based-logging).
    ```
 

genai_cookbook/nbs/5-hands-on-curate-eval-set.md

@@ -10,6 +10,11 @@
 
 *Time varies based on the quality of the responses provided by your stakeholders.  If the responses are messy or contain lots of irrelevant queries, you will need to spend more time filtering and cleaning the data.*
 
+```{admonition} [Code Repository](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code)
+:class: tip
+You can find all of the sample code referenced throughout this section [here](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code).
+```
+
 #### **Overview & expected outcome**
 
 This step will bootstrap an evaluation set with the feedback that stakeholders have provided by using the Review App.  Note that you can bootstrap an evaluation set with *just* questions, so even if your stakeholders only chatted with the app vs. providing feedback, you can follow this step.
@@ -44,6 +49,6 @@ Databricks recommends that your Evaluation Set contain at least 30 questions to
 
 2. Inspect the Evaluation Set to understand the data that is included. You need to validate that your Evaluation Set contains a representative and challenging set of questions. Adjust the Evaluation Set as required.
 
-3. By default, your evaluation set is saved to the Delta Table configured in `EVALUATION_SET_FQN` in the `00_global_config` Notebook.
+3. By default, your evaluation set is saved to the Delta Table configured in `EVALUATION_SET_FQN` in the [`00_global_config`](https://github.com/databricks/genai-cookbook/blob/main/rag_app_sample_code/00_global_config.py) Notebook.
 
 > **Next step:** Now that you have an evaluation set, use it to [evaluate the POC app's](./5-hands-on-evaluate-poc.md) quality/cost/latency.

genai_cookbook/nbs/5-hands-on-evaluate-poc.md

@@ -10,6 +10,11 @@
 
 *Time varies based on the number of questions in your evaluation set.  For 100 questions, evaluation will take approximately 5 minutes.*
 
+```{admonition} [Code Repository](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code)
+:class: tip
+You can find all of the sample code referenced throughout this section [here](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code).
+```
+
 ### **Overview & expected outcome**
 
 This step will use the evaluation set you just curated to evaluate your POC app and establish the baseline quality/cost/latency.  The evaluation results are used by the next step to root cause any quality issues.
@@ -25,7 +30,7 @@ The aggregated metrics and evaluation of each question in the evaluation set are
 
 ### **Instructions**
 
-1. Open the `05_evaluate_poc_quality` Notebook and press Run All.
+1. Open the `05_evaluate_poc_quality` Notebook within your chosen POC directory and press Run All.
 
 2. Inspect the results of evaluation in the Notebook or using MLflow.
 

genai_cookbook/nbs/5-hands-on-improve-quality-step-1-generation.md

@@ -10,7 +10,7 @@ The following is a step-by-step process to address **generation quality** issues
 
 
 
-1. Open the `quality_iteration/01_root_cause_quality_issues` Notebook
+1. Open the [`B_quality_iteration/01_root_cause_quality_issues`](https://github.com/databricks/genai-cookbook/blob/main/rag_app_sample_code/B_quality_iteration/01_root_cause_quality_issues.py) Notebook
 
 2. Use the queries to load MLflow traces of the records that retrieval quality issues.
 

genai_cookbook/nbs/5-hands-on-improve-quality-step-1-retrieval.md

@@ -11,7 +11,7 @@ Retrieval quality is arguably the most important component of a RAG application.
 
 Here's a step-by-step process to address **retrieval quality** issues:
 
-1. Open the `quality_iteration/01_root_cause_quality_issues` Notebook
+1. Open the [`B_quality_iteration/01_root_cause_quality_issues`](https://github.com/databricks/genai-cookbook/blob/main/rag_app_sample_code/B_quality_iteration/01_root_cause_quality_issues.py) Notebook
 
 2. Use the queries to load MLflow traces of the records that retrieval quality issues.
 

genai_cookbook/nbs/5-hands-on-improve-quality-step-1.md

@@ -13,6 +13,11 @@
     - If you followed the previous step, this will be the case!
 2. All requirements from previous steps
 
+```{admonition} [Code Repository](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code)
+:class: tip
+You can find all of the sample code referenced throughout this section [here](https://github.com/databricks/genai-cookbook/tree/main/rag_app_sample_code).
+```
+
 #### **Overview**
 
 Retrieval and generation are the 2 primary buckets of root causes.  To determine where we focus on first, we  use the output of the Mosaic AI Agent Evaluation's LLM judges that you ran in the previous [step](./5-hands-on-evaluate-poc.md) to identify the most frequent root cause that impacts your app's quality.  W
@@ -27,7 +32,7 @@ Each row your evaluation set will be tagged as follows:
 
 The approach depends on if your evaluation set contains the ground-truth responses to your questions - stored in `expected_response`.  If you have `expected_response` available, use the first table below.  Otherwise, use the second table.
 
-1. Open the `B_quality_iteration/01_root_cause_quality_issues` Notebook
+1. Open the [`B_quality_iteration/01_root_cause_quality_issues`](https://github.com/databricks/genai-cookbook/blob/main/rag_app_sample_code/B_quality_iteration/01_root_cause_quality_issues.py) Notebook
 2. Run the cells that are relevant to your use case e.g., if you do or don't have `expected_response`
 3. Review the output tables to determine the most frequent root cause in your application
 4. For each root cause, follow the steps below to further debug and identify potential fixes: