diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/MedallionArch_Fabric+Databricks.md b/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/MedallionArch_Fabric+Databricks.md index 09364d39..defa8a80 100644 --- a/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/MedallionArch_Fabric+Databricks.md +++ b/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/MedallionArch_Fabric+Databricks.md @@ -14,9 +14,43 @@ Last updated: 2025-02-13
List of References (Click to expand) +- [TechExcel: Microsoft Fabric with Azure Databricks for Data Analytics (lvl 300 / CSU) lab](https://microsoft.github.io/TechExcel-Fabric-with-Databricks-for-Data-Analytics/) +- [Databricks Unity Catalog tables available in Microsoft Fabric](https://blog.fabric.microsoft.com/en-us/blog/databricks-unity-catalog-tables-available-in-microsoft-fabric/) +- [Integrate OneLake with Azure Databricks](https://learn.microsoft.com/en-us/fabric/onelake/onelake-azure-databricks) +- [Tutorial: Configure Microsoft Fabric mirrored databases from Azure Databricks (Preview)](https://learn.microsoft.com/en-us/fabric/database/mirrored-database/azure-databricks-tutorial) +- [Integrating Microsoft Fabric with Azure Databricks Delta Tables](https://techcommunity.microsoft.com/blog/fasttrackforazureblog/integrating-microsoft-fabric-with-azure-databricks-delta-tables/3916332) +- [Data Intelligence End-to-End with Azure Databricks and Microsoft Fabric](https://techcommunity.microsoft.com/blog/azurearchitectureblog/data-intelligence-end-to-end-with-azure-databricks-and-microsoft-fabric/4232621)
+## Overview + +| Resource | Key Components | Details | +|------------|--------------------------|-------------------------------------------------------------------------| +| Databricks | - ETL Tasks
- Data Ingestion | ETL Tasks: Databricks efficiently handles Extract, Transform, Load (ETL) tasks. It manages the bronze, silver, and gold layers of the medallion architecture:
- **Bronze Layer**: Raw data ingestion from various sources.
- **Silver Layer**: Data cleaning and transformation.
- **Gold Layer**: Aggregated and refined data ready for analysis.
Data Ingestion: Supports batch and streaming data ingestion, ensuring real-time data processing capabilities.
- **Data Cleaning**: Utilizes Apache Spark's powerful processing engine to clean and transform data at scale.
- **Aggregation**: Performs complex aggregations and computations, making data ready for downstream analytics. | +| Fabric | - Data Integration
- Orchestration
- Monitoring and Management | Fabric seamlessly integrates with Databricks, providing a unified interface for managing data workflows.
- **Data Integration**: Facilitates the orchestration of data pipelines, ensuring smooth data flow between different stages of processing.
- **Monitoring and Management**: Offers robust monitoring and management tools to track data pipeline performance and troubleshoot issues. | + +> Here is a [reference of a medallion architecture using only Fabric](https://github.com/MicrosoftCloudEssentials-LearningHub/MS-Fabric-Essentials-Workshop/tree/main/AzurePortal/1_MedallionArch): + +

+ image +

+ +> If you need to handle `complex data transformations and large-scale data processing`, you can use our combined solution of **Fabric + Databricks**. This powerful combination leverages the strengths of both platforms to provide a robust data processing pipeline. This workshop on [Fabric with Databricks for Data Analytics](https://microsoft.github.io/TechExcel-Fabric-with-Databricks-for-Data-Analytics/) offers a comprehensive step-by-step guide on developing Medallion Architecture using Fabric and Databricks.
+ +

+ image +

+ +## Integration with Multiple Sources: + +> In these two examples of Medallion Archuitecture, `Azure SQL Database` was used as the input source, but the solution is highly flexible and can integrate with multiple data sources, including: + +- **Cloud Storage**: Azure Blob Storage, AWS S3, Google Cloud Storage. +- **Databases**: SQL Server, MySQL, PostgreSQL, Oracle. +- **Streaming Sources**: Kafka, Event Hubs, IoT Hub. +- **APIs and Web Services**: REST APIs, SOAP services. +

Total Visitors