Databricks Labs

All

35 repositories

dqx
Public
Databricks framework to validate Data Quality of pySpark DataFrames
spark spark-streaming databricks data-quality-checks data-quality data-profiling dlt data-quality-monitoring
Python
•
Other
•23•167•28•4•Updated Jan 28, 2025Jan 28, 2025
ucx
Public
Automated migrations to Unity Catalog
databricks unity-catalog databricks-cli-installable databricks-governance
Python
•
Other
•87•256•254•15•Updated Jan 28, 2025Jan 28, 2025
remorph
Public
Accelerates migrations to Databricks by automating code conversion and migration validation
reconciliation data-validation transpiler databricks code-converter
Python
•
Other
•34•50•390•18•Updated Jan 28, 2025Jan 28, 2025
dbldatagen
Public
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines
spark faker pyspark spark-streaming data-generation databricks synthetic-data datagen datagenerator deltalake
Python
•
Other
•67•387•13•13•Updated Jan 27, 2025Jan 27, 2025
partner-connect-api
Public
Scala
•
Other
•4•12•1•11•Updated Jan 24, 2025Jan 24, 2025
tempo
Public
API for manipulating time series on top of Apache Spark: lagged time values, rolling statistics (mean, avg, sum, count, etc), AS OF joins, downsampling, and interpolation
python scala timeseries-data timeseries-analysis data-science timeseries time-series pandas data-analysis
Jupyter Notebook
•
Other
•53•317•31•0•Updated Jan 24, 2025Jan 24, 2025
sandbox
Public
Experimental labs projects
databricks databricks-api databricks-sdk
Go
•
Other
•10•23•26•28•Updated Jan 23, 2025Jan 23, 2025
brickster
Public
R Toolkit for Databricks
R
•
Apache License 2.0
•11•55•4•0•Updated Jan 22, 2025Jan 22, 2025
blueprint
Public
Baseline for Databricks Labs projects written in Python
python cli databricks databricks-cli-installable
Python
•
Other
•10•24•15•3•Updated Jan 14, 2025Jan 14, 2025
dlt-meta
Public
Metadata driven Databricks Delta Live Tables framework for bronze/silver pipelines
Python
•
Other
•74•161•31•1•Updated Jan 14, 2025Jan 14, 2025
discoverx
Public
A Swiss-Army-knife for your Data Intelligence platform administration.
scanning data-retrieval pii-detection semantic-classification multi-table-operations
Python
•
Other
•13•111•17•13•Updated Jan 8, 2025Jan 8, 2025
mosaic
Public
An extension to the Apache Spark framework that allows easy and fast processing of very large geospatial datasets.
Jupyter Notebook
•
Other
•70•286•60•17•Updated Dec 23, 2024Dec 23, 2024
lsql
Public
Lightweight SQL execution wrapper only on top of Databricks SDK
databricks databricks-sql databricks-sdk
Python
•
Other
•4•15•53•8•Updated Nov 22, 2024Nov 22, 2024
overwatch
Public
Capture deep metrics on one or all assets within a Databricks workspace
monitoring databricks
Scala
•
Other
•66•230•131•11•Updated Nov 20, 2024Nov 20, 2024
pytester
Public
Python Testing for Databricks
python3 pytest pytest-plugin databricks databricks-sdk
Python
•
Other
•5•59•35•0•Updated Nov 15, 2024Nov 15, 2024
pylint-plugin
Public
Databricks Plugin for PyLint
python databricks pylint-plugin
Python
•
Other
•4•15•10•1•Updated Nov 15, 2024Nov 15, 2024
tika-ocr
Public
Rich Text Format
•
Other
•3•19•1•5•Updated Oct 21, 2024Oct 21, 2024
dbx
Public
🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.
ci cicd databricks mlops databricks-cli databricks-api
Python
•
Other
•124•445•90•5•Updated Sep 16, 2024Sep 16, 2024
geoscan
Public
Geospatial clustering at massive scale
library spark-ml clustering
Scala
•
Other
•19•97•2•6•Updated Jul 11, 2024Jul 11, 2024
waterbear
Public
Automated provisioning of an industry Lakehouse with enterprise data model
python sql spark data-model databricks delta-lake lakehouse
Python
•
Other
•3•8•0•1•Updated Jul 11, 2024Jul 11, 2024
dataframe-rules-engine
Public
Extensible Rules Engine for custom Dataframe / Dataset validation
Scala
•
Other
•30•134•8•4•Updated May 7, 2024May 7, 2024
splunk-integration
Public
Databricks Add-on for Splunk
cybersecurity databricks databricks-notebooks splunk-addon
Python
•
Other
•17•26•16•3•Updated May 7, 2024May 7, 2024
databricks-sdk-r
Public
Databricks SDK for R (Experimental)
data-science r sdk databricks
R
•
Apache License 2.0
•4•20•6•1•Updated Apr 8, 2024Apr 8, 2024
delta-sharing-java-connector
Public
A Java connector for delta.io/sharing/ that allows you to easily ingest data on any JVM.
java delta-sharing
Java
•
Apache License 2.0
•6•13•2•2•Updated Mar 25, 2024Mar 25, 2024
transpiler
Public
SIEM-to-Spark Transpiler
Scala
•
Other
•5•42•13•5•Updated Mar 18, 2024Mar 18, 2024
doc-qa
Public
Python
•
Other
•7•47•0•4•Updated Feb 23, 2024Feb 23, 2024
databricks-sync
Public
An experimental tool to synchronize source Databricks deployment with a target Databricks deployment.
Python
•
Other
•14•46•16•7•Updated Jan 21, 2024Jan 21, 2024
arcuate
Public
Delta Sharing + MLflow for ML model & experiment exchange (arcuate delta - a fan shaped river delta)
big-data spark data-sharing mlflow delta-sharing
Python
•
Other
•2•22•0•9•Updated Dec 27, 2023Dec 27, 2023
feature-factory
Public
Accelerator to rapidly deploy customized features for your business
Python
•
Other
•26•55•1•4•Updated Dec 10, 2023Dec 10, 2023
delta-oms
Public
DeltaOMS is a solution that help build a centralized repository of Delta Transaction logs and associated operational metrics/statistics for your Delta Lakehouse. Unity Catalog supported in the v0.7.0-rc1 release.Documentation here - https://databrickslabs.github.io/delta-oms/v0.7.0-rc1/
monitoring metrics delta centralized databricks delta-lake lakehouse
Scala
•
Other
•4•39•6•1•Updated Nov 27, 2023Nov 27, 2023