data_warehouse_bq_dbt

This is a project of data warehouse built on BigQuery using dbt with serverless.

Data Structure

Local Build FastAPI using HTTP requests to call dbt commands

1. Add service account crediental file in folder /secrets with file naming dbt-admin_credential.json
2. Build Docker image

   docker build -t data_warehouse_bq_dbt:latest .

3. Run Docker Container

   sh ./start.sh

4. (Optional) Use OpenAPI documentations

   To access from your browser, navigate to: http://localhost:5000/docs

Local Build using dbt

1. Add service account crediental file in folder /secrets with file naming dbt-admin_credential.json
2. Run Docker Container

   sh ./dbt_start.sh

3. Run DBT commands in Docker Container

   cd /dbt
   # Example (can refer to dbt_script.sh)
   dbt debug --profiles-dir .

4. (Optional) Generate DBT documentations

   dbt docs generate --profiles-dir .
   dbt docs serve --profiles-dir .

   To access from your browser, navigate to: http://localhost:8080

• Reference:
  • DBT Offical Docs
  • Structure / Coding Style
    • DBT Style Guide
    • How we structure our dbt projects
  • DBT with BigQuery
    • BigQuery Configurations
    • Benchmarking: Incremental Strategies
    • Incremental Changes with Insert Overwrite
    • BigQuery Merge Statement
  • DBT Guidance
    • DBT Best Practices
    • DBT Methods
    • What are the dbt commands you run in your production deployment
    • Understanding State (Advanced)
    • Defer (Advanced)

Local Build FastAPI

1. Run Docker Container

   sh ./app_start.sh

2. Run FastAPI server in Docker Container

   cd /app
   # install python packages
   pip install -r requirements.txt
   # run FastAPI server
   python main.py

3. (Optional) Use OpenAPI documentations

   To access from your browser, navigate to: http://localhost:5000/docs

• Reference:
  • FastAPI Offical Docs

Local Build Cube.js

1. Run Docker Container

   sh ./cube_start.sh

2. (Optional) Use Cube.js playground site to query data

   To access from your browser, navigate to: http://localhost:4000

• Reference:
  • Cube.js Offical Docs