Examples of architecture using AWS ETL and DB services:
- Data Pipelines (S3 to S3 and S3 to RDS)
- Data Pipeline (S3 to Redshift)
- S3 to DynamoDB
- Glue Example 1
- Glue Example 2
- Glue Example 3
- Kinesis (Streams and Firehose)
- Step-functions
- EventBridge
All source code can be found in this repository and the scripts for the various examples are stored in the respective named folders.
For running some of the scripts locally, first install poetry
pip install poetry
and then install the dependencies from the poetry.lock file Ref
poetry install
Installing dependencies from lock file
The command below will spawn a new shell with a virtual env containing the newly installed dependencies
$ poetry shell
Spawning shell within /Users/rk1103/Library/Caches/pypoetry/virtualenvs/aws-etl-fV9WWBi4-py3.9
(base) rk1103@Ryans-MacBook-Air aws_etl % . /Users/rk1103/Library/Caches/pypoetry/virtualenvs/aws-etl-fV9WWBi4-py3.9/bin/activate
(aws-etl-fV9WWBi4-py3.9) (base) rk1103@Ryans-MacBook-Air aws_etl %
Most of the resources are created via cloudformation templates. The process for doing this is described here