It a real time simulation of stock market using. kafka and aws services
Architecture Diagram for the end to end pipe line
Steps:
- Create an Amazon EC2 free tier t2 micro instance and install Kafka on it.
- run the Kafka Zookeper
- Start the Kafka Server
- Create a topic
- Run Producer
- Run Consumer
- Create jupyter notebooks to simulate Stock market data using the CSV file given. Kafka Producer and Kafka Consumer respectively
- Create A S3 bucket. and allow puclic access to the bucket
- Create AWS CLI on local machine to give access to to upload on S3.
- start uploading files into s3
- once enough files are uploaded crawl the the S3 bucket with crawler from AWS Glue.
- Create a database
- Access the database with Athena
Examples :-
1.
2.
3.
