Big Data Project – UniTn, 2021/22

About The Project

The goal of the project is to implement a big data system to support a real estate investor who is considering buying property for short-term rentals in Europe, and direct him or her to choose the area where he or she can buy.

Built With

The technologies used in the implementation of the project are listed below.

• Instant Data Scraper
• Web Scraper
• Python
• Docker
• Redis
• FastAPI
• Plotly Dash

Getting Started

This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.

Prerequisites

In order to start the system, you have to first install the dependencies, which are listed below. Then, start the various scripts in the order to follow.

You should always pay attention to the directory inside which you are executing terminal commands.

All dependencies needed to start the project are listed in the requirements.txt file, so simply run the terminal command

pip install requirements.txt 

Diagram

(back to top)

Starting the system

First you have to start the Docker yml in order to have a working Redis instance.

cd redis
docker compose up

APIs have been made through FastAPI to make Redis database endpoints available in localhost. Therefore, one would need to start the Uvicorn server to make the endpoints work.

uvicorn main:app --reload

At this point all the elements needed to have the system running are running.

(back to top)

Usage

For the purposes of this project, the data were collected manually by web scraping and are already collected inside the scraped.json file. However, it is possible to implement a data stream processing system, since the Data Storage and Data Preparation scripts are already set up for this.

(back to top)

Importing data into the database

Make sure you are in the 'redis' folder. Then run the command:

python3 import_to_redis.py

This script imports all the Real Estates records obtained in the scraped.json file into the Redis database.

(back to top)

Getting the data from the database

Return to root. Then run the command to start the getData script, which performs a retrieve from the database and merges all the JSON records into a CSV file.

cd ..
python3 getData.py

(back to top)

Data Preparation and Data Cleaning

Two separate scripts perform data preparation and cleaning.

python3 dataPreparation.py
python3 dataCleaning.py

(back to top)

Data Visualization

The final script performs the generation of the various plots aimed at directing the operator in the choice of purchase.

python3 dataVisualisation.py

(back to top)

Front-End

It is possible to start a web application that provides the Front-End side to our system via Plotly Dash. Just start the .py script to run the server, and consult the Front-End locally:

python3 webApp.py

or run it from your favorite IDE.

Roadmap

• Scraping
• Data Collection
• Implementing Redis DB
• Dockerize DB
• Implementing FastAPI and various endpoints
• Implementing Redis UI
• Realizing importing to Redis script
• Script to retrieve data from Redis and converting JSON to CSV
• Implementing data cleaning and preparation scripts
• Implementing data visualization script
• Diagram
• Web App

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

Contact

Luca Maccacaro - [email protected]

Simone Bellavia – [email protected]

Project Link: https://github.com/lucamac99/big-data-project

(back to top)