In this project you will use the knowledge and technical skills you have developed in this module to build a novel dataset!
In this project, you will choose a topic and build a novel dataset-- one that no one else has or has used before. You can obtain data by connecting to an API, scraping from the web, or by transforming and merging several publically available datasets to make a new one. You must organize, save, and provided an overview of both your dataset and the workflow you used to obtain it.
- Learn how to develop an interesting question and find the data to answer it.
- Learn how to obtain data from different sources, including APIs, open source datasets, and/or scrape data from the web.
- Save the data you find in a database.
- Explain how you built your database including your sources, any transformations, groupings, or merges you made, and how your data would help you answer your question.
- You must plan your project. Creating a Kanban or Trello Board is mandatory.
- You CANNOT CODE until you project is planned.
- Create a .gitignore file and include it in your repository.
- Your project must include data from at least 2 different data sources (APIs & web, dataset & APIs, ...)
You are required to turn in the following:
- Link to your repository with the code you used to scrape or connect to an API, as well as any cleaning or transformation you did to put it in the database.
- Access information to your database with a description of each table and how they relate. A schema is a great way to show this.
- Links to the data you are using (sources) and your organization (trello).
- Slides for your presentation (template provided).
Day 1
- Look for an interesting topic and the data you would need to answer it.
- Investigate which data sources are available for that topic.
- Reach some best practices agreements if working as a team.
Day 2
- Plan your project and organization. Try to think ahead to some problems you might encounter.
- Start thinking about your data workflow (how does it get from raw data to being in your database?) as well as the structure you want for your final dataset.
Day 3
- Be ready for the database validation with the Lead Teacher / TAs.
- Start working on the descriptions of your dataset and process. Start creating the schema detailing your workflow and database structure.
Day 4
- Start working on your presentation.
Day 5
- Finish your dataset description and practice your presentation.
Day 6
- Presentation!
You will have 5 minutes to present your project. The below are some ideas for slides you could include in your presentation; those marked with an (M) are mandatory!
-
(M) Title of the project
-
(M) Your topic. Why did you choose it?
-
(M) Presentation of the team (if working in a team)
-
Main challenges & strengths
-
(M) Data sources. Where did the raw data for your final data base come from? How did you get it?
-
(M) Workflow. After obtaining your original datasources, how did you transform the data you had and match it with other data to create your database? What is the format of your final dataset?
-
(M) Main insights: What variables do you have in your final dataset and why? How will those variables help you to answer your question? What does your final dataset look like (structure, size, data type, missing values, descriptive statistics)?
-
Going Further: Were there sources you could not access? What other data would have helped you answer your question better? Were there questions you could not answer?
-
Something funny that happened during the project.
-
Things you learned during this project.
-
If you could start from scratch, what would you do differently?
AnyAPI
Top 50 Most Popular APIs on RapidAPI
18 Fun APIs For Your Next Project
WeatherBit
Strava
GitHub
Twitter
LastFM
Spotify
NYTimes
News
Reddit
Medium
Twitch
IGDB
OMDB
GIPHY
StackExchange
YouTube
TheSportsDB
NBA API