The-Devfolio-Project

Largest ever data set of Devfolio hackathons, sponsors, projects and much more. Scraped from 60K+ Devfolio pages and hackathons.

Data set date: 21 July 2024

The data set is huge. The Raw Datasets folder has two files: one with all external links and another with Internal links.

For specific use cases, this needs filtering and cleaning.

Use Cases:

1. Finding sponsors
2. Building AI tools to find sponsors
3. Exploring Projects being built on Devfolio
4. Analysing patterns in Projects
5. Tech trends among participants
6. Whatever you can think of......

About Datasets:

1. raw/ External Links => Contains all Links that go outside of Devolio, includes projects, sponsors etc..
2. raw/ Internal Links => Contains all Devfolio internal links such as devfolio hackathons etc..
3. raw/ Site Structure => Has a lot of data specific to Devfolio structure along with user profiles and description. What would you do with it? IDK!

TODO:

• Create a dataset of Sponsors on Devfolio
  • Filter all outlinks
  • Remove all social media links
  • Remove all web hosting services link
  • Remove all readme file links
  • Remove all Github Links
  • Remove duplicates
  • Finally, run all the domains against the "whois" database and purge any domain that is not older than 8 months or does not exist.
  • We have a clean database 🎉
• Create a dataset of all Hackathons on Devfolio
• Create a dataset of Sponsor frequency on Devfolio (Top Sponsors etc..)
• Create a dataset of all Projects on Devfolio
• Use Google Data Studio and create nice Graphs
• Create dataset of Sponsors along with their emails and point of contact.

Contribution:

I am accepting contributions for this project and need help with cleaning and filtering data.

To contribute:

1. Form this project
2. Pick one of the "Todo" from above
3. Check if a PR for your "Todo" already exists, either work with them or choose another "Todo"
4. Create a PR
5. I will review and merge

License

The raw data, any dataset and all contributions to this project are released under GNU GPLv3. By contributing to or using this project you accept the GNU GPLv3 license.

Credits

If you use this dataset for any project, please mention this GitHub repo or me.

Happy Building