The Waiterbildung is a online course recommending application designed to extract and process course content from universities across the world. This tool is intended for developers and data analysts who need to gather educational information from web pages efficiently.
- Efficient Web Scraping: Extract course data from multiple university websites with ease.
- Customizable: Easily configure scraping rules and targets.
- Data Processing: Built-in tools for cleaning and processing scraped data.
- Logging: Comprehensive logging for monitoring scraping activities.
- Error Handling: Robust error handling to manage unexpected issues.
To install the Waiterbildung, follow these steps:
-
Clone the repository:
git clone https://github.com/sonny-thomas/waiterbildung.git cd poc-scraper -
Install dependencies:
pip install -r requirements.txt
To use the scraper, run the following command:
docker compose up -dEnsure that all environment variables are set in the .env file before starting the application.
The .env file contains all the necessary configurations for the scraper.
You can find the entire API documentation at the following endpoints:
/api/v1/docs/api/v1/redoc
We welcome contributions from the community. To contribute, please follow these steps:
- Fork the repository.
- Create a new branch (
git checkout -b feat/your-feature-name). - Make your changes.
- Commit your changes following conventional commits (
git commit -m 'feat: add new feature'). - Push to the branch (
git push origin feat/your-feature-name). - Create a new Pull Request.
This project is licensed under the MIT License. See the LICENSE file for details.
For any questions or inquiries, please contact us at [email protected].