Skip to content

Latest commit

 

History

History
90 lines (60 loc) · 4.18 KB

02c_TESTING.md

File metadata and controls

90 lines (60 loc) · 4.18 KB

Testing

Overview

This test plan aims to ensure the robustness, reliability, and compliance of processes for downloading, processing, and ingesting the Hugging Face dataset into SQLite3. By implementing Test-Driven Development (TDD), we create a comprehensive suite of granular tests covering all critical functionalities of the data pipeline. These tests are organized into structured directories to enhance maintainability.

Objectives

  • Validate dataset download: Ensure successful download from Hugging Face.
  • Verify PDF text extraction: Confirm accurate text extraction from PDF files.
  • Test dataset processing: Ensure correct processing into a DataFrame.
  • Validate data ingestion: Confirm correct insertion into the SQLite3 database.
  • Ensure data cleaning accuracy: Validate operations such as removing duplicates and handling missing values.
  • Confirm data transformation: Validate operations like merging datasets, enriching data, and deriving new features.
  • Verify data validation: Ensure schema validation and data integrity checks.
  • Quality assurance: Verify the final dataset meets quality standards and performance metrics, incorporating user feedback.

Directory Structure

The tests are organized into the following directory structure:

tests/
├── dataset/              # Tests related to dataset operations
│   ├── test_download.py  # Tests focused on downloading the dataset
│   ├── test_extract.py   # Tests focused on extracting text from PDFs
│   ├── test_process.py   # Tests focused on processing the dataset
│   └── test_ingest.py    # Tests focused on ingesting the dataset into SQLite3
├── clean/                # Tests related to data cleaning
│   ├── test_remove_duplicates.py  # Tests focused on removing duplicates
│   ├── test_handle_missing.py     # Tests focused on handling missing values
├── transform/            # Tests related to data transformation
│   ├── test_merge_datasets.py  # Tests focused on merging datasets
│   ├── test_enrich_data.py     # Tests focused on enriching data
│   ├── test_derive_features.py # Tests focused on deriving new features
├── validate/             # Tests related to data validation
│   ├── test_schema_validation.py  # Tests focused on schema validation
│   ├── test_data_integrity.py     # Tests focused on data integrity
└── qa/                   # Quality assurance tests
    ├── test_quality_standards.py   # Tests focused on meeting quality standards
    ├── test_performance_metrics.py # Tests focused on performance metrics

Test Cases

1. Dataset Download

Objective: Verify successful download and integrity of the dataset from Hugging Face.

2. PDF Text Extraction

Objective: Ensure accurate text extraction from PDF files.

3. Dataset Processing

Objective: Confirm the dataset is processed correctly into a DataFrame.

4. Data Ingestion into SQLite3

Objective: Validate that processed data is correctly inserted into the SQLite3 database.

5. Data Cleaning

Objective: Ensure accuracy of data cleaning operations like removing duplicates and handling missing values.

6. Data Transformation

Objective: Confirm that data transformation operations like merging datasets, enriching data, and deriving new features are performed correctly.

7. Data Validation

Objective: Validate the schema and integrity of the data, ensuring consistency across subsets.

8. Quality Assurance

Objective: Verify the final dataset meets quality standards and performance metrics, incorporating user feedback.

Running Tests

To run the tests, navigate to the root of your project directory and use the following command:

pytest tests/

This command runs all the tests in the specified directory, ensuring that each component of the data pipeline functions as expected.

Conclusion

By implementing granular and focused tests organized in a clear directory structure, we ensure that our data pipeline is thoroughly validated at each stage. This approach helps maintain code quality, prevent bugs, and facilitate future enhancements.