Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Meta knn few shot #1894

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

Meta knn few shot #1894

wants to merge 8 commits into from

Conversation

ammirsm
Copy link
Collaborator

@ammirsm ammirsm commented Dec 6, 2024

This pull request introduces a new MetaKNNFewShot class for the dspy.teleprompt module and adds comprehensive testing for this new functionality. The most important changes include the addition of the MetaKNNFewShot class, updates to the __init__.py file to include this new class, and the creation of new test files to ensure the functionality works as expected.

New functionality:

  • dspy/teleprompt/meta_knn_fewshot.py: Introduced the MetaKNNFewShot class, which implements meta-learning with KNN for few-shot prompt selection. This class includes methods for compiling programs, generating multiple programs using bootstrap, building a performance cache, and selecting the best program based on performance.

Updates to existing files:

Testing:

  • testing/meta_knn_tester.py: Added a new tester script to test the MetaKNNFewShot optimizer. This script includes loading environment variables, initializing the teleprompter, and running tests on specified datasets.
  • tests/teleprompt/test_meta_knn_fewshot.py: Added a comprehensive test suite for the MetaKNNFewShot class. This includes tests for initialization, compilation, performance caching, similar example influence, demo exclusion in performance calculations, and best program selection with NaN values.

Refactoring:

- Import specific task classes conditionally based on the dataset name.
- Update load_dataset method to return an instance of the selected task.
- Introduce logging directory for optimization tests.
- Organize imports by adding MetaKNNFewShot module.
- Exclude unnecessary files from version control.
- Add `MetaKNNFewShot` class to implement a teleprompter using meta-learning with KNN.
- Include methods for program compilation, generation, performance caching, and best program selection.
- Introduce a testing suite to validate functionality, including metrics, cache management, and program selection based on similar examples.
- Create environment setup for API key management and integrate with `OptimizerTester`.
- Ensure proper parallel execution of performance evaluation and handle NaN values during program selection.
@chiragshah285
Copy link

interesting what is the use case here?

- Add .index and virtual environment directories to .gitignore
- Simplify import statements by removing unnecessary imports
- Adjust MetaKNNFewShot initialization parameters for clarity
- Update main function to include additional parameters for the OptimizerTester
@ammirsm
Copy link
Collaborator Author

ammirsm commented Dec 9, 2024

interesting what is the use case here?

@chiragshah285
The primary objective here is to discover the most effective few shots dynamically, rather than solely relying on vector similarity.

- Add `max_bootstrap_demos` parameter to `MetaKNNFewShot` class.
- Update relevant dictionary to include `max_bootstrapped_demos
…zer support

- Add vectorizer parameter to KNN constructor to compute embeddings.
- Modify MetaKNNFewShot to accept vectorizer in its initialization.
- Update setup functions and tests to include vectorizer configuration.
- Clean up unnecessary comments and streamline import statements.
- Include additional parameters such as max_bootstrapped_demos.
- Include `venv-cyrus/` and `.venv/` in .gitignore to prevent tracking of virtual environments.
- Initialize `dspy.Embedder` with OpenAI model and API key in `meta_knn_optimizer_c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants