-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Meta knn few shot #1894
Draft
ammirsm
wants to merge
8
commits into
main
Choose a base branch
from
meta-knn-few-shot
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Meta knn few shot #1894
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Import specific task classes conditionally based on the dataset name. - Update load_dataset method to return an instance of the selected task. - Introduce logging directory for optimization tests. - Organize imports by adding MetaKNNFewShot module. - Exclude unnecessary files from version control.
- Add `MetaKNNFewShot` class to implement a teleprompter using meta-learning with KNN. - Include methods for program compilation, generation, performance caching, and best program selection. - Introduce a testing suite to validate functionality, including metrics, cache management, and program selection based on similar examples. - Create environment setup for API key management and integrate with `OptimizerTester`. - Ensure proper parallel execution of performance evaluation and handle NaN values during program selection.
interesting what is the use case here? |
- Add .index and virtual environment directories to .gitignore - Simplify import statements by removing unnecessary imports - Adjust MetaKNNFewShot initialization parameters for clarity - Update main function to include additional parameters for the OptimizerTester
@chiragshah285 |
- Add `max_bootstrap_demos` parameter to `MetaKNNFewShot` class. - Update relevant dictionary to include `max_bootstrapped_demos
…zer support - Add vectorizer parameter to KNN constructor to compute embeddings. - Modify MetaKNNFewShot to accept vectorizer in its initialization. - Update setup functions and tests to include vectorizer configuration. - Clean up unnecessary comments and streamline import statements. - Include additional parameters such as max_bootstrapped_demos.
…s. Remove unused imports in KNN class.
- Include `venv-cyrus/` and `.venv/` in .gitignore to prevent tracking of virtual environments. - Initialize `dspy.Embedder` with OpenAI model and API key in `meta_knn_optimizer_c
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new
MetaKNNFewShot
class for thedspy.teleprompt
module and adds comprehensive testing for this new functionality. The most important changes include the addition of theMetaKNNFewShot
class, updates to the__init__.py
file to include this new class, and the creation of new test files to ensure the functionality works as expected.New functionality:
dspy/teleprompt/meta_knn_fewshot.py
: Introduced theMetaKNNFewShot
class, which implements meta-learning with KNN for few-shot prompt selection. This class includes methods for compiling programs, generating multiple programs using bootstrap, building a performance cache, and selecting the best program based on performance.Updates to existing files:
dspy/teleprompt/__init__.py
: Added an import statement for the newMetaKNNFewShot
class.Testing:
testing/meta_knn_tester.py
: Added a new tester script to test theMetaKNNFewShot
optimizer. This script includes loading environment variables, initializing the teleprompter, and running tests on specified datasets.tests/teleprompt/test_meta_knn_fewshot.py
: Added a comprehensive test suite for theMetaKNNFewShot
class. This includes tests for initialization, compilation, performance caching, similar example influence, demo exclusion in performance calculations, and best program selection with NaN values.Refactoring:
testing/optimizer_tester.py
: Refactored theload_dataset
method to import task modules dynamically based on the dataset name, and made minor formatting changes for consistency. [1] [2] [3] [4] [5]