Incorporate reranking for improved dataset retrieval and automatic dataset transformation for reduced manual effort

What's Changed

The main changes are that:

1. Dataset retrieval has been improved significantly, by using an LLM as a reranker! This should hopefully give better retrieved datasets with less manual configuration required (e.g. no need to manually specify which of the top-N datasets and configs to choose from)
2. Automatic dataset transformation is here, which means that you can get more out of your retrieved dataset than was previously possible

• Dataset Reranking by @ritugala in #383
• Add data transformation capability to dataset retrieval step by @saum7800 in #385
• Improving Transform and Rerank Module by @ritugala in #396
• Migrate to the latest openAI version by @saum7800 in #397
• Add mistral fine-tuning and examples by @saum7800 in #395

New Contributors

• @7flash made their first contribution in #392

Full Changelog: v0.0.9...v0.1.0

v0.0.9

What's Changed (selected changes)

• Specify library versions for openai and litellm by @viswavi in #378
• Enable hyperparameter optimization in Prompt2Model in #315 and #374
• Configure dataset size in demo script and notebook by @viswavi in #358
• Automatic Column Selection by @ritugala in #359
• Column Selection: Improvements by @ritugala in #366
• Loosen version requirement for protobuf by @saum7800 in #372

Full Changelog: v0.0.8...v0.0.9

v0.0.8

What's Changed

• Dataset size on CLI by @hitenvidhani in #345
• Add buffer in the maximum number of tokens generated (to fix #353) by @viswavi in #354

Full Changelog: v0.0.7...v0.0.8

v0.0.7

What's Changed

• Support HuggingFace's inference API by @viswavi in #352

Full Changelog: v0.0.6...v0.0.7

v0.0.6

Fix bugs in API tools (to fix #348) (#349)

* Use litellm.utils.get_max_tokens appropriatel

* Set max_tokens appropriately

* Use 16k model for HyDE

* Use updated max_tokens logic in async call

* Reduce context size needed for HyDE

v0.0.5

Make API-based model configurable (#344)

* Make models configurable

* Additional description on colab

* Revert unrelated change

* Remove unused argument

* Remove output from notebook

* Remove noqa

* Fix test

* Update prompt2model/utils/api_tools.py

Co-authored-by: Saumya Gandhi <[email protected]>

* Update prompt2model/utils/api_tools.py

Co-authored-by: Saumya Gandhi <[email protected]>

* Change test comments

* Default to maximum tokens for model

---------

Co-authored-by: Saumya Gandhi <[email protected]>

v0.0.4

Improvements, mostly aimed at getting out the colab demo.

What's Changed

• change end of contribution by @zhaochenyang20 in #283
• Fix prompt_examples.md typo by @Farhaduneci in #284
• Refactor dataset retriever to isolate CLI functions by @neubig in #294
• Wording fixes in prompt_examples.md by @neubig in #303
• Relax dependencies for the library by @neubig in #304
• Make a directory for the dataset retriever by @neubig in #308
• Add colab demo by @zhaochenyang20 in #290

New Contributors

• @Farhaduneci made their first contribution in #284

Full Changelog: v0.0.3...v0.0.4

v0.0.3

Fix handling of numerical labels in certain retrieved datasets

v0.0.2

There was a problem with the release workflow when we cut the v0.0.1 release, so v0.0.2 fixes this and makes a proper release.

v0.0.1

This is the initial release of prompt2model. It implements the entirety of the pipeline as described in our tech report.