Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assertion-aware metrics and optimization #274

Merged
merged 17 commits into from
Feb 2, 2024
Merged

Assertion-aware metrics and optimization #274

merged 17 commits into from
Feb 2, 2024

Conversation

manishshettym
Copy link
Collaborator

@manishshettym manishshettym commented Jan 15, 2024

Tasks

  • option to enable assert-aware-metric for each assert/suggest (i.e., optimizable):
    dspy.Suggest(constraint, msg, is_metric)
  • track suggest/assert failures during evaluation
  • propagate suggest/assert failures as an attribute of a bootstrapped program
  • penalize optimization score according to the number of failures
  • incorporate feedback traces into compilation

@manishshettym manishshettym assigned Shangyint and unassigned Shangyint Jan 15, 2024
@okhat
Copy link
Collaborator

okhat commented Feb 2, 2024

Thanks so much. Should we merge this? Does this have the latest from @arnavsinghvi11 ?

@manishshettym manishshettym marked this pull request as ready for review February 2, 2024 18:23
@okhat okhat merged commit 0bba7b9 into main Feb 2, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants