Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run a judge with the current inference pipeline, both with rubric as system prompt and with sample-wise rubric #860

Open
aittalam opened this issue Feb 13, 2025 · 0 comments
Open

Run a judge with the current inference pipeline, both with rubric as system prompt and with sample-wise rubric #860

aittalam opened this issue Feb 13, 2025 · 0 comments
Assignees
@aittalam
Labels
api Changes which impact API/presentation layer backend

Comments

@aittalam
Copy link
Member

aittalam commented Feb 13, 2025 •
edited
Loading

See rationale here.

The main idea for this task is:

  • take a sample dataset and annotate it with rubrics to be passed to a LLM judge
  • run our own inference job by passing the previous dataset as input
  • vibe-check results
@aittalam aittalam self-assigned this Feb 13, 2025
@ividal ividal added backend api Changes which impact API/presentation layer labels Feb 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aittalam aittalam

Labels
api Changes which impact API/presentation layer backend
Projects
Lumigator Public Roadmap H1 2025
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ividal @aittalam