🗺 Annotation Queues

[mikeldking]

Annotation queues provide an effective way to streamline and manage human annotation tasks by allowing groups of traces to be reviewed and annotated systematically. This feature will enhance the efficiency of tracking and analyzing feedback on specific traces in Phoenix.

Goals:

• Implement annotation queues within Phoenix to facilitate dedicated feedback collection.
• Enable users to manage runs more effectively through queue creation, collaboration settings, and automated run assignments.
• Enhance collaboration through configuration options for reviewer roles and permissions.

Technical Requirements:

1. Queue Management

   • Allow creation of annotation queues via the Phoenix interface.
   • Enable managing queue details such as the name, description, and default dataset assignments.

2. Annotation Rubric

   • Incorporate high-level instructions and desired feedback keys for annotators.
   • Implement a feedback mechanism with appropriate feedback categories and descriptions.

3. Collaboration Settings

   • Options for setting the number of mandatory reviews per run.
   • Support for reservation settings to avoid simultaneous reviews by multiple users.
   • Allow adjustments to queue settings through an intuitive UI.

4. Run Assignments

   • Implement methods to add runs to annotation queues from various trace and runs table views.
   • Provide automation for assigning runs to queues based on specific conditions or rules.

5. Review Interface

   • Create a focused interface for reviewing runs in the annotation queues.
   • Allow flexible input, such as comments, scores, or dataset assignments, during the review process.
   • Facilitate easy removal of runs from the queue.

Spike

• [annotation] annotation PRD
• [annotation] annotation configuration and queue ERD

Configuration

• [ENHANCEMENT]Allow for different eval colors in Phoenix #5842
• [annotation] [db] annotation configuration #6207
• [annotation] [db] project level annotation configuration

UI

• [annotation] [ui] add the ability to page through traces (up / down)
• [annotation] [ui] hot keys on annotation details view
• [annotation] [ui] add the ability to associate annotations
• [ENHANCEMENT] more cleanly display metadata in the ui, add quick filters on the row #6125