Annotation Queue

Annotation Queues organize and manage manual evaluation workflows for traces and observations. They allow users to assign reviewers, apply predefined score configurations, and track annotation progress.

Create an Annotation Queue

Use New Annotation Queue to create a queue for reviewing and annotating traces or observations.

Fields

Field	Description
Name	Name of the annotation queue. Example: Chatbot Evaluation Queue
Description (Optional)	Additional information about the queue. Example: Used to review chatbot responses for quality and accuracy.
Score Configurations	One or more predefined score configurations used during annotation.
Assigned Users	Users responsible for reviewing and annotating items in the queue.

Workflow

1. Create an annotation queue.
2. Select one or more score configurations.
3. Assign reviewers to the queue.
4. Add traces or observations for evaluation.
5. Review, score, and annotate each item.