Annotation Queues¶

An Annotation Queue is the central object in the annotations system. It defines what to review (the items), how to review it (the schema), who reviews it (the assignees), and how many reviews each item needs.

Annotation Schema¶

Before creating a queue, you need to decide what fields reviewers will fill in. These fields are defined as part of the queue's schema.

Field Types¶

Each field has a name and an optional description to guide reviewers.

Example schema:

Creating a Queue¶

Navigate to Annotation Queues in the left sidebar and click New Queue.

Queue Status¶

Changing Reviews Required After Annotation Starts¶

You can change the Reviews required setting on a queue even after annotation is underway. Because this affects how item statuses are calculated, the system recomputes the status of every non-flagged item when you save the change.

Before applying the change, a confirmation dialog summarises what will happen. Review it carefully before confirming.

Effects of raising the requirement¶

Increasing Reviews required can move items that were previously Completed back to In Progress or Awaiting resolution — if they no longer have enough reviews to meet the new threshold.

Effects of lowering the requirement¶

Decreasing Reviews required does not automatically complete items that still have unresolved reviews. Items where all required reviews are in but no authoritative annotation has been selected remain in Awaiting resolution until a reviewer picks one annotation as authoritative.

Switching to multi-reviewer mode¶

When you increase Reviews required from 1 to more than 1, any authoritative flags that were set automatically by the system are cleared. Authoritative flags that a reviewer set manually are kept. Aggregate scores are refreshed after the change.

Adding Items¶

Items can be added to a queue in three ways.

1. Session Selector¶

From the queue detail page, click Add Sessions. This opens a filterable table of experiment sessions. You can then choose how many sessions to add:

Sessions that are already in the queue are excluded from the selector.

2. Import from Evaluations Dataset¶

Click Import from Dataset on the queue detail page to add sessions from an existing evaluations dataset. This is useful when you want to annotate the same set of sessions you used for automated evaluations.

3. From a Session Detail Page¶

On any session detail page, use the Add to Queue button to add that session directly to one or more annotation queues.

Monitoring Progress¶

The queue detail page shows a progress summary:

• Total items in the queue
• Completed items (reached required number of reviews; for multi-reviewer queues, an authoritative annotation has also been selected)
• Flagged items
• Overall review progress as a percentage (reviews done / reviews needed)

For multi-reviewer queues, the summary also surfaces two additional counts:

• Resolved — items that have an authoritative annotation, shown as X / N items resolved
• Awaiting resolution — items where all required reviews are in but no authoritative annotation has been selected yet (see Resolving Multi-Reviewer Conflicts)

Aggregate Scores¶

After annotations are submitted, aggregate scores are automatically computed and displayed on the queue detail page.

Aggregates are recomputed after each annotation submission, so you always see up-to-date stats.

Exporting Results¶

From the queue detail page (requires queue management permissions), you can export all submitted annotations. Both formats pivot the results so that each reviewer's answers appear side by side, making it easy to compare reviewers on the same item.

The export covers every item's submitted annotations. Flagged items are always included, even when they have no annotations, so every flagged item appears regardless of review status.

Each exported item carries these shared fields:

Each reviewer's answers to the schema fields are attached per format:

• CSV adds a field column (the schema field name) and one column per reviewer email. Each row is one item/field combination; the reviewer columns hold that reviewer's value, blank where they did not answer.
• JSONL adds a fields object shaped {field: {reviewer_email: value}}. A field a reviewer did not answer is null, and values keep their JSON types (numbers stay numbers).

Example JSONL record:

{
  "item_id": 67, "item_type": "session", "session_id": "d99c...",
  "flagged": false, "flags": [],
  "authoritative_annotator": "marcus@example.com", "annotated_at": "2026-07-15T13:17:02Z",
  "fields": {
    "tone":     {"fatima@example.com": "friendly", "marcus@example.com": "too casual"},
    "accuracy": {"fatima@example.com": "accurate", "marcus@example.com": "missed grammar"}
  }
}

Managing Assignees¶

Use Manage Assignees on the queue detail page to add or remove reviewers. Assignees receive access to the queue and can see it in their annotation queue list.