A dataset is a collection of test cases. Each item has an input (the variables you feed a prompt) and an optional expected output (ground truth used by scorers). You reuse the same dataset while prompt wording and models change.

Where To Open It

In the app, go to Evaluate → Datasets. From a dataset, use Run In Playground to expand Run On A Dataset on Prompt Playground and save an experiment.

Create A Dataset

Click New Dataset and give it a name. A short ID appears under the name in the list so duplicate names stay distinguishable.
Optionally add a short description.
Optionally turn on Input Schema or Expected Output Schema (JSON Schema) to lock the shape of new items.

Edit Name And Description

After create, open Edit Dataset to rename, change the description, or adjust schemas. Renaming updates the label everywhere (experiments keep the same dataset id and show the current name). Each experiment still stores the inputs and outputs recorded when that run finished. The dataset header shows when it was last updated.

Schema Toggles

Schemas are off by default. When you enable one, every new item (and CSV import) must match. If you turn a schema on for a dataset that already has items, Traccia checks existing rows first and blocks the change if any do not match.

Add Items

Upload CSV with an input column (JSON object). Optional columns: expected_output, metadata.
Add Manually for a single row (JSON input and optional expected output).
Select From Traces — pick a recent production trace, multi-select spans that have prompt or completion content, optionally redact PII, and add them as items.
From a span in Traces, use Add To Dataset to send that span into an existing dataset (same extraction and optional redact).
Edit any row from the Items table (pencil) to adjust input or expected output after import.

What Gets Extracted

Traccia builds an input object from span prompt attributes (messages and a convenient input field from the last user message when present). If the span has a model completion, that becomes the optional expected output. You can edit items later or leave expected blank and score with LLM-as-judge or custom code.

From Traces

Turning real traffic into eval cases closes the loop between Observe and Evaluate. Use Select From Traces on a dataset (empty state or Items toolbar), or Add To Dataset on a span detail panel. Optional Redact PII before save applies best-effort email, phone, and ID patterns before the row is stored. Review the item afterward.

Source Column

On the Items table, Source shows where a row came from:

Manual / CSV — you typed the row or uploaded a file.
A span ID — the row was added from a production span (Select From Traces or Add To Dataset).

Expected Output And Scorers

Expected output lives on the dataset item, not in a separate grader object. When you run on a dataset in Prompt Playground you choose scorers:

Contains Expected Text — pass if the expected string appears in the model reply (best default for chat).
Exact Match To Expected — pass only if the reply equals the expected output.
Output Is Valid JSON — pass if the reply is a JSON object or array (does not use expected output).
LLM As Judge and Custom Code — create these in Scorers, then attach them in Run On A Dataset.

Run Against A Prompt

Open Prompt Playground, expand Run On A Dataset, pick the dataset and scorers, then Run And Save Experiment. Results save as an experiment you can attach when promoting.

Next Steps

Scorers

Built-in checks, LLM-as-judge, and custom code.

Experiments

Saved graded runs used as promote evidence.

Prompt Playground

Compare panels and run on a dataset.