Experiments

How A Run Works

1. Load one or more prompt panels in Prompt Playground.
2. Expand Run On A Dataset, attach a dataset, and optionally limit to the first N items.
3. Select scorers such as Contains Expected Text, Exact Match To Expected, Output Is Valid JSON, or custom Scorers (LLM-as-judge and code). Manage the library under Evaluate → Scorers.
4. Click Run And Save Experiment. Progress shows item counts in Playground. You can leave the page; live runs also appear under Evaluate → Experiments in the Runs in progress strip. The experiment row is created when the job finishes.
5. Open the experiment to review overall pass rate, pass rate by scorer, cost, mean latency, linked prompt version ids, and expandable per-row outputs.

Runs In Progress

Dataset batch runs are asynchronous. While a job is pending or running, Evaluate → Experiments shows a strip with status, item progress (N / M), and an Open Experiment link when the job completes. Failed runs surface a short error in the same strip.

List And Detail

The experiments table shows the experiment name (with a short ID underneath), the dataset name (also with a short ID), and the scorers used on that run. Dataset names are live labels: if you rename a dataset, older experiments show the new name and the same id. Experiment names are set when you run; they are not renamed later so promote evidence stays stable.

On the detail page, review overall pass rate, pass rate by scorer (including the judge model for LLM-as-judge), cost, mean latency across panel cells, and expandable per-row outputs. Use Compare with… to open a side-by-side view against another run on the same dataset.

Compare Two Runs

Compare answers whether a prompt or model change actually helped. Pick two experiments on the same dataset, choose a baseline and a candidate, and review score, cost, and latency deltas together. Aggregates and row grades are computed for the selected panel pair only.

1. On Evaluate → Experiments (or the dataset Experiments tab), select exactly two runs, then click Compare. The older run is the baseline by default; swap anytime on the compare page. From an experiment detail page, use Compare with….
2. If either run used multiple Prompt Playground panels, choose which panels to compare (see below). Switching panels keeps the page in place and refreshes metrics for that pair.
3. Read the summary first: overall grade (Improvement, Regression, Tradeoff, or Tie), pass-rate delta, cost delta, and mean latency delta.
4. Filter the per-row table to Regressions, Improvements, Cost Up, or Any Fail. Expand a row for baseline/candidate pass chips and outputs. Use Show Both Outputs for a side-by-side preview, or Show Output Diff for a highlighted text diff.
5. When you are ready to ship, copy the suggested better experiment ID and attach it under Promote To Production on the prompt (see below).

Multi-Panel Runs

A single experiment can include more than one playground panel (for example two prompts or two models in one dataset run). Compare always evaluates one baseline panel against one candidate panel.

• Same panel count: pick a matching pair (Panel A vs Panel A, then switch to Panel B vs Panel B).
• Different panel counts: choose one panel from the baseline run and one from the candidate run.
• Single-panel runs skip this step. Pass rate, cost, latency, and row grades always reflect the active pair only.

Promote With Evidence

On a prompt detail page, Promote To Production opens a review step. Attach an experiment that covers the version you are promoting. After a compare, prefer the better run (candidate on an Improvement, baseline on a Regression) so production labels stay tied to measured evidence. To promote without an experiment, choose Promote Without Evidence and enter an Override Reason. That override is recorded in the audit log.