Scoring

Every grader returns two independent things:

Field	Type	Meaning
passed	boolean	Did this check pass?
score	0.0 – 1.0	How well did it do?

They can disagree, and which one drives your verdict depends on your scoring config.

Verdicts exist at three levels:

Level	Covers	Passes when
Trial	One run of a stimulus	Its graders pass (or its score clears the threshold)
Stimulus	All trials of that stimulus	Every trial passed; mixed results are flagged flaky
Eval	All stimuli	Every trial passed, or the eval score clears the threshold

runs defaults to 1, so unless you ask for more trials each stimulus has exactly one and the first two levels coincide.

Default verdict

With no scoring block in your eval.yaml, a trial passes when every grader passed:

passed = all graders passed     // score is not consulted

One failed grader fails the trial, and a failed trial fails its stimulus and the eval. When a stimulus has multiple graders, vally lint points this out so the binary behavior isn’t mistaken for weighted scoring:

⚠ warning [scoring-defaults-applied] Multiple graders with no scoring config
  Using defaults: equal weights, no threshold — the verdict is binary, so any single failing grader fails the stimulus. Add a `scoring` block to weight graders or set a threshold.

All-or-nothing is the default and needs no configuration.

Aggregate score

Alongside each trial’s verdict, Vally computes an aggregate score. With no weights, it is the mean of the individual grader scores. It is independent of the verdict and is reported whether or not you configure scoring.

Three graders where one fails:

score  = 0.667      ← mean of 1.0, 1.0, 0.0
passed = false      ← the verdict

Because it is a mean, the score reflects partial credit even when the verdict is all-or-nothing. It is diagnostic; the verdict is what passes or fails the trial.

Thresholds

If you need a verdict that isn’t “every grader passed”, set a threshold. Each trial’s verdict then compares its aggregate score against it:

eval.yaml

scoring:
  threshold: 0.7

passed = score >= threshold     // each grader's own `passed` is not consulted

This applies to trials that ran and were graded. Execution errors fail the eval regardless of score, even at --threshold 0.

It can fail a trial where every grader passed. Graders may pass with a score below 1.0 — an LLM judge scoring 4/5, or a program grader printing {"passed": true, "score": 0.6}. All graders pass, the mean lands at 0.9, and threshold: 1.0 fails it.

It can pass a trial where a grader failed. A grader reporting {"passed": false, "score": 0.95} alongside a passing check averages to 0.975, which clears threshold: 0.9. The report keeps the contradiction visible — verdict ✅ with the footnote Failed grader(s): program python3.

Thresholds are for genuinely graded work — judged quality, partial credit — not for pass/fail assertions like file-exists.

Eval verdict

The eval-level score is the mean across scored stimuli, so one failed stimulus can be carried by its neighbors:

stimuli scoring 0.8, 1.0, 0.5  →  eval score 76.7%  →  ✔ PASS (threshold 0.7)

With no threshold, the same three stimuli fail the eval, because one of them failed.

Stimuli with no graders are unscored and excluded from that mean, as are stimuli that never ran. An eval whose stimuli are all unscored scores 0% and fails when a threshold is set — but passes with no threshold, since it has no failing grader.

Weights

weights adjusts how much each grader type contributes to the score:

eval.yaml

scoring:
  weights:
    file-exists: 0.7
    output-contains: 0.3
  threshold: 0.7

Weights change the score, not the default verdict. They always shape the reported score, but with no threshold the verdict is still every grader passing.

Graders of the same type are averaged first, then the type’s weight applies once. Two file-contains graders (one passing, one failing) plus a passing program grader, weighted file-contains: 0.3 / program: 0.7:

0.3 × 0.5  +  0.7 × 1.0  =  0.85

A type you omit gets weight 0 — and disappears from the score entirely:

weights: { program: 1.0 } # file-contains omitted

score = 100%   ← even though only 1/2 graders passed

That eval passes at threshold: 0.9 while a file-contains grader is failing. Omitting a type silences it, so list every type you care about.

Weights in a non-empty map must sum to 1.0 (±0.01); vally lint reports scoring-weight-sum otherwise. The score is renormalized over the types that actually ran, so a stimulus exercising only some of the weighted types can still reach 1.0. An empty map behaves as if weights were omitted.

Trials

Agents are non-deterministic: the same stimulus can pass on one run and fail the next. Set defaults.runs to repeat each stimulus (a positive integer — 0 is rejected). Each trial gets its own verdict and score; the stimulus score is their mean:

✔ ✘ ✔ ✔   3/4 trial(s) passed
per-trial scores: 1, 0.5, 1, 1   →   stimulus score 0.875

Even when every grader is strictly 0 or 1, this mean makes the stimulus score a decimal: a single 0/1 grader scores 0.75 when three of four trials pass. The per-trial verdicts stay binary, so use the pass rate to see how many trials passed.

Metrics

Metric	Meaning
Pass rate	Trials that passed, e.g. 3/4
pass^k	Chance all k trials succeed — p^k. For 3/4: 31.6%
pass@k	Whether any trial passed. Per stimulus: 100% or 0%
Flakiness	Minority outcome ÷ total. 4/5 passing → 20%

pass^k is the reliability number: 75% per-trial becomes 31.6% over four runs — the gap between “usually works” and “safe to gate CI on”.

pass@k is computed per stimulus with k set to the number of trials you ran, and a run that already succeeded is guaranteed to contain a success — so a stimulus reports either 100% or 0%. Read it as “did this ever work?” The eval-level figure is the mean of those per-stimulus values, so it can land anywhere in between.

Flakiness is flagged whenever outcomes are mixed (🟡 in reports) and usually means an ambiguous prompt, a tight timeout, or an over-strict grader. Compare a failing trial’s trajectory against a passing one.

Grader thresholds

Separate from scoring.threshold, LLM judge graders binarize their own result. The prompt grader turns a rubric score into passed using its own config.threshold, which defaults to 0.5 normalized. (panel uses the same threshold, plus aggregation rules across judges and required criteria.)

Normalization is min-max over the scale’s range, not division — on scale_1_5, a raw 4 is (4 - 1) / (5 - 1) = 0.75, not 0.8. Judges emit integers, so the default 0.5 threshold first passes at 3 on scale_1_5 and at 6 on scale_1_10.

So “no scoring block” is all-or-nothing over grader verdicts, and a judge grader applies its own 0.5 bar to produce that verdict. Set it explicitly for a stricter bar:

- type: prompt
  config:
    scoring: scale_1_5
    threshold: 0.75 # normalized, so a raw 4 of 5

Where values appear

Surface	Shows
Console	Verdict; the score only when a threshold is configured
Markdown report	Per-grader results, pass rate, pass^k, pass@k, verdict
results.jsonl	gradeResult.passed and gradeResult.score
JUnit	Scores plus multi-trial metrics and flakiness
Dashboard	Per-trial verdicts and scores, and per-grader scores

A decimal in results.jsonl or the dashboard is diagnostic, not the verdict. The exit code follows passed.

What to configure

Goal	Configuration
Assertions that must all hold	No scoring block
Judged quality with partial credit	threshold, tuned to your rubric
Some checks matter more than others	weights and a threshold

Context	runs	Why
Inner loop	1	Fast feedback while writing graders
CI gate	3	Detects flakiness without slowing PRs
Nightly / benchmarks	5–10	Meaningful pass^k, especially with LLM judges

Next steps

• How it works — where scoring sits in the pipeline
• Writing eval specs — configuring scoring
• Scoring functions reference — formulas and types