Regression Testing Prompts and Pipelines

Prompts are code. They change behavior, they have bugs, and they interact with the rest of the system in ways that are hard to predict by reading them. Yet many teams still validate prompt changes by trying a few examples by hand. Regression testing replaces that with a repeatable process: a fixed set of cases, a set of checks, a threshold and a record of how each version performed.

Why prompts and pipelines regress

• Prompt edits fix one case and break another, often in a category nobody thought to try.
• Model changes, including provider-side updates behind a stable alias, shift formatting, verbosity and refusal behavior.
• Pipeline changes to chunking, retrieval, reranking or output parsing alter what the model sees or how its output is used.
• Nondeterminism means a single passing run is weak evidence; behavior has a distribution, not a single value.

Building a golden set

A golden set is a versioned collection of inputs with expectations about the output. It does not need to be large to be useful; a few dozen carefully chosen cases catch more regressions than hundreds of near-duplicates. Aim for coverage of the behaviors you care about rather than volume.

• Typical cases sampled from real traffic, with personal data removed.
• Edge cases: empty or very long inputs, mixed languages, ambiguous requests and questions the system should decline to answer.
• Past incidents: every bug report that reached production becomes a case, so the same failure cannot recur silently.
• Adversarial inputs: prompt-injection attempts in user text and in retrieved documents.

Store expectations as properties rather than exact strings wherever possible: allowed categories, phrases that must or must not appear, fields that must parse, the source document that must be cited. Exact-match expectations break on harmless rewording and train the team to ignore failures.

Deterministic checks versus model-graded checks

Deterministic checks are ordinary assertions: the output parses against a schema, the label is in the allowed set, a required citation ID is present, a forbidden phrase is absent, the response is under a length limit. They are fast, cheap, reproducible and easy to debug. Use them for everything they can express.

Model-graded checks use a second model with a rubric to judge qualities that are hard to assert in code, such as whether an answer is grounded in the provided context or addresses the question. They extend coverage to the properties users actually notice, but they come with real caveats.

Writing the tests with pytest

Plain pytest is enough to get started. Run the pipeline once per case in a session-scoped fixture, assert deterministic properties per case, and gate model-graded quality on an aggregate rate rather than on individual cases.

1import json2from pathlib import Path3 4import pytest5 6from app.pipeline import PipelineResult, run_pipeline7from evals.graders import is_grounded  # hypothetical rubric-based grader8 9CASES = [json.loads(line) for line in10         Path("tests/golden/support.jsonl").read_text().splitlines() if line.strip()]11GROUNDED_THRESHOLD = 0.90  # set from your own baseline runs12 13 14@pytest.fixture(scope="session")15def results() -> dict[str, PipelineResult]:16    return {case["id"]: run_pipeline(case["input"]) for case in CASES}17 18 19@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])20def test_case_properties(case: dict, results: dict[str, PipelineResult]) -> None:21    result = results[case["id"]]22    assert result.category in case["allowed_categories"]23    answer = result.answer.lower()24    for phrase in case.get("must_include", []):25        assert phrase.lower() in answer, f"missing {phrase!r}"26    for phrase in case.get("must_not_include", []):27        assert phrase.lower() not in answer, f"forbidden {phrase!r}"28 29 30def test_grounded_rate(results: dict[str, PipelineResult]) -> None:31    verdicts = [is_grounded(r.answer, r.context) for r in results.values()]32    rate = sum(verdicts) / len(verdicts)33    assert rate >= GROUNDED_THRESHOLD, f"grounded rate {rate:.0%}"

Choosing thresholds

Thresholds should come from measurement, not intuition. Run the current production version several times to see how much the aggregate score moves on its own; that spread is your noise floor. Set the gate below the baseline by more than that spread, or compare against the baseline directly and fail only on a drop larger than the noise. Suppose the baseline grounded rate varies between 0.92 and 0.95 across repeated runs: a gate at 0.94 would flake, while a gate near 0.90 catches real drops without crying wolf.

Keep a small set of must-pass cases, such as safety refusals and past incidents, that fail the build individually regardless of the aggregate. Everything else can be judged in aggregate.

Nondeterminism deserves its own policy. Lowering temperature reduces variation but does not guarantee identical outputs across runs or over time, so do not build a suite that only passes when outputs are byte-for-byte stable. For cases that flip between runs, either loosen the expectation to the property you actually care about or run the case several times and assert on the pass rate. A flaky case left in place teaches everyone to re-run the job until it turns green, which defeats the purpose of the gate.

Running evaluations in CI

Trigger the suite on changes to prompts, pipeline code, model configuration and the golden set itself. Because these tests call real models, they are slower and cost money, so scope the trigger with path filters and keep a fast deterministic subset for every commit.

1name: prompt-regression2 3on:4  pull_request:5    paths:6      - "prompts/**"7      - "app/pipeline/**"8      - "config/models.yaml"9      - "tests/golden/**"10 11jobs:12  golden-set:13    runs-on: ubuntu-latest14    timeout-minutes: 2015    steps:16      - uses: actions/checkout@v417      - uses: actions/setup-python@v518        with:19          python-version: "3.12"20      - run: pip install -r requirements.txt21      - name: Run golden set22        env:23          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}24        run: pytest tests/test_golden.py --junitxml=eval-results.xml25      - uses: actions/upload-artifact@v426        if: always()27        with:28          name: eval-results29          path: eval-results.xml

Tracking diffs across prompt and model versions

A pass or fail result is not enough to review a change. Persist per-case outputs and scores for every run, keyed by prompt version, model identifier, pipeline commit and golden-set version. Then a reviewer can see a side-by-side diff: which cases flipped from pass to fail, which improved, and how the outputs changed in wording and length.

• Version everything that affects output, including the grader prompt and the golden set, so results are comparable.
• Review flips, not just totals. An unchanged aggregate can hide five fixes and five new failures.
• Watch secondary metrics such as output length, refusal rate and token usage, which often shift with model upgrades.
• Re-baseline deliberately when a change is accepted, and record why in the pull request.

Regression testing will not prove a prompt is correct. It gives you something more practical: a consistent, reviewable signal that a change did not break what already worked, and a growing record of the behaviors your system is expected to keep.