Evaluation¶

An agent that worked yesterday may not work today — the model changed, a tool was renamed, the prompt got a one-line tweak. Tulip ships a small evaluation harness so regressions become failing tests, not silent failures in production.

from tulip.evaluation import EvalCase, EvalRunner

cases = [
    EvalCase(
        name="order_lookup",
        prompt="Look up order ord-4821.",
        expected_tools=["lookup_order"],
        expected_output_contains=["status", "ord-4821"],
        max_iterations=5,
    ),
]

report = EvalRunner(agent=agent).run(cases)
print(report.summary())

When to reach for an eval suite¶

Situation	Run evals?
You changed a tool's signature, default args, or system prompt	yes — every commit that touches it
You're swapping models (gpt-4o → gpt-5, llama-3.3 → llama-4)	yes — same suite, two providers, diff the report
You're debating "is the agent better than last week?"	yes — nightly soak with `n=20` per case to see variance
One-shot exploration, scratch agent	no — overhead's not worth it
Heavy LLM-as-judge needed (open-ended quality)	yes — set a `rubric` on the case and give the runner an `LLMJudge` (below)

Getting started¶

1. Define cases¶

EvalCase is a Pydantic model — every field is optional except name and prompt. The runner only checks fields you set.

from tulip.evaluation import EvalCase

refunds_eligible = EvalCase(
    name="refunds_eligible_order",
    prompt="Refund order ord-4821 — return approved.",
    expected_tools=["issue_refund"],
    expected_output_contains=["ord-4821", "refunded"],
    max_iterations=4,
)

rejects_unknown = EvalCase(
    name="rejects_unknown_order",
    prompt="Refund order zz-999.",
    expected_output_contains=["not found"],
    expected_output_not_contains=["refunded"],
)

2. Run them¶

from tulip.evaluation import EvalRunner

runner = EvalRunner(agent=agent)
report = runner.run([refunds_eligible, rejects_unknown])

print(report.summary())
# Eval Report: 2/2 passed (avg score: 1.00)
# Total duration: 4321ms
#   [PASS] refunds_eligible_order (score: 1.00, 1872ms)
#   [PASS] rejects_unknown_order (score: 1.00, 2449ms)

run() returns an EvalReport — a Pydantic model with per-case results, aggregate pass/fail counts, average score, and total duration. JSON-serialisable, drop into CI artifacts.

3. Wire it into CI¶

# tests/test_agent_evals.py
import pytest
from tulip.evaluation import EvalRunner

def test_agent_passes_eval_suite(agent):
    report = EvalRunner(agent=agent).run(load_cases())
    failures = [r for r in report.results if not r.passed]
    assert not failures, report.summary()

Built-in checks¶

Every check runs only when the corresponding field is set on the case. Each check contributes equally to the per-case score.

Field	Passes when
`expected_tools`	All listed tools appear in the run's tool executions.
`expected_output_contains`	Every string is a case-insensitive substring of the final message.
`expected_output_not_contains`	None of the strings appear in the final message.
`max_iterations`	The run finished in ≤ N ReAct turns.
`max_duration_ms`	Wall-clock duration ≤ N milliseconds.

A case passes when every check passed; the score is the fraction of checks that passed (handy for partial-credit scoring across a soak).

Tags and filtering¶

EvalCase(name="..." , prompt="..." , tags=["smoke", "happy-path"])
EvalCase(name="..." , prompt="..." , tags=["adversarial"])

# Run only smoke cases on every commit; full suite nightly.
smoke = [c for c in all_cases if "smoke" in c.tags]
runner.run(smoke)

tags is just a list — slice it however your CI matrix expects.

LLM-as-judge for open-ended quality¶

The built-in checks are structural ("did the right tool fire?", "did the answer mention 'reputation'?"). For free-text quality ("is the triage rationale sound?", "is the finding correctly grounded?"), write a rubric on the case and give the runner an LLMJudge:

import asyncio

from tulip.evaluation import EvalCase, EvalRunner, LLMJudge
from tulip.models import get_model

case = EvalCase(
    name="sound_triage_rationale",
    prompt="Explain why alert SOC-4821 is a false positive.",
    expected_output_contains=["benign"],  # structural checks still run
    rubric="Passes if it names the evidence that makes the alert benign.",
)

judge = LLMJudge(model=get_model("openai:gpt-4o-mini"))
runner = EvalRunner(agent=agent, judge=judge)
report = asyncio.run(runner.arun([case]))

Grading requires await runner.arun(cases) — the judge is a model call, and run() is synchronous. run() never reads rubric at all, so a rubric case passed to it is scored on its structural checks alone and can report green with the rubric silently unevaluated. On arun(), a rubric case with no judge configured fails loudly instead, under a rubric:no_judge_configured check.

On a graded case, EvalResult.score is the judge's 0–1 score, not the fraction of checks passed. A judge reply that cannot be parsed yields Verdict(unparseable=True) and is recorded as a failing judge_unparseable:... check rather than a silent pass. Use a different model from the one under test where you can — a model grading its own output is measuring self-consistency, not correctness. LLMJudge and Verdict are documented in the evaluation API reference.

Common gotchas¶

Symptom	Likely cause
Case passes locally, fails in CI	Model output varies between runs. Pin the model id, lower `temperature`, run with `n=5` and look at variance.
`max_duration_ms` flakes	Cold-start network latency. Use a wall-clock budget at the suite level, not per-case, or bump the per-case budget by 2×.
`expected_tools` reports failure even though the tool ran	Case-sensitive name match — `issue_refund` != `Issue_Refund`.
Score is 0.5 every time	One of two checks is consistently failing. Read `result.checks` — it carries the full pass/fail map.

Evals are audit evidence¶

The runner drives the same agent through the same loop as production, so an eval run is made of the same typed events — and any admission decisions land on the same hash-chained AuditTrail (each entry commits to the one before it, so editing any record breaks verification) the gate always writes. A passing eval is audit evidence, not a screenshot. For grounded-claim scoring — did each claim survive contact with its evidence? — pair the structural checks here with the GSAR evaluation layer.

Source and notebook¶

notebook_55_evaluation.py — runnable end-to-end suite.
tulip.evaluation.framework — EvalCase, EvalRunner, EvalReport.