Evaluation

Test harness for agents — define EvalCases, run them with EvalRunner, and collect results into an EvalReport.

EvalCase

Bases: BaseModel

A single evaluation test case.

Defines what to send to the agent and what to expect back.

Example

case = EvalCase( ... name="ioc_triage", ... prompt="Is 198.51.100.23 malicious? Enrich it and decide.", ... expected_tools=["enrich_indicator"], ... expected_output_contains=["malicious", "198.51.100.23"], ... max_iterations=5, ... )

EvalResult

Bases: BaseModel

Result from evaluating a single case.

EvalReport

Bases: BaseModel

Aggregated report from running an eval suite.

summary

summary() -> str

Generate a human-readable summary.

Source code in .sdk/src/tulip/evaluation/framework.py

def summary(self) -> str:
    """Generate a human-readable summary."""
    lines = [
        f"Eval Report: {self.passed}/{self.total_cases} passed "
        f"(avg score: {self.avg_score:.2f})",
        f"Total duration: {self.total_duration_ms:.0f}ms",
        "",
    ]
    for r in self.results:
        status = "PASS" if r.passed else "FAIL"
        lines.append(
            f"  [{status}] {r.case_name} (score: {r.score:.2f}, {r.duration_ms:.0f}ms)"
        )
        if not r.passed:
            for check_name, check_passed in r.checks.items():
                if not check_passed:
                    lines.append(f"         - {check_name}: FAILED")
            if r.error:
                lines.append(f"         - error: {r.error}")
    return "\n".join(lines)

EvalRunner

EvalRunner(agent: Any, *, judge: Any = None, concurrency: int = 4)

Run evaluation cases against an agent.

Example

runner = EvalRunner(agent=my_agent) report = runner.run( ... [ ... EvalCase( ... name="basic", prompt="Hello", expected_output_contains=["hello"] ... ), ... EvalCase( ... name="tool_use", prompt="Search for X", expected_tools=["search"] ... ), ... ] ... ) print(report.summary())

Parameters:

Name	Type	Description	Default
agent	Any	The agent under test.	required
judge	Any	Optional :class:~tulip.evaluation.judge.LLMJudge. Cases carrying a rubric are graded by it; without one those cases are skipped rather than silently passing, so a suite cannot look green because nobody read the answers.	None
concurrency	int	Cases run at once in :meth:arun. Evals are mostly latency, so running them serially wastes wall clock; the cap keeps a large suite from stampeding a rate limit. Set it to 1 if the agent's model is stateful — a scripted test double hands out its turns in order, so concurrent cases will consume each other's and fail in ways that have nothing to do with the agent.	4

Source code in .sdk/src/tulip/evaluation/framework.py

def __init__(self, agent: Any, *, judge: Any = None, concurrency: int = 4) -> None:
    """
    Args:
        agent: The agent under test.
        judge: Optional :class:`~tulip.evaluation.judge.LLMJudge`.
            Cases carrying a ``rubric`` are graded by it; without one
            those cases are skipped rather than silently passing, so a
            suite cannot look green because nobody read the answers.
        concurrency: Cases run at once in :meth:`arun`. Evals are
            mostly latency, so running them serially wastes wall clock;
            the cap keeps a large suite from stampeding a rate limit.
            Set it to 1 if the agent's model is **stateful** — a
            scripted test double hands out its turns in order, so
            concurrent cases will consume each other's and fail in ways
            that have nothing to do with the agent.
    """
    self.agent = agent
    self.judge = judge
    self.concurrency = max(1, concurrency)

run

run(cases: list[EvalCase]) -> EvalReport

Run all eval cases and produce a report.

Source code in .sdk/src/tulip/evaluation/framework.py

def run(self, cases: list[EvalCase]) -> EvalReport:
    """Run all eval cases and produce a report."""
    results: list[EvalResult] = []

    for case in cases:
        result = self._run_case(case)
        results.append(result)

    passed = sum(1 for r in results if r.passed)
    scores = [r.score for r in results]
    total_duration = sum(r.duration_ms for r in results)

    return EvalReport(
        results=results,
        total_cases=len(cases),
        passed=passed,
        failed=len(cases) - passed,
        avg_score=sum(scores) / len(scores) if scores else 0.0,
        total_duration_ms=total_duration,
    )

arun async

arun(cases: list[EvalCase]) -> EvalReport

Run every case concurrently, then report.

The async path is the one that can grade with a judge: scoring is a model call, and run() is synchronous.

Source code in .sdk/src/tulip/evaluation/framework.py

async def arun(self, cases: list[EvalCase]) -> EvalReport:
    """Run every case concurrently, then report.

    The async path is the one that can grade with a judge: scoring is
    a model call, and ``run()`` is synchronous.
    """
    semaphore = asyncio.Semaphore(self.concurrency)

    async def one(case: EvalCase) -> EvalResult:
        async with semaphore:
            return await self._arun_case(case)

    results = list(await asyncio.gather(*(one(c) for c in cases)))
    return self._report(results, len(cases))

Grading an answer that has no single right string

Structural checks verify shape, not quality. expected_output_contains in particular fails in both directions: it passes on an answer that happens to contain the word, and fails on a correct answer phrased differently. Where the right answer is not one exact string, something has to read it.

LLMJudge grades against a written rubric and returns a typed Verdict. Two things it deliberately does not do:

• It does not retry until it passes. A judge you re-roll is not a judge.
• It does not score zero when it cannot be reached. An unusable judge raises, because a "failing" eval that actually means the judge was down is worse than no eval.

Use a different model from the one under test where you can — a model grading its own output is measuring self-consistency, not correctness.

LLMJudge

LLMJudge(model: Any, *, max_tokens: int = 2048)

Grade a response against a rubric using a model.

Parameters:

Name	Type	Description	Default
model	Any	Any object satisfying ModelProtocol. Use a different model from the one under test where you can — a model grading its own output is measuring self-consistency, not correctness.	required
max_tokens	int	Budget for the verdict. The reply is one small JSON object, but the default is deliberately generous: a reasoning model spends the allowance thinking first and returns empty content if it runs out, which parses as a failed verdict rather than an error. Measured on a local Qwen3.6-35B — measured against a local Qwen3.6-35B, the reply was empty at 256 and 512, and a clean verdict from 1024 up. 1024 sits on the boundary — reasoning length varies per call, so the same budget that worked once returned empty on the next run. The default is well clear of it.	2048

Source code in .sdk/src/tulip/evaluation/judge.py

def __init__(self, model: Any, *, max_tokens: int = 2048) -> None:
    self.model = model
    self.max_tokens = max_tokens

score async

score(*, prompt: str, output: str, rubric: str) -> Verdict

Grade output against rubric.

Raises:

Type	Description
RuntimeError	If the model call fails. An eval that reports failure when the judge was simply unreachable is worse than one that stops and says so.

Source code in .sdk/src/tulip/evaluation/judge.py

async def score(self, *, prompt: str, output: str, rubric: str) -> Verdict:
    """Grade ``output`` against ``rubric``.

    Raises:
        RuntimeError: If the model call fails. An eval that reports
            failure when the judge was simply unreachable is worse than
            one that stops and says so.
    """
    request = _PROMPT.format(rubric=rubric.strip(), prompt=prompt, output=output)
    try:
        response = await self.model.complete(
            [Message.user(request)], max_tokens=self.max_tokens
        )
    except Exception as exc:
        raise RuntimeError(f"LLM judge could not be reached: {exc}") from exc

    return self._parse(response.content or "")

Verdict

Bases: BaseModel

One judge decision.

Asserting the order, not just the membership

expected_tools asks whether a tool appeared somewhere in the run. It cannot tell "looked the order up, then refunded it" from "refunded it, then looked it up" — and for an agent that acts, that ordering is most of what correctness means.

check_trajectory

check_trajectory(actual: Sequence[str], expected: Sequence[str], *, exact: bool = False) -> tuple[bool, str]

Check the order tools were called in, not merely that they were.

expected_tools asks whether a tool appeared anywhere in the run, which cannot distinguish "looked the order up, then refunded it" from "refunded it, then looked it up". For an agent that acts, that ordering is most of what correctness means.

Parameters:

Name	Type	Description	Default
actual	Sequence[str]	Tool names in the order they were called.	required
expected	Sequence[str]	The order required.	required
exact	bool	When True the sequences must match exactly. The default requires only that expected appears in order, so unrelated extra calls — a retry, a lookup the agent added — do not fail a test about ordering.	False

Returns:

Type	Description
bool	(passed, reason), where the reason names what was missing or out
str	of order so a failure is actionable without a debugger.

Source code in .sdk/src/tulip/evaluation/judge.py

def check_trajectory(
    actual: Sequence[str],
    expected: Sequence[str],
    *,
    exact: bool = False,
) -> tuple[bool, str]:
    """Check the order tools were called in, not merely that they were.

    ``expected_tools`` asks whether a tool appeared anywhere in the run, which
    cannot distinguish "looked the order up, then refunded it" from "refunded
    it, then looked it up". For an agent that acts, that ordering is most of
    what correctness means.

    Args:
        actual: Tool names in the order they were called.
        expected: The order required.
        exact: When ``True`` the sequences must match exactly. The default
            requires only that ``expected`` appears in order, so unrelated
            extra calls — a retry, a lookup the agent added — do not fail a
            test about ordering.

    Returns:
        ``(passed, reason)``, where the reason names what was missing or out
        of order so a failure is actionable without a debugger.
    """
    actual = list(actual)
    expected = list(expected)

    if exact:
        if actual == expected:
            return True, "trajectory matches exactly"
        return False, f"expected exactly {expected}, got {actual}"

    remaining = iter(actual)
    for step in expected:
        if not any(call == step for call in remaining):
            missing = expected[expected.index(step) :]
            return False, (f"expected {expected} in order; {missing} did not follow, got {actual}")
    return True, "trajectory contains the expected order"

Running a suite against a graph

EvalRunner expects an agent: it calls run_sync(prompt) and reads .message, .iterations and .tool_executions. A StateGraph takes a dict and returns a GraphResult with none of those. as_eval_target adapts one to the other.

It has to make two translations a graph cannot make for itself:

• A prompt is not a state. input_key says which field of the graph's initial state the case prompt becomes. There is no universally right default, so "prompt" is a starting guess to correct.
• Nodes are the graph's tools. expected_tools and expected_tool_sequence match on node ids, which makes "a change sent this down the wrong branch" — the regression a graph suite exists to catch — an ordinary assertion.

Addressing the answer takes two arguments rather than one, because final_outputs is keyed by node id while a graph's result accumulates in final_state: output_key reads the state, output_node reads one node's own output, and passing both raises rather than silently preferring one.

as_eval_target

as_eval_target(graph: Any, *, input_key: str = 'prompt', output_key: str | None = None, output_node: str | None = None, initial_state: dict[str, Any] | None = None) -> GraphEvalTarget

Wrap a graph so :class:~tulip.evaluation.EvalRunner can run it.

runner = EvalRunner(agent=as_eval_target(graph, input_key="question"))
report = runner.run([
    EvalCase(
        name="routes_billing_to_the_billing_node",
        prompt="I was charged twice",
        expected_tool_sequence=["classify", "billing"],
    )
])

expected_tools and expected_tool_sequence match on node ids; everything else — expected_output_contains, budgets, a rubric graded by :class:~tulip.evaluation.LLMJudge — behaves exactly as it does for an agent.

Source code in .sdk/src/tulip/evaluation/graph.py

def as_eval_target(
    graph: Any,
    *,
    input_key: str = "prompt",
    output_key: str | None = None,
    output_node: str | None = None,
    initial_state: dict[str, Any] | None = None,
) -> GraphEvalTarget:
    """Wrap a graph so :class:`~tulip.evaluation.EvalRunner` can run it.

        runner = EvalRunner(agent=as_eval_target(graph, input_key="question"))
        report = runner.run([
            EvalCase(
                name="routes_billing_to_the_billing_node",
                prompt="I was charged twice",
                expected_tool_sequence=["classify", "billing"],
            )
        ])

    ``expected_tools`` and ``expected_tool_sequence`` match on **node ids**;
    everything else — ``expected_output_contains``, budgets, a rubric graded by
    :class:`~tulip.evaluation.LLMJudge` — behaves exactly as it does for an
    agent.
    """
    return GraphEvalTarget(
        graph,
        input_key=input_key,
        output_key=output_key,
        output_node=output_node,
        initial_state=initial_state,
    )

GraphEvalTarget

GraphEvalTarget(graph: Any, *, input_key: str = 'prompt', output_key: str | None = None, output_node: str | None = None, initial_state: dict[str, Any] | None = None)

A StateGraph wearing the shape EvalRunner expects.

Parameters:

Name	Type	Description	Default
graph	Any	The graph under test.	required
input_key	str	Which field of the graph's initial state the case prompt becomes. There is no universally right default — a graph's input schema is its own — so "prompt" is a starting guess to correct rather than a convention to rely on.	'prompt'
output_key	str | None	Which field of the graph's final state holds the answer to grade. That is where a graph's result accumulates — final_outputs is keyed by node id, which is a different question and has its own argument below.	None
output_node	str | None	Grade the output of one named node instead. Use this when the answer never lands in shared state, or when only the last node's own return value matters.	None
initial_state	dict[str, Any] | None	Fields merged under the prompt on every case — a graph that needs a config blob or a session id to start at all should not have to encode it in every EvalCase.	None

Source code in .sdk/src/tulip/evaluation/graph.py

def __init__(
    self,
    graph: Any,
    *,
    input_key: str = "prompt",
    output_key: str | None = None,
    output_node: str | None = None,
    initial_state: dict[str, Any] | None = None,
) -> None:
    if output_key is not None and output_node is not None:
        raise ValueError(
            "pass output_key (a field of the final state) or output_node "
            "(one node's own output), not both — they name different things"
        )
    self.graph = graph
    self.input_key = input_key
    self.output_key = output_key
    self.output_node = output_node
    self.initial_state = dict(initial_state or {})