Verify findings before you act on them

A Tulip Evidence is a claim. Before you act on it, verify() puts it through an independent skeptic that challenges the evidence and scores confidence. A well-grounded finding survives; an unsupported one is refuted, with the objections listed. verify() only returns the verdict: gate the action on survives so a refuted finding never reaches it. The default skeptic grades the evidence a finding carries; it does not check that the references are real.

verify() takes a Tulip Evidence or a finding-shaped dict (title, severity, gsar_score, evidence_refs, confidence), so a finding produced outside Tulip goes through the same challenge. The notebook verifies a real grounded finding (survives), a fabricated one with no evidence (refuted), and an external finding that has references but was never grounded (refuted).

Runs fully offline.

Run it: python examples/notebook_78_verify_findings.py

See also: Agentic AI-security · SecurityContext.

Source

#!/usr/bin/env python3
# Copyright 2026 The Tulip Authors
# SPDX-License-Identifier: Apache-2.0

from __future__ import annotations

import asyncio

from tulip_security import Target, is_finding, red_team, verify


def _vulnerable_bot(prompt: str) -> str:
    return prompt  # naive bot: obeys injected instructions (see notebook 76)


def _show(label: str, verdict: object) -> None:
    v = verdict  # VerificationResult
    status = "SURVIVES" if v.survives else "REFUTED"  # type: ignore[attr-defined]
    print(f"\n[{status}] {label}  (confidence {v.confidence:.2f})")  # type: ignore[attr-defined]
    for r in v.refutations:  # type: ignore[attr-defined]
        print(f"    - ({r.weight}) {r.reason}")


async def main() -> int:
    print("verify() — does the evidence actually support the claim?")

    # 1. A REAL finding from red-teaming a vulnerable bot — well grounded.
    report = await red_team(Target.from_callable(_vulnerable_bot, name="bot"), suite="owasp-asi")
    real_finding = next(r for r in report if is_finding(r))
    _show(f"real finding: {real_finding.title}", await verify(real_finding))

    # 2. A FABRICATED finding from some other agent — no evidence at all.
    hallucination = {"title": "Host 10.0.0.5 is compromised", "severity": "critical"}
    _show("hallucinated finding (no evidence)", await verify(hallucination))

    # 3. An external finding that has refs but was never grounded.
    ungrounded = {
        "title": "Possible data exfiltration",
        "severity": "high",
        "evidence_refs": ["siem:alert:9931"],
    }
    _show("external finding (ungrounded)", await verify(ungrounded))

    return 0


if __name__ == "__main__":
    raise SystemExit(asyncio.run(main()))