🥷 Ninja Harness

Trace-first evals for agents that need to survive production.

â–¶ Try the playground GitHub Masterclass
Apache-2.0Python 3.11+framework-agnosticdeterministic-firstno API keys

What is it?

Most agent tests check the final answer. Ninja Harness evaluates the full execution trajectory an agent took to get there, scores it across eight dimensions into a single 0–100 reliability index, and issues a PASS / WARN / FAIL certification. It's the layer that grades agents — not another agent framework.

1Why the whole trajectory?

In production, failures hide in the path, not the sentence at the end — the wrong tool got called, a handoff dropped context, a secret leaked, or the agent looped 40 times. Ninja Harness inspects every step:

Goal→ Plan→ Tool Call→ Observation→ Handoff→ Guardrail→ Recovery→ Final Answer

A polished final answer can hide an unsafe or unreliable path. The trajectory tells the truth.

2What it measures

Eight weighted metrics combine into the Ninja Agent Reliability Index (NARI). Metrics that don't apply to a trace are skipped and their weight is redistributed.

Goal Success25%
Did the final answer match what was expected?
Tool Call F115%
Right tools, right arguments — precision & recall.
Handoff Integrity15%
Do multi-agent handoffs carry full context?
Grounding15%
Are claims supported by the provided references?
Safety10%
Leaked secrets, injection compliance, unsafe escalation.
Efficiency10%
Steps, repeats, loops, latency, tokens, cost.
Recovery5%
Did it recover after a failure or blocked guardrail?
Stability5%
Did the score regress vs a saved baseline?
Output Hygienereported
Penalizes noisy answers (logs, dumps); not in the composite.

3How it certifies

The composite score maps to a certification. Safety is a hard gate — a low safety score forces FAIL no matter how good everything else is.

FAIL
WARN
PASS
06080100

PASS = score ≥ 80 and safety ≥ 0.8 · WARN = score ≥ 60 · FAIL = score < 60 or safety < 0.5

4Reliability, not luck (pass^k)

Agents are stochastic — a single passing run can be luck. Run the same task k times and look at two numbers:

pass@k (≥1 pass) ✓ "it can work" — fine for a demo
pass^k (all pass) ✓ "it always works" — what production needs

The gap between them is the flaky-agent signature. ninja-harness run --repeat 5 reports both.

5Use it in four steps

  1. Install

    pip install ninja-harness        # or: pipx run ninja-harness --help

  2. Evaluate a trace against an eval case

    ninja-harness eval --trace trace.json --case case.yaml

    A trace is JSON; framework traces (OpenAI Agents, LangGraph, Hermes, CrewAI, AutoGen, OpenTelemetry) are auto-detected.

  3. Or run your agent end-to-end

    ninja-harness run --task task.yaml --solver-cmd "python my_agent.py" --repeat 5

    Your agent reads the task on stdin and prints a trace on stdout — any language. You get a certified report + reliability stats.

  4. Gate CI & schedule it

    ninja-harness gate --results results.json --policy policy.yaml
ninja-harness suite --suite scenarios/suite.yaml --format summary

    Exit codes fail your build; the summary posts to Slack/Telegram. Findings export to SARIF/JUnit.

â–¶Try it in your browser

This runs Ninja Harness entirely client-side via Pyodide — no server, no install, nothing leaves your browser. First launch downloads the Python runtime (a few MB), then it's cached.

Evaluate →
Open source · Apache-2.0 · github.com/gagans23/ninja-harness
A behavior-certification harness — not an agent framework, and not a substitute for human review in high-risk domains.