Case study · Applied AI research
A reliability benchmark for AI agents in regulated industrial work.
Most AI evaluations grade a model’s final answer. This one grades how a multi-step agent behaved to get there — the tool calls, the escalations, and the claims it can actually back up.
- Built by
- Marcellus Spears
- Type
- Solo build · open source
- Domain
- Aerospace · regulated ops
The problem
In regulated work, how the agent got the answer matters as much as the answer.
When an AI agent supports a supply-chain, compliance, or bid decision in an industrial setting, a single misstep is not a bad demo — it is a safety, compliance, or cost event. An agent that selects the wrong tool, invents a part number, skips a required regulatory check, or never escalates a high-risk decision can do real damage while sounding perfectly confident.
Standard AI evaluations score the final text. They say nothing about the path the agent took to produce it. For a buyer deciding whether to let an agent near a regulated workflow, that gap is exactly the thing they need measured — and almost nobody measures it.
What I built
A benchmark that scores the trajectory, not just the output.
The benchmark runs 18 synthetic aerospace scenarios spread evenly across three workflow families: six supply-chain scenarios (inventory shortage, approved alternates, expedite paths, traceability, dual release, export review), six regulatory scenarios (controlled technical data, PMA repair evidence, safety-critical approval, classification uncertainty), and six bid / no-bid scenarios. Every scenario is synthetic and safe for public review.
A Python harness runs an agent through each scenario and scores the full trajectory on six behavioral dimensions. Two baselines anchor the scale: a well-behaved ReferenceAgent and a FlawedAgent — a negative control built to fail in realistic ways. The distance between them is the benchmark’s discriminating power. A Next.js dashboard reads the JSON artifacts and renders the leaderboard, scenario coverage, and per-scenario failure analysis, and GitHub Actions validates every run with no secrets and no API spend.
The numbers
A clean, well-behaved agent and a broken one, scored side by side.
18
Synthetic aerospace scenarios across three workflow families
6
Behavioral dimensions scored on every agent run
92%
Composite score the reliability gate requires to pass
95%
Hallucination rate the benchmark catches in the flawed control
| Agent | Composite | Task | Tool calls | Loop term. | Grounding | Governance | Hallucination |
|---|---|---|---|---|---|---|---|
| ReferenceAgent | 100% | 100% | 100% | 100% | 100% | 100% | 0% |
| FlawedAgent | 33% | 7% | 74% | 67% | 5% | 3% | 95% |
The reference agent clears the reliability gate; the flawed agent fails it by design — proving the benchmark detects missing evidence, unsupported claims, runaway tool loops, and absent escalation. Composite scores use weighted dimensions (task 0.28, tool calls 0.24, loop termination 0.18, grounding 0.18, governance 0.12). The gate requires a composite of 92% or higher, 100% tool-call accuracy, 100% loop termination, and grounding of 80% or higher.
What it measures
Six dimensions, each tied to a real failure mode.
Task completion
Confirms the agent satisfied the operational request, not just produced fluent text.
Tool-call accuracy
Checks the selected tools, their order, and the allowed-tool policy across multi-step workflows.
Loop termination
Detects runaway tool loops, repeated calls, and max-step failures.
Grounding
Penalizes forbidden or unsupported claims and missing evidence citations.
Governance
Rewards risk flags, confidence, evidence, escalation, and human-review markers.
Hallucination risk
Surfaces fabricated part numbers, invented approvals, and unsupported recommendations.
Honest trade-offs
What this is — and what it is not yet.
- The data is synthetic. The scenarios model real aerospace workflows but use no proprietary or classified data, which is what makes the whole benchmark safe to publish and reproduce. A client engagement would swap in their real workflows.
- Eighteen scenarios is a v0. It is enough to prove the scoring discriminates; the roadmap expands to 50+ scenarios across more industrial domains and adds adversarial prompt-injection cases.
- The baselines are deterministic by choice. Public CI runs without secrets or API spend, so the two reference agents are deterministic. A frontier-model leaderboard — running live agents through the same scenarios — is the next milestone.
Why it matters for your business
Deploying an agent is easy. Knowing it is safe to trust is the work.
Anyone can wire up an AI agent that produces confident output. The question a serious operator asks is different: does it pick the right tools, cite real evidence, stop when it should, and escalate the calls a human needs to make? This project exists to answer that question with numbers.
That is the same discipline I bring to a client’s automation — grounding, governance, and escalation designed in from the start, not bolted on after something goes wrong in production.
Want AI in your operation that you can actually trust?
The AI Readiness Assessment maps where AI fits your workflows — and where it does not. Flat $500, refund guarantee.