The Anti-Hallucination System
Every agent platform eventually meets the same wall: a language model that says it did something it never did. It reports "Email sent" with no email in the outbox. It cites a source it never fetched. It calls a tool with a hallucinated recipient, a malformed amount, or arguments that quietly drop a required field. For a chatbot this is an annoyance. For an agent wired to a real inbox, a trading venue, or a customer's CRM, it is the whole risk. Melaya is built around a single conviction about that risk. An evaluation that detects a failure after the fact, especially an LLM judge that carries its own hallucination risk, is the weakest possible guarantee. So we do not lean on it. Fidelity comes from making the wrong outcome structurally impossible or deterministically caught, in this order: constrain → intercept → gate → repair → deterministically detect, with a fallible LLM judge placed last and never allowed to certify anything on its own.
This is defense in depth, and it is already built. Each layer is independent; anything that slips one layer is caught by the next. The layers run as runtime middleware around every tool of every agent (an onion model where the outermost wrapper runs first), plus framework hooks, so they apply to every model, every provider, and every template with zero prompting from the pipeline author. The author declares a tool and its schema; the framework does the rest. That placement is the entire point: enforcement lives in the runtime, not in a prompt, so it holds even when a model ignores its instructions. The hard guarantees are deterministic (schema checks, span checks, receipt regexes, and human-approval gates), while the fallible judge is fenced off from ever finalizing a write, a trade, a citation, or a grounding claim.
Contents: the weakest guarantee and why we don't rely on it, the middleware onion, each defensive layer in turn (input safety, constrain, gate, validate, reliability, provenance, the loop engine, and the deterministic forensic eval), the trading-safety overlay, a failure-mode map, the implemented component inventory, and the invariants that make the whole thing hold.
The weakest guarantee: why we don't rely on an LLM judge
An eval that detects failures after the fact, especially an LLM judge that has its own hallucination risk, is the weakest possible guarantee. Fidelity comes from making the wrong outcome structurally impossible or deterministically caught: constrain, intercept, gate, repair, then deterministically detect, with the LLM judge last and never self-certifying.
Most "AI reliability" stories end at a scorecard: run the agent, then ask a bigger model whether the output looks right. That is detection, and detection is the last thing you want to depend on, for two reasons. First, it happens after the side effect: by the time a judge reads the transcript, the email is already sent and the order is already placed. Second, the judge is itself a language model, so asking a hallucination-prone system to certify that another hallucination-prone system didn't hallucinate is circular, and it fails silently exactly when both models share the same blind spot.
Melaya inverts the priority. Everything that can be a deterministic check is one, and it runs before the irreversible action wherever possible. A required field is enforced by a schema validator, not a judge. "Did the write actually happen" is answered by a span in the trace, not an opinion. "Is this citation grounded" is answered by whether the source was fetched this run and is locatable in the trace, not by vibes. The LLM judge still exists, and it is genuinely useful for fuzzy quality signals, but it runs last, it never PASSes a trading, grounding, citation, or write-happened verdict, and it can never finalize an output by itself. Prevention outranks detection, and determinism outranks the model. Every layer below is an application of those two rules.
Defense in depth: AI agent guardrails as a middleware onion
The layers are registered as toolkit middleware in a strict onion. Middleware registered last runs first, so the ordering is deliberate: schema validation is the outermost wrapper and executes before anything else touches a tool call, then the human-approval gate, then the reliability wrapper, and only then does the tool actually run. After the tool returns, an input-safety pass scans its output, the runtime records the call and its result as spans and memory, and the run's result flows into the loop engine and the deterministic forensic evaluator. Because this lives in the runtime rather than in any single agent's prompt, it applies uniformly: the same guarantees wrap a Claude agent, a local Llama agent, and a GPT agent, with no per-template wiring.
Read top to bottom, a single tool call travels this path:
The diagram above shows how the layers run (registered last executes first); the numbers themselves are conceptual labels, not a running order. The sections that follow take each layer in turn in number order, Layer 0 through Layer 7, so the catalogue is easy to scan. The flow above remains the reference for how they actually fire at runtime.
Layer 0: Input safety
Where most of these layers protect the world from the model, this one protects the model from the world. When a tool pulls in untrusted content, such as a fetched web page, a scraped document, or an external API payload, that content can carry a prompt-injection attack designed to hijack the agent's next move. The input-safety pass wraps tool output and scans it for injection patterns before the model ever reads it.
Crucially, it is best-effort and non-destructive: it annotates a suspicious result rather than deleting it, so a false positive can never silently break a legitimate tool result. It raises the model's guard against adversarial content without becoming a new single point of failure, the same fail-open-on-our-own-bugs discipline the rest of the system follows.
- An injection scan wrapped around every tool result as postprocess middleware
- Untrusted web, document, and API output flagged before it reaches the model
- Annotates rather than deletes, and fails open so a false positive never breaks a run
Layer 1: Constrain the model
The strongest layer is the one that shrinks the space in which the model can free-form in the first place. Prevention beats every downstream cure, so before any validation or gating, Melaya constrains what the model is even able to produce.
It starts with authored tool schemas. Every tool ships a JSON Schema for its parameters straight from the registry: types, required fields, and, where the author tightens them, enumerations, patterns, and a closed additionalProperties. That schema is the contract the later layers enforce; a tool without a crisp schema is a tool without guarantees, so the platform makes the schema the primary authoring surface.
Where the provider supports it, that schema becomes constrained decoding. Local models running through Ollama, LM Studio, or vLLM receive schema and grammar constraints directly in the request, and the Claude Code CLI path is pinned to JSON output. Anthropic's hosted tool use is schema-guided rather than schema-guaranteed, and its ReAct loop is steered toward finishing through tool-choice controls. But wherever a decoder cannot guarantee the shape, the next layer guarantees it after the fact, provider-agnostically. The result is that "the model emitted a well-formed call" is a property of the runtime, not a hope about a given vendor.
Two further constraints close common gaps. A forced-gate mechanism re-adds designated high-risk tools to the human-approval set even when a template author forgot to tick them, so the safety gate cannot be silently dropped by omission. And a Claude Code identity gate sends the exact system-prompt preamble the subscription OAuth path requires; without it, that path soft-blocks the strongest models with a misleading rate-limit error that carries no rate-limit headers at all. Both are the same idea: make the safe configuration the default that the framework enforces, not a checkbox a human has to remember.
- A JSON Schema authored per tool from the registry: types, required fields, and optional enum, pattern, and closed additionalProperties
- Constrained decoding for local models (Ollama, LM Studio, vLLM) and JSON output on the Claude Code CLI path
- A forced-HITL auto-gate that re-adds designated high-risk tools even when an author forgot to tick them
- A Claude Code identity gate that sends the exact system preamble the subscription OAuth path requires
Layer 2: Human-in-the-loop
This is the layer that most directly means "respect the user's will." Any tool that performs a consequential external action can be placed under a human-in-the-loop gate. When a gated tool is called, the run pauses, an approval card is emitted, and execution blocks until a human makes an explicit decision. Three implemented invariants turn this from a nicety into a ground-truth guarantee.
First, the action can only happen through the gate. There is no path that executes a write around the onion. The rescue path that repairs malformed tool blocks explicitly refuses to dispatch write-family tools: it patches the block so that the gate and the audit trail still see the intended action, but it never fires the write itself. A hallucinated or malformed write cannot sneak in through the error-recovery machinery.
Second, empty or garbage gated calls fail closed. A gated call that arrives with no arguments cannot slip through as a harmless no-op; it is rejected rather than executed. The gate treats an absent payload as a reason to stop, not a reason to shrug.
Third, the card shows the exact arguments and lets the operator edit them. For editable tools, the human sees the real recipient, subject, body, or amount and can correct a hallucinated value before the tool fires, including on individual rows of a batched action. The approval is not a vague "allow this?"; it is a precise, last-mile edit of exactly what will happen. That combination of pause, exact arguments, human edit, and fail-closed on garbage is what makes the gate a guarantee rather than a suggestion.
- Per-agent approval middleware that pauses any gated tool and blocks on an explicit human decision
- A rescue path that refuses to dispatch write-family tools, so no write can fire around the onion
- Empty-argument gated calls that fail closed
- An approval card that shows the exact arguments with per-field editing, including individual batch rows
Layer 3: Validate, coerce, repair
Schema validation is registered last and therefore runs first, before the human-approval gate ever sees a call. On every tool invocation it checks the model's arguments against the tool's authored schema and takes one of three paths.
When the call is valid, it still gets cleaned: stringified numbers and booleans are coerced to their real types, declared defaults are filled in, and unknown keys are dropped when the schema forbids extras. The corrected arguments are written back so that both the approval card and the tool handler see the same clean shape, and the human reviews exactly what will execute.
When the call is invalid, the tool is not executed and not gated. Instead, an actionable error is returned as the tool result, for example "'subject' is a required property; re-call with all required fields." Because this rides back through the ReAct loop as a normal tool result, the model repairs itself on its next turn rather than crashing the run. The repair loop is bounded: after a few attempts the message escalates rather than spinning forever.
And when the validator itself is broken, say a missing dependency or a malformed schema, the layer fails open. Our own bug must never break a customer's run. This is the deliberate asymmetry that recurs throughout the system: fail open on our faults, fail closed on the agent's. The net effect is that the "schema-guided but not schema-guaranteed" gap, most visible with hosted Anthropic tool use, is closed for every provider. A malformed, incomplete, or hallucinated-shape call is caught and repaired here instead of executing with junk arguments.
- Schema-validation middleware registered outermost, so it runs before the human-approval gate
- Valid calls coerced and cleaned (types fixed, defaults filled, unknown keys dropped), with clean args written back
- Invalid calls blocked and returned as an actionable repair error the model fixes on its next turn
- A broken validator that fails open so our own bug never breaks a run
Layer 4: Reliability
Not every failure is a hallucination; some are just the messy physical world. A webhook times out, an API returns a transient 503, a downstream service briefly refuses connections. Without a reliability layer, those transient faults get read by the model as results, so the agent "learns" that the tool returned nothing and reasons on from a false premise.
The reliability wrapper prevents that by classifying failures and handling them deterministically: per-tool timeout caps, retries that distinguish retryable from terminal errors, and a circuit breaker that short-circuits a dependency that is clearly down. Structured logging around every call makes the behavior auditable. A flaky call is retried or cleanly short-circuited so it never masquerades as a bad answer, and the idempotency discipline described under the loop engine ensures a retry cannot double-execute a side effect.
- Per-tool timeout caps and retries that classify retryable versus terminal failures
- A circuit breaker that short-circuits a dependency that is clearly down
- Structured logging around every tool call
- An idempotency key reused across retries so a side effect never double-fires
Layer 5: Provenance and observability
The deterministic layers only work because the runtime never has to take the model's word for what happened. As each tool call executes, the framework records it as ground truth: a span in the run's trace that captures the call, its arguments, its result, and its timing, plus an entry in the agent's working memory that captures the tools actually called, the results they returned, and the tools that were granted. Cloud runs stream these spans through the tracing exporter, and local-runner runs record them the same way. This provenance is the connective tissue between doing and checking.
It is what turns the platform's hardest claims into deterministic questions. "Did the write actually happen?" is answered by whether a send span exists, not by reading the reply. "Is this citation grounded?" is answered by whether the cited source was fetched this run and is locatable in the trace. "Did the agent use the tool it said it used?" is answered by comparing the granted set to the called set. Without a faithful record, every one of those collapses back into asking a model to grade a model. Layer 5 makes the record trustworthy, and the loop engine and forensic eval that follow read from it rather than from the model's prose.
- Every call, its arguments, result, and timing recorded as spans in the run trace
- The tools called, results returned, and tools granted lifted into the agent's working memory
- Cloud spans streamed through the tracing exporter, with local-runner runs recorded identically
Layer 6: The loop engine
Everything above operates on a single tool call. The loop engine governs the attempt, the agent's whole turn, and decides what to do when the turn didn't actually accomplish the goal. For each attempt it builds an evaluation context from the agent's real memory: the tool results it produced, the tools it actually called, and the tools it was granted. It runs a deterministic evaluator over that context, classifies the verdict into a closed set (finalize, retry, retrieve, ask-human, or halt, paired with a typed failure kind), and, in enforce mode, revises the input rather than simply re-asking the same question.
That "revise by changing the input" detail matters. A naive retry loop re-sends the identical prompt and gets the identical failure; the loop engine instead feeds a typed revision note back into the next attempt so the model has something new to work with. And every revision reuses an idempotency key, so a retry can never double-execute a side effect: no double email, no duplicate order.
The engine has three modes, and the first is a promise about restraint:
- Off. Byte-for-byte identical to the un-looped generated code. Turning the safety system off leaves nothing behind; the parity guarantee means there is no hidden behavior tax.
- Observe-only. Deterministic scoring plus a signal on the evaluation dashboard, with no retries and outputs left unchanged. You get the visibility without any change to behavior.
- Enforce. Retries and escalation on a non-finalize verdict, revising the input each time.
- An evaluate, classify, revise cycle run over the agent's real memory each attempt
- A closed verdict set (finalize, retry, retrieve, ask-human, halt) paired with a typed failure kind
- Three modes: off (byte-identical parity), observe-only, and enforce
- Enforce that revises the input rather than re-asking, reusing the idempotency key
Layer 7: Deterministic forensic eval
The final layer is where "the agent claimed something it didn't do" becomes a deterministic catch rather than a judgment call. It runs a composite evaluator, deterministic-first, in a fixed order: non-empty, schema, required sections, tool success, then a set of tool-forensics, then optional grounding and trading-safety gates, and only then, last, an LLM judge. The forensic checks use only data the runtime already lifts from the trace and toolkit, with zero user input, which is what makes them trustworthy.
- Coverage flags a granted delivery or write tool that was never called this attempt. This is exactly the "Operator said Email sent but never called the send tool" case: the send capability was granted, no send span exists, so the claim is flagged.
- Contract flags a write tool that returns no confirmation token (no id, no ok, no message id) as a possible silent no-op. The write "ran" but delivered nothing, and the missing receipt is a deterministic tell.
- False-fallback catches reply prose that claims failure while a tool actually returned data, and drives a hard retry rather than shipping a defeatist non-answer.
- Stall detection catches "asked instead of acted," an agent that punts a question back to the user when it had the means to proceed.
- A result classifier normalizes auth, error, empty, oversized, and bad-argument outcomes across every tool so the checks above run uniformly regardless of which tool produced the result.
The LLM judge runs last, is non-finalizing, and never PASSes trading, grounding, citation, or write-happened verdicts; those are deterministic gates only. In observe-only mode these signals surface on the evaluation dashboard as a soft score penalty; in enforce mode they drive a retry with a typed revision note. And because a granted tool is not a mandated tool, coverage is scored rather than blindly failed, so an agent that legitimately needed no tools, or only optional ones, is never mis-flagged. Grant is not requirement, so "was it used" is a signal, never an accusation.
- A deterministic-first composite: non-empty, schema, required sections, then tool success
- Coverage, contract (confirmation-token), false-fallback, stall, and result-classifier checks
- Grounding and trading-safety gates that are deterministic; the LLM judge runs last and never PASSes
- Signals surfaced on the eval dashboard in observe-only, or driving a typed retry in enforce
Trading safety: the domain overlay
Trading is where a hallucinated action is most expensive, so it gets a domain overlay on top of the general layers, but the overlay is the same pattern, not a different one. Orders route through ten safety rails, with dry-run and paper-soak gating before anything touches a live venue, human approval on every order, and an idempotent client order id so a retry can never double-place. The whole path is audited and replayable. The general shape (dry-run first, then live-with-human-approval, deterministic gates throughout, and a judge that never certifies the consequential verdict) is exactly the shape the layers above apply to every tool. Trading just turns the dials to their strictest setting.
- Ten safety rails on every order
- Dry-run and paper-soak gating before anything touches a live venue
- Human approval on every order and an idempotent client order id
- A fully audited and replayable path
AI agent failure modes and the layer that catches each
Defense in depth is only credible if you can name the failure and point at the mechanism that stops it. Each row below is a real failure class and the implemented layer that catches it.
| Failure | Example | Caught by |
|---|---|---|
| Malformed, incomplete, or hallucinated-shape arguments | An oversized send body truncates the tool JSON | Layer 3 schema validate and repair; the call is blocked, a repair error is returned, and junk never executes |
| Wrong-but-schema-valid value | A recipient off the allow-list | Layer 1 tight schema (enum/pattern) enforced by Layer 3, plus the Layer 2 operator edit before firing |
| False completion, prose claims a side effect that never fired | "Email sent" with no send span in the trace | Layer 7 coverage plus contract checks, surfaced in observe-only, retried in enforce |
| Silent no-op, the write ran but delivered nothing | A send-family tool returns no id | Layer 7 contract check (no confirmation token) |
| Write attempted outside the onion | The rescue path tries to dispatch a write | Layer 2 onion invariant: write-family rescue is refused |
| Gated call with empty arguments | A garbage or empty gated invocation | Layer 2 fail-closed on empty args |
| Model soft-blocked, mistaken for a rate limit | A strong model returns a bogus rate-limit error with no rate headers | Layer 1 Claude Code identity gate |
| Fabricated citation, facts with no fetched source | The agent invents a source | Layer 7 deterministic grounding: the source must be fetched this run and locatable in the trace |
| Double execution, a retry re-sends | Re-sending an email or re-placing an order | Layer 6 idempotency key reused across attempts |
| Prompt injection, tool output hijacks the model | A malicious web page | Layer 0 input safety |
| Transient I/O read as a bad result | A webhook timeout | Layer 4 reliability: retry, timeout, circuit breaker |
The implemented component inventory
For readers who want the map from concept to code, every layer above corresponds to a real, shipping component. Nothing in this post is aspirational; the table is an inventory of what runs today.
| Layer | Component |
|---|---|
| Layer 0 · input safety | Injection scan wrapped around tool output |
| Layer 1 · tool schemas | Authored parameter schemas from the tool registry |
| Layer 1 · constrained decoding | Local schema/grammar constraints; CLI JSON output |
| Layer 1 · forced gating | Auto-gate for designated high-risk tools |
| Layer 1 · Claude Code identity gate | Required system-prompt preamble on the OAuth path |
| Layer 2 · human-in-the-loop gate | Per-agent approval middleware that blocks on an explicit decision |
| Layer 2 · fail-closed empty-arg gate | Empty gated calls rejected, never executed |
| Layer 2 · onion write-safety | The rescue path refuses write-family dispatch |
| Layer 2 · arg-editable approval card | Exact-argument review with per-field editing |
| Layer 3 · schema validate and repair | Validate, coerce, and return actionable repair errors |
| Layer 4 · reliability | Logging, timeouts, classified retries, circuit breaker |
| Layer 5 · provenance and observability | Span trace plus memory lift of every call, result, and grant |
| Layer 6 · loop engine | Evaluate, classify, revise; modes off / observe-only / enforce |
| Layer 7 · forensic eval | Coverage, contract, false-fallback, and result classification |
| Layer 7 · eval dashboard | Run-quality and forensic signals surfaced in the product |
| Trading overlay | Ten rails, dry-run, per-order approval, idempotent order id |
Design invariants: why the system holds
Strip away the specifics and five invariants explain why the whole system holds together.
- Deterministic beats LLM. Every gate that can be a schema, span, or receipt-regex check is one. The LLM judge is last, non-finalizing, and never self-certifies a write, a trade, grounding, or a citation.
- Enforcement is the framework's job, not the prompt's. The author declares a tool and its schema; the onion validates, coerces, gates, and repairs. There is no prompt engineering to get right, and the guarantees hold for any model.
- Grant is not requirement. Because a tool is granted, not mandated, "was it used" is a scored forensic signal rather than a blind assertion, so a zero-tool or optional-tool agent is never mis-flagged.
- Fail open on our bugs, fail closed on the agent's. A broken validator never breaks a run; a malformed or ungated agent action never executes. The asymmetry is deliberate and consistent.
- Prevention before detection. Constrain what the agent can do (tight schemas, validated and coerced arguments, forced gating) before relying on catching what it did.
Where this goes next
The honest framing is that this is a living system, not a finished monument. The layers are built and running today, and the deterministic-first philosophy is not going to change: detection stays last, the judge stays fenced off from consequential verdicts, and prevention keeps outranking cure. What evolves is coverage: more authored schemas tightened with enumerations and patterns, more delivery and write tools registered into the coverage and contract checks, and more domain overlays that turn the same pattern to its strictest setting for the highest-stakes actions.
There is an honest limit worth stating plainly, because it marks the boundary of what any orchestration layer can do. Everything described here is enforcement around the model, not intelligence inside it. Melaya can guarantee that a malformed call never executes, that a consequential action passes a human, that a claim with no matching span is caught, and that every step is recorded and replayable. What Melaya cannot do is make the model itself better at choosing the right tool, filling in the right arguments, or knowing when to act at all. That competence is the model's native tool-calling reliability. It is set upstream by the providers, and it is exactly the number their own benchmarks keep chasing.
It helps to picture two independent axes. The orchestration layer owns the floor: whichever model runs, nothing wrong fires silently and nothing false slips through undetected. The model owns the ceiling: how often the agent gets it right on the first try, with no repair loop, no gate intervention, and no retry. We raise the floor to the level of hard guarantees; the providers raise the ceiling as their tool-calling gets smarter. The two compound rather than compete. A weaker model simply spends more time in our repair and gate paths, which is the whole point, because that is how a silent wrong action becomes visible, correctable friction. A stronger model spends less time there, and the same guarantees cost almost nothing.
Even a model with perfect tool-calling still cannot prove to you that it sent the email, cited a real source, or placed the order it claimed. Capability is not evidence. The gates, the human approval, the span-backed grounding, and the audit trail are what turn a capable agent into an accountable one, and that holds at every point on the model curve.
The reason we build this way is simple. An agent that can take real action is only as trustworthy as the weakest guarantee standing between its confident prose and your systems. If that guarantee is "a bigger model checked the transcript afterward," you have bought very little. If it is "the wrong outcome was structurally impossible, or deterministically caught before it fired," you have bought the thing that actually matters. That is the system Melaya runs on every tool of every agent, for every model, with nothing asked of the author but a tool and its schema.
