At Fullscript, I worked on a back-office workflow where a steady stream of documents had to be reviewed and routed. Each document arrived with an Asana task, and it had to land in the right queue before anyone could act on it. A person opened every document, read enough to decide where it belonged, and moved it by hand. A single review could take hours.
We wanted an AI agent to do the first read. A background job sends the document to the model, and the model answers with a small JSON object we called the verdict: the queue it picked, and a short rationale explaining why. The job reads the verdict and moves the document. People only step in when something looks off.
The agent was great in chat and broke in the background job
While we built the prompt, we tested it in a chat window. The agent explained its reasoning, then ended with a neat JSON verdict. In chat, a person skims past "Sure! Here's my analysis…" and finds the object without thinking about it.
The background job can't skim. Once the agent ran on real traffic, documents started landing in the wrong queue, or in no queue at all. When we pulled the raw responses, the problems fell into six patterns:
- Prose wrapped around the JSON. A friendly preamble, the object, then a closing remark.
- Missing or inconsistent code fences. Sometimes fenced, sometimes bare, sometimes labeled as a different language.
- Leaked tool-call markup. The model "called" a tool in plain text, so the response held tool syntax instead of a verdict.
- Empty responses. Nothing at all, usually after an upstream timeout.
- Several objects. A draft verdict, then a corrected one.
- Truncation. The token limit cut the object off mid-string.
TL;DR: Once model output feeds code, treat it as untrusted input from an unreliable API. Request structure, extract it with a linear-time scanner, validate it against a schema, and fail loudly when any step says no.
A regular expression can't count braces
Our first fix was a regular expression that grabs everything from the first opening brace to the last closing brace. It looked reasonable, because most responses held exactly one object.
The patterns above broke it. When a response holds two objects, first-brace-to-last-brace captures both plus the prose between them, which isn't valid JSON. A lazy match stops at the first closing brace and cuts nested objects in half. A brace inside a string, like a rationale that quotes one, fools both. The root cause is theory, not tuning: balanced braces form a context-free language, and regular expressions can't count nesting depth.
Four layers, each catching what the last one missed
We replaced the single pattern with four small layers. Anything that fails goes to a review queue for people, the same place documents went before the agent existed. Follow a response to either a stored verdict or that review queue:
flowchart TD
A["Request structured output"] --> B{"Response empty?"}
B -- yes --> F["Fail loudly<br/>send to review queue"]
B -- no --> C["Extract first balanced object"]
C -- "none found" --> F
C --> D{"Matches schema?"}
D -- yes --> G["Store verdict and rationale"]
D -- no --> E{"Already retried?"}
E -- no --> A
E -- yes --> F
The only loop is a single retry that feeds the validation error back to the model.
1. Request structure. Most providers now accept a JSON Schema that the output has to follow. In Ruby, ruby_llm exposes this as with_schema. In JavaScript, the AI SDK uses Output.object. Structured output reduces failures but doesn't remove them: refusals, truncation, and fallback models without the feature still reach the parser.
2. Extract with a scanner, not a pattern. The extractor is a small state machine plus a depth counter, which makes it a one-counter automaton, the simplest machine that can recognize balanced braces. It reads each character once:
stateDiagram-v2 [*] --> Prose Prose --> Object: open brace, depth 1 Object --> Object: open brace adds 1, close brace subtracts 1 Object --> InString: quote InString --> Escaped: backslash Escaped --> InString: any character InString --> Object: quote Object --> Done: depth reaches 0 Done --> [*]
String and escape states keep braces inside the rationale from counting, and prose, fences, and leaked markup outside the object are skipped. There's no backtracking, so the scanner is O(n). We kept jsonrepair, which closes truncated objects, out of the verdict path: it makes up an ending the model never wrote.
3. Validate at the trust boundary. Valid JSON isn't a correct verdict. The model can return a queue that doesn't exist, skip the rationale, or spell a queue name three different ways. We defined one JSON Schema with an enum of allowed queues and used it both for the structured-output request and for validation, so the two can't drift apart. json_schemer validates JSON Schema in Ruby, and zod fills the same role in JavaScript.
4. Fail loudly. Each failure has its own error type: empty response, no object found, schema mismatch. Each one goes to the review queue and to a counter we can alert on. The documents that landed in no queue at all were exactly this failure, swallowed silently. Defaulting to a catch-all queue would have been worse, because it looks exactly like success.
Unreadable documents never reach the model
Some documents have no extractable text: scans without OCR, password-protected files, empty attachments. Given nothing to read, a model still returns a confident verdict. So the job checks for readable text before the call and sends unreadable documents straight to the review queue.
Each skipped call saves its full latency and token cost, and it removes one guaranteed hallucination. To size that gain on your own pipeline, multiply the number of jobs with empty extracted text by the average call latency and per-call cost.
Every verdict keeps its rationale and raw response
We stored the rationale, the raw response, and the model version with every verdict, so the reason for every move lives in the record, not in a rotated log.
Those stored responses also became our regression suite. Before shipping a new extractor or schema, we replay every stored response through the old and new versions and compare the verdicts. This is differential testing against real traffic. If the new version flips verdicts that people already confirmed, it isn't ready.
Each side effect got its own flag
Choosing a queue is safe. Acting on it isn't. The agent had two side effects: posting a summary of its verdict as a comment on the document's Asana task, where reviewers already worked, and moving the document to a queue. Each got its own feature flag. With both off, the agent runs in shadow mode: it records its verdict while people keep sorting by hand, and we compare the two. Follow a stored verdict through the flags:
flowchart LR
A["Verdict stored"] --> B{"Comments flag?"}
B -- on --> C["Post summary comment<br/>on Asana task"]
B -- off --> D{"Routing flag?"}
C --> D
D -- on --> E["Move to queue"]
D -- off --> F["Shadow mode<br/>compare with human decision"]
Flipper handles per-feature rollouts in Rails. We turned on the Asana comments first, since reviewers could see them and ignore a wrong one, and turned on routing once shadow-mode agreement was high enough that we stopped double-checking.
The team no longer opens every document to sort it. Documents arrive with a verdict and its rationale attached, and people spend their time on the exceptions the pipeline sends them.
The scanner stays linear where the regex went quadratic
Backtracking regex engines can go quadratic on the truncated responses from the incident. To measure it, I built an opening fragment (brace, quoted key, colon) repeated with no closing brace, at three sizes, and timed one match of the greedy pattern and one pass of the scanner:
| Input size | Greedy regex, Node 24 | Greedy regex, Ruby 3.4 | One-pass scanner, Node 24 |
|---|---|---|---|
| 50 KB | 195 ms | 0.7 ms | under 1 ms |
| 100 KB | 779 ms | 1.3 ms | under 1 ms |
| 200 KB | 3,154 ms | 3.8 ms | under 1 ms |
On Node 24 (V8), every doubling of the input roughly quadrupled the regex time. Ruby 3.4 stayed at a few milliseconds, because Ruby 3.2 added cache-based regex matching. The correctness problem remains on every runtime.
To check this yourself, build that unclosed fragment at 10k, 20k, and 40k repetitions and time one match at each size. If the time grows about 4× per doubling, the matcher is quadratic. A linear one grows about 2×.
Strict schemas reject some good verdicts
A strict schema rejects some responses a person would accept. That's intentional, but a sudden jump in rejections usually meant a model or prompt had changed. The retry doubles latency for the responses that need it, which is why it's capped at one.
The benchmark above covers only the extractor. This post has no production numbers for how often each layer fired, and I'd track that per layer from day one if I built it again.
Model output is input
Validate model output at the boundary, the same way you'd validate a request body from a client you don't control.