The tests I wrote were testing my imagination

Ava's decomposer had 62 hand-written tests, all passing. Two independent critics with fresh context then found twenty ways to plan the wrong thing, none of which those tests caught. That is not a coincidence, it is the structure of the problem: a test written by whoever wrote the code tests the cases that person thought of.

So we went looking for a corpus somebody else built.

Borrowing a dead competitor's schema

BrianKnows was a Web3 intent-recognition and execution engine that shut down around May 2025. The repositories are still public and MIT licensed, and the useful artifact is not the code, it is the SHAPE of the API they ran in production:

ExtractParametersResult.completion: Completion[]
Completion { action, token1, token2, chain, address, amount, dest_chain }

Two things in there are a verdict on Ava's original design. completion is an ARRAY, so Brian returned a list of actions per prompt from day one. Ava's intent type held exactly one, which is precisely why a request naming three actions collapsed into a single swap. And token1 alongside token2 says a swap has an input and an output token in the extraction schema itself. Ava carried one asset per leg, which is why "swap USDC to WETH then supply the WETH" planned a USDC supply.

Both were found independently in Ava and fixed before we read Brian's schema. Two systems converging on the same shape is decent evidence the shape belongs to the domain rather than to anyone's taste.

What the corpus caught on its first run

Running Brian's action vocabulary through Ava's decomposer failed twice immediately, and both failures were about the direction money moves:

"Borrow 100 USDC from Aave on base"   ->  planned a LEND leg
"Repay 100 USDC to Aave on base"      ->  planned a SWAP leg

The borrow one is the dangerous one. Ava would have SUPPLIED capital in response to a request to BORROW it, and the preview a user reads would have looked entirely reasonable. Both came from inference filling a gap: no rule matched, so the code guessed.

Operations Ava cannot execute are now refused by name, before any kind is inferred. That ordering is the fix, because inference is what produced the wrong kind.

Then the first real tool call found another

All of that work was still unreachable. Production MCP advertised eleven tools and none of them decomposed a request, so Claude, Cursor and Grok could not call any of it. A capability no client can discover is a capability nobody has.

Two tools went into the default MCP set, and the very first call through the real tools/call path found a bug eighty unit tests had missed: "then supply what arrives to Aave" was read as having no amount, so the third leg was silently dropped. "What arrives" is the same runtime relationship as "the proceeds"; the pattern list simply did not include it.

And then a text file found two more

The testbench is a plain text file of prompts and a runner that sends each line through the real MCP call. At the bottom is a section for sloppy, realistic sentences. It caught:

"put 200 usdc into the best base lending market"   ->  refused, no asset found

Ticker matching was [A-Z]{2,10}. Lowercase tickers were invisible, so every casual prompt failed. Real users type lowercase; all eighty of my tests used USDC.

It also showed that refusals, while correct, were useless: "no chain named and none inferable from the preceding leg" tells nobody what to type. Every refusal now names the fix and says why Ava will not choose for you.

The pattern

Four different external signals, five bugs, zero found by writing more self-authored tests. The order of value was: an independent critic reading the deployed behaviour, a schema from a system that ran in production, the real tool call, and a text file of sentences somebody would actually type.

Writing the sixty-third unit test was never going to find any of them.