Vortex Research · Field Notes · 2026-09
Does the most capable model catch the most?
In our last note, four models premortemed one spec. The most useful thing we learned was about our own audit: we’d checked citations against a stale commit and nearly accused two models of making things up.
This is the follow-up. Three reviewers, one commit, one prompt. The two findings that mattered came from one of them, and it wasn’t the one billed as most capable.
22 distinct problems. Shaded: found by all three. Both high-severity problems are in Opus 5.5’s 8.
The setup
Same spec, same commit, same prompt
A premortem assumes the plan already failed and works backward to find out why.
The plan here was a draft spec for hardening a production Postgres database. Our staging rehearsal had been running PostgreSQL 16 while production ran 17.6, so a privilege that only exists in 17 slipped past every check we had. The spec proposed a version pin, a new check, and one migration to close the gap.
We gave it to three reviewers: Claude Fable 5.1, which Anthropic lists “for demanding reasoning and long-horizon agentic work”; Claude Opus 5.5, the current Opus as of September 2026; and Claude Opus 5, the one before it.
All three got an identical prompt. The only differences were the ticket label and the model line. All three ran at high effort.
All three read the same commit, and we recorded which one. That is the specific fix for last time’s mistake. Each had the same thirty-minute box, the same read-only access to the repository, and the same permission to run read-only catalog queries against production.
None saw another’s work. The spec was written by none of the reviewers, so no reviewer was grading its own writing.
What they found
Each one alone would’ve missed something
| Measure | Fable 5.1 | Opus 5.5 | Opus 5 |
|---|---|---|---|
| Verdict | Fix first | Fix first | Fix first |
| Problems found (of 22) | 10 | 17 | 10 |
| Found only by this model | 1 | 8 | 3 |
| High-severity caught (of 2) | 0 | 2 | 0 |
| Wall time | 9.6 min | 12.7 min | 7.9 min |
Together they surfaced 22 distinct problems. Only five were found by all three.
Those five were real and mostly mechanical. The draft failed one of the repository’s own documentation checks. A mode switch it relied on could never be passed through the path that runs against production. A proposed safety guard would’ve blocked the very tool you’d use to diagnose the problem it guards against. A temporary table would’ve polluted a static model of the database. And its list of checks blocked by a broken local tool needed revisiting.
The two that mattered
Both came from Opus 5.5 alone
The rollback would’ve widened access. The spec’s undo step granted the privilege back on every table. Opus 5.5 measured production and found four tables where the browser role never held it in the first place, one of them a table the system deliberately keeps away from the browser.
Running the “undo” would’ve left the database more open than before the change. A rollback is the step people run under pressure without rereading, which is what makes this one dangerous.
The main new check could never fail. It relied on a function that, in the local rehearsal environment, would report false no matter what. The rehearsal’s setup script creates the browser roles with inheritance switched off, with a comment saying this “matches the platform.”
Opus 5.5 queried production and found inheritance switched on. So the check would’ve passed in rehearsal forever. A second rehearsal-versus-production gap, the exact kind the spec was written to close, was sitting inside the fix.
The other two
Faster, sharper on facts, blind to the same two
Opus 5 was the quickest of the three and the sharpest at checking the spec’s own statements. It caught the spec blaming an earlier migration for a grant that actually comes from the platform’s defaults. It found a sanity floor set so low that twelve of the tables and views it guards could vanish unnoticed. And it named a catalog assumption the migration silently depends on. It missed both high-severity problems.
Fable 5.1 overlapped the others the most. Its one unique find was a script reading a table the spec said nothing outside one directory touches. It also missed both high-severity problems.
Method
What we did differently
Same tree
- All three reviewers read the same commit, and we recorded it.
Checked at the source
- We checked eleven of the twenty-two problems ourselves against the cited file and line, including both high-severity ones, Fable 5.1’s unique find, and the two of Opus 5’s that can be read from the repository. All eleven held. The rest are as reported.
Same effort
- All three ran at high effort, with the same prompt and the same thirty-minute box.
Where we landed
A panel, not a winner
We’re folding the union of all three lists into the spec rather than picking a winner.
Twenty-two problems from three reviewers, only five of them shared, and the two most dangerous from just one reviewer. That’s last time’s lesson from a different angle: a single reviewer, however capable, misses things the others catch. The new part is that the model billed as most capable wasn’t the one that found them.
Talk to us about your buildLimits, plainly. One spec: a single comparison, not a ranking. The previous hard catch in this same codebase, a design that died on a repository-wide rule nobody had opened, was Fable’s. Measuring isn’t knowing: both of Opus 5.5’s big finds came from querying production rather than reasoning about it, so some of the gap is a working style that happened to suit this spec. Severity is our grading: the two we call high-severity are the ones we judged most dangerous, and each reviewer labelled its own findings differently. Two harnesses: Fable 5.1 and Opus 5.5 ran as sub-agents inside one session; Opus 5 ran as a separate session with the same prompt, commit and read-only tools. Same family: an Opus-class agent wrote the spec, and two of the three reviewers are Opus models. De-duplication is judgment: where two reviewers described one problem from different sides, we counted it once.