Vortex Research · Field Notes · 2026-09
Which model actually premortems properly?
A premortem assumes the thing you’re about to build already failed, then works backward to find out why. It’s the cheapest hour you’ll ever spend, because it’s the last moment a fatal flaw is still a sentence in a document instead of a board you already paid to have made.
We gave the same spec to four AI models and asked each one what would kill it. Then we checked their work, got the check wrong, and nearly published the mistake.
The outlines are the models
read the current code
read the current code
read the older copy
read the older copy
The shading is how many agreed
How to read it. Every number counts problems, and sits in the zone covered by exactly the models that found it.Add up the numbers inside one outline to get that model’s total. Gemini 3.1 Pro found 3 + 1 + 1 + 1 = 6. The 3 is its alone. The rest it shared.27 distinct problems in 41 findings across four reports. 17 of them only one model found. Exactly one was caught by all four.
Why we run these
Normal review asks whether the plan is right, and normal review says yes. Of course it does. A plan is a document, and documents agree with themselves.
A premortem asks a harder question. It failed. What killed it?
One pass caught three things: a headline number our pipeline couldn’t actually produce, a safety limit written in one unit and tested in another, and a final check a program could pass by doing nothing at all.
None of those are typos. All three would have shipped green. A test suite tells you the code does what someone wrote it to do. Never that it was worth writing.
What we found
They barely overlapped
27 distinct problems between them. 17 were found by exactly one model. Only one was caught by all four. Pick any single model and you miss most of the list.
The best individual find came from the model that looked weakest on paper. Gemini 3.8 Flash spotted a sorting step that only looks at one thing, so anything tied keeps whatever order it arrived in, which quietly makes the whole result depend on the order of the input file. It cited the line exactly. Nobody else saw it.
Grok 4.6 filed only seven problems, and the most useful ones. Its best was not in the spec at all: our spec demands two measurements match exactly, and there is already a safety check in our own code that refuses to compare them that way, explaining why in a comment sitting right above it. Somebody had answered the question before we asked it.
Where we went wrong
We accused two of them of making things up
Every model cites line numbers in real files, so we opened the files and checked. Two reports looked like they were inventing evidence. Functions that weren’t there. Line numbers past the end of the file. Ticket numbers for work nobody did.
We had a draft of this page written around that. It was wrong.
✗ the command · exists
✗ the timeout behavior · exists
✗ both ticket numbers · real commits
✓ our accusation · wrong on every count
We had checked against an older copy of the repo, 24de19f7, while the models had read c63eff5c. One file was 387 lines in the copy we read and 744 in theirs. The bigger version landed two days before the runs. Everything we called invented had been added in that gap.
Re-run against the right code: 55 citations, none fabricated. All four models were accurate. The audit was the thing that was broken.
The part that should worry you
We had it confirmed three times
Catching a stale checkout is ordinary. What happened next is not.
Two more models were sent to verify the finding, and both agreed. We wrote that down as independent confirmation. It wasn’t. We had pointed all three at the same stale copy. One mistake, read three times, reported as three checks.
That is the failure worth carrying home. Not that a check can be wrong, but that a check which picks its own witnesses will agree with itself all day, and the agreement feels exactly like evidence.
And one more thing we got wrong
We were partly comparing checkouts, not models
Once we knew a stale copy existed, the obvious next question was who had read it. The answer was uncomfortable. Two of the four models had been pointed at the older copy, by us, because the spec names that location. The other two read the current branch.
You can prove it from their own citations. Two models cite line numbers that do not exist in the older copy. The other two cite that copy by name, dozens of times.
So some problems were findable by only half the field, and one model’s “verified” is another’s “that isn’t true any more.” The headline count above survives, but part of what reads as models disagreeing is two copies of a repository disagreeing, and we handed them out that way.
We are keeping the number and the caveat together rather than quietly dropping either. A cleaner run would give every model the same commit and say which one.
The debrief
What worked, what didn’t, and why
✓ Worked
- Running four. 17 of 27 problems were found by only one model. Any single model misses most of the list.
- Handing them the real code. The best finds were about what the code already does.
- Someone outside checking us. That is how this page got fixed before it shipped.
✗ Didn’t
- Our citation check. Right method, wrong copy of the code, confident output.
- Counting problems found. A short list beat a long one.
- Treating agreement as proof. Three checks, one source, no independence.
∴ Why
- A check only sees what it opened. Ours never said what that was.
- Stale copies look identical to current ones. Nothing warns you.
- Being wrong feels the same as being right. That is the whole problem.
Why this matters for what we build
We ship the checking, not the confidence
Put AI near work that’s regulated, or expensive to undo, and the question is never whether it sounds sure of itself. It always does. So do we. The question is whether anything you own catches it when it’s sure and wrong.
We publish this one because we failed it. The models did their job. Our verification didn’t, and nothing in our process caught that until someone outside looked. The fix wasn’t a better model. It was writing down what the check had actually read.
Talk to us about your buildLimits, plainly, largest first. The four runs did not read the same code, so this is not a clean model comparison and the overlap figure carries that caveat on its face. The two Gemini runs also came through a different tool than ours, which we cannot control for. One spec, one set of instructions, one run each. This is a report on four reports, not a ranking of models, and we would not draw one from a single spec. The citation re-audit records the exact commits, and so does this page: the current branch is c63eff5c, the older copy is 24de19f7, and the two branches part company at db7e8a90 on 2026-09-04. Each citation was re-checked in the tree its own model read, and the mapping from all 41 findings down to 27 distinct problems is written down and reconciled against each model’s own count, rather than living only in the figure. That de-duplication is a judgement call, not a measurement, and we say so there. File names, what the spec builds, and how we run these are left out on purpose.