Vortex Computation Logo

Vortex Research · Field Notes · 2026-07

Does telling a model to be “careful” make it think harder?

We wanted to know if framing a task as “careful, meticulous” vs. “breezy, casual” changes how deeply a model actually reasons. So we ran 140 blind agents across two runs and two models. The accuracy score said nothing. Measuring the model’s hidden reasoning found a real effect — and then a follow-up traced it to the frame’s instruction, not its tone. A clean answer, reached cheaply, and an honest NO-GO on the idea we started with.

140 blind agents · 2 runs2 models · one small, one frontiermetric pivot · accuracy → reasoning tokensverdict · NO-GO on “priming”

Why we publish this

This is the rigor we bring to building systems for regulated industries: we form a hypothesis, pre-register the condition that would kill it, test it against blind agents on real frontier models, and ship only what survives. Publishing the null results is part of the discipline — a technique that doesn’t hold up shouldn’t make it into your product just because it sounded good.

The finding

A ~6% “deliberate-more” gap, replicated across two models

Mean reasoning tokens per arm on the task battery that finally produced variance. The care frame spent more than the casual frame — same direction, near-identical size — on both a small model and a frontier one. Output was clamped to answers only, so the extra spend is hidden reasoning, not longer answers.

Small model · trap batterycare − casual +5.9%

▲ axis zoomed to show the gap · accuracy identical (83.3%) in all three arms

Frontier model · trap batterycare − casual +5.7%

▲ axis zoomed to show the gap · accuracy ceiling (100%) in all three arms

Directional, not yet significant. And a catch: the care frame literally contains the word “careful,” so this might be soft instruction-following, not the connotation we set out to test. Disambiguating that was the whole point of Run 2.

Why the metric choice was everything

One instrument was blind. The other saw.

The obvious thing to measure is accuracy. On a competent model, accuracy was dead flat — identical to the decimal across every frame. There was simply no room for the score to move. The model’s own reasoning spend — free to capture, continuous — held a signal accuracy could never show.

Accuracy — the obvious metric

▪ DEAD FLAT — identical to the decimal. The score has no room to move.

Reasoning tokens — the free continuous metric

▪ LIVE — retains variance even when accuracy is pinned to the ceiling.

Getting there took work: standard reasoning puzzles — even hard, novel ones — all ceilinged. It took a purpose-built battery we developed in-house to open a measurable band at all — and even then only on the smaller model.

Run 2 · the attribution test

Purify the frame, and the effect evaporates

Forty more agents on the frontier model. The registered care frame spent more than 2× the reasoning of casual — a huge gap. But move the identical care-words off the solver’s own reasoning and onto a neutral third party (describing someone else’s meticulous habits), and the gap vanishes. Only frames that say something about how to do the task move the needle. Tone alone does nothing.

Mean output tokens per frame · frontier modelregistered gap +117%

▲ accuracy ceilinged (95.8–100%) in every cell — the entire signal is deliberation

registered care − casual = +3,148purified care − casual = −427 ≈ 0▪ the three center frames = the baseline cluster

The debrief

What worked, what didn’t, and why

✓ Worked

  • Measuring reasoning, not just answers — a free, continuous signal accuracy was blind to.
  • A purpose-built trap battery — an item design we developed broke a ceiling that off-the-shelf hard puzzles couldn’t.
  • Cross-model replication — +5.9% / +5.7% is not one model’s fluke.

✕ Didn’t

  • Accuracy as a depth proxy — saturated everywhere; flat where it wasn’t at ceiling.
  • “Use a smaller model” — the small model matched the frontier one; that didn’t open a band.
  • The connotation hypothesis — purified of instruction, tone moved nothing.

∴ Why

  • Models deliberate by default — there’s no “fast-wrong” error for a frame to fix.
  • Competence > framing — the frame shifts effort; these models were going to be right anyway.
  • Meaning > mood — an instruction about the task is a lever; a vibe is not.

Why this matters for what we build

If you’re deploying AI into a regulated workflow, the difference between a technique that sounds right and one that holds up under a blind test is the difference between a demo and a system you can stand behind. This is how we decide what goes into yours: measured, adversarially checked, and honest about the null results. Field notes like this are how we show our work.

Talk to us about your build

Vortex Computation · Applied-AI research · vortex.llc/research