The Model Was Never the Bottleneck

The Model Was Never the Bottleneck

For months I assumed the ceiling on my AI agents was the model. Swap in a better model, get better results — that was the whole theory. Then I read a benchmark this week that made me stop and rethink the entire stack I run at home.

Someone on r/PiCodingAgent ran the exact same model — DeepSeek V4 Flash, the very model I use in my own Hermes setup — through two different harnesses. One was minimal: just four tools, read, write, edit, and a shell. The other was a fork with thirty-one tools, a language server, a debugger, a browser, the works. Same thirty tasks, same verifier, same time limit.

The minimal harness won. Twenty out of thirty tasks passed, against seventeen for the tool-heavy fork. It was faster, cheaper, and burned fewer tokens. More tooling did not win. That single result landed harder than any model release I’ve tracked this year.

What I Actually Control

I run a lot of MCP servers on my own agent — home automation, search, email, a whole toolbox. Reading that benchmark, I had to ask myself a blunt question: how much of that surface is dead weight? The lesson isn’t “tools are bad.” It’s that the harness — how context gets built, how tools get routed, how the agent recovers from a wrong turn — does more of the work than I gave it credit for.

There’s a formula floating around the agent-engineering world right now: perceived quality equals model times harness times context. Same weights, different system prompt, different constraints — different result. I’ve seen it myself. The same model that wanders off and “improves” things I never asked for can stay perfectly on task with a tighter harness around it.

So the question I’m sitting with this week is simple: if the harness is the durable asset and the model is replaceable, which parts of my own stack am I actually building to last?

What’s your experience — have you ever gotten better results by removing tools rather than adding them? I’d genuinely like to know.