I Looked at My AI Stack and Asked What I Actually Own

A benchmark that made me ask a different question

Yesterday I read a benchmark that ran the same model, with the same prompt, on the same fourteen tasks — across three different runtimes. One of them finished with 63% fewer tokens and 30% lower cost at the identical solve rate. Not a better model. Not a better prompt. Just a different harness. And it made me stop and ask something I had been avoiding: if the runtime is what really controls the bill, then what do I actually own in my own AI stack?

I run my assistant on a model I picked deliberately — deepseek-v4-flash — because it is cheap and good enough. But the benchmark (a community test of fourteen cross-system tasks run across Claude Managed Agents, langchain deepagents, and a self-hosted open-source runtime) kept turning up the same result: the software around the model decides the cost more than the model itself. That runtime got eleven of fourteen tasks solved at 3.7M tokens and 19 tool calls, where a managed option needed 10.0M tokens and 32 tool calls for the same score. Same model, half the cost per success.

The part I keep forgetting to count

Here is the uncomfortable truth I sat with: I think of my stack as having three parts — the model, the context I feed it, and the tooling around them. I have spent months polishing the middle one. My knowledge lives in a plain-text vault I can read with any editor, my durable memory sits in a database I could query by hand, and my procedures are collected into repeatable skills. All of it is mine, portable, and inspectable. That is the asset I built.

And then the runtime — the harness that decides how my prompts are assembled and which tools get routed in — sits between me and the model, and I had treated it as a fixed part of the furniture. The benchmark says otherwise. The harness is exactly the part that changes the token bill by 63%, and it is also the part I do not actually own.

What portable actually costs

So I asked the practical version of the question: if I had to move my context layer onto a different runtime tomorrow, what would survive? The vault would — it is plain Markdown. The memory database would — it is a SQLite file. The procedures would — they are mostly text. What would not survive is the glue: the way my harness talks to my memory provider, the gateway that routes my messages, the specific skill-file format this platform expects. That glue is real, it is useful, and it is also the invisible tax I pay every month.

The honest lesson from the benchmark is not that I should switch runtimes today. It is that I should know which parts of my setup are load-bearing and which are rented. The context layer is mine. The harness is rented. And now that I know the rent can move the bill by almost two-thirds, I am going to start designing as if I might move.

Over to you

If you run an agent, have you ever separated the model from the software around it in your head? Do you know which parts of your stack you actually own, and which ones you would lose if you switched platforms overnight? I would genuinely like to hear how you answer that — because I only got an honest answer myself today.