Home / Vault & Second Brain / The Wrong Question to Ask About Local AI Models

The Wrong Question to Ask About Local AI Models

The Wrong Question to Ask About Local AI Models

The Wrong Question to Ask About Local AI Models

Last week I watched a guide about choosing a local AI model, and one line stopped me cold: “What is the best local AI model” is the wrong question. The only question that matters is what is the best model for the machine you actually own. I’ve lived that mistake for months without realizing it.

I run everything through Ollama in the cloud — no GPU in the house, just a couple of small laptops and my trusty server. My workhorse is deepseek-v4-flash, with a vision model on the side. It works. But every time I read about some new 100-billion-parameter model making headlines, I get the itch. Surely I’m missing out, right? Surely I should be running something bigger.

Why Bigger Isn’t Better

The guide laid out exactly what happens when you ignore your hardware: you download a model that doesn’t fit your VRAM, and your laptop turns into a frozen brick. First token arrives “next Tuesday,” as the video joked. I’ve been there — the temptation to grab the latest flagship model and squeeze it onto hardware that can’t breathe.

The real lesson is about quantisation. The right amount of quantisation often beats a “better” model that doesn’t fit. A smaller, well-quantised model that responds in seconds beats a bigger model that makes you wait. That’s the trade I should have been making all along: speed and reliability over benchmark numbers I’ll never feel.


Choose the Tool You Can Actually Use

For my setup, that means staying with my small, fast models and treating the big ones as occasional experiments. My agents need to respond quickly, run on my schedule, and not cost me a fortune. The video’s hardware-to-model tables are a useful reference for the next time I upgrade, but they also validated what I already had right.

Here’s the question I want to leave you with: are you running the model your hardware can actually handle, or the model the hype told you to run? Sometimes the smartest upgrade isn’t a bigger model — it’s learning to love the one that fits.

Wat vond je van dit bericht?

Leave a Reply

Your email address will not be published. Required fields are marked *