The AI in Your Basement Doesn't Phone Home
Photo: HP_Envy_X2.jpg original photo: Beau Giles from Sydenham, Sydney, Australia derivative work: Jbarta / Nagualdesign, CC BY 2.0, via Wikimedia Commons
Wes Coyle keeps a tower PC under his desk in Akron, Ohio, that hasn't touched the internet in three weeks. It isn't broken. It's running a language model, and Coyle — a contracts paralegal who moonlights reviewing lease agreements for a handful of small landlords — would just as soon it stayed ignorant of the outside world.
"I paste in a client's lease, ask it to flag the weird clauses, and nothing leaves the room," he says. "No account. No retention policy to read. No bill that changes because I asked too many questions."
Coyle is part of a growing crowd of Americans running open-weight AI on hardware they already own — laptops, gaming desktops, hand-me-down workstations, the odd home server humming in a closet. Not because they think a 30-billion-parameter model beats the frontier chatbots at everything. Because they'd rather own the thing that reads their documents.
Why bother when the cloud is right there
Fair question. The cloud chatbot is faster, smarter, and free-ish. The case for going local isn't that it's better — it's that it's yours. Four arguments come up over and over:
- The data stays put. Medical notes, client contracts, an unpublished manuscript, a kid's IEP draft. Once it's in a cloud prompt, you're trusting a retention policy you didn't write and a terms-of-service you didn't negotiate.
- It works when the pipe doesn't. Flights, cabins, long stretches of rural highway, or just the Tuesday your ISP decides to have opinions.
- The bill is fixed. Twenty bucks a month is cheap until you're pushing serious token volume through an API and the meter starts moving. Local inference costs electricity and nothing else.
- Nobody can pull the model out from under you. No deprecation notice, no rate limit at 2 a.m. before a deadline, no quiet change to what the model will and won't answer.
That last one is the sleeper. Ask anyone who built a workflow on a model that later got retired.
What actually runs on a normal machine
The tooling got boring, which is the highest compliment software can receive. Ollama wraps llama.cpp into a single command. LM Studio puts a friendly chat window over the same engine. Open WebUI gives you something that looks like the assistants you're used to, running against a model sitting on your own disk. If you like knobs, llama.cpp's server has plenty.
The models are the open-weight families — Llama, Qwen, Mistral, Gemma, and a long tail of fine-tunes and distillations built on top of them. Size matters more than brand. An 8B model at 4-bit quantization is roughly a 5GB file and runs on almost anything. A 32B at the same quantization wants around 20GB of video memory. Past that, you're building a hobby around it.
The hardware reality check
Here's where the fantasy meets the spec sheet:
- A 16GB laptop will run an 8B model at something like 8 to 15 tokens per second. Usable for drafting and summarizing. Not fast.
- Apple Silicon with 32 to 64GB of unified memory is the quiet winner of this whole scene. Memory bandwidth is high and the GPU can address most of it, so a 30B model at 4-bit lands in the genuinely pleasant zone.
- A 24GB GPU — the used RTX 3090 is the community's default answer — handles 32B quantized models comfortably and leaves headroom for a long context.
- A mountain of DDR5 and a midrange GPU doesn't help as much as the sticker suggests. Inference is memory-bandwidth-bound, so a 128GB box with a slow bus loses to a 64GB box with a fast one.
- Laptops throttle. A machine that hits 20 tokens per second for two minutes sits at 12 for the next twenty.
Nobody warns you that context eats memory too. A model that fits at 8K tokens may not fit at 32K, and the cache grows as the conversation does.
What you give up, honestly
Local models in 2026 are genuinely good at summarizing, classifying, rewriting, pulling structured data out of messes, and drafting things a human will edit anyway. They are not the frontier. Push a local 32B on a messy multi-step research task or an ugly refactor and you'll feel the gap — weaker tool use, shakier long-context reasoning, more confident wrongness.
You also inherit the maintenance. Quantization levels, prompt formats, context settings, model updates that quietly change behavior. It isn't hard, but it isn't zero either, and it's yours to do at 11 p.m. when something breaks.
And if you're chasing image or video generation at the quality people expect from the big services, budget for a lot more GPU than a text model wants.
Three setups people actually run
The laptop holdout. A three-year-old MacBook with 32GB, Ollama, and a handful of 8B and 14B models. Cost: nothing new. Handles email drafting, meeting notes, and document Q&A just fine.
The converted gaming rig. A used 3090 picked up for around $700, a fresh power supply, LM Studio. Runs the 32B class all day. Coyle's box is this, minus the network cable.
The closet server. A refurbished workstation with 128GB of RAM and no GPU, serving an Open WebUI instance to the whole house. Slow per-token, great for overnight batch work, and the household's "AI account" is now a URL on the local network.
The hybrid answer is the honest one
Almost everyone I talked to for this piece uses both. The rule is simple: sensitive or high-volume work stays local, the genuinely hard stuff goes to a frontier model, and nothing goes to the cloud that you wouldn't paste into a public forum.
That isn't a compromise of the principle. It's the principle working — deciding per task who gets to see your data, instead of handing the entire pile to one company because it was convenient in 2023.
The stack you own is the stack you can turn off. That's the whole pitch.