Want to Run AI Agents Locally? The One Number That Decides Everything

Most people build a machine for local AI the way they would build a gaming PC. Faster processor, bigger graphics card, more power. Then they wonder why it runs like treacle.

There is one number on your graphics card that matters more than everything else combined, and it is not the one you think. It is not the chip. It is the memory. Get that number right and a modest, affordable machine runs capable models smoothly. Get it wrong and a expensive one crawls. This is the plain guide to getting it right.

Why run models locally at all

Frontier cloud models are extraordinary, and for the hardest reasoning you should use them. But for a lot of real work, running models on hardware you own beats the cloud on three things that matter.

**Privacy.** Your data never leaves the machine. No prompts sitting on someone's servers, no training on your secrets, no API logs of your private business. For a law firm, a health provider, a bank, or anyone under the New Zealand Privacy Act or the Australian equivalent, that is not a nice-to-have, it is the whole conversation.

**Cost.** You pay once for the hardware. Every conversation after that is free. No token meter running, no surprise bill the month a workflow goes wide.

**Uptime and ownership.** The internet drops, a vendor changes its terms, a model gets deprecated. Your local model does not care. It runs on your schedule, and it is yours.

The kitchen, because one analogy makes this click

Picture your setup as a restaurant kitchen. The graphics card is the chef: it does the heavy maths, and its speed is how fast the chef's hands move. The memory on that card, the VRAM, is the size of the counter the chef works on.

Here is the part nobody leads with: the counter size matters more than the hand speed.

A model is a giant recipe. When you see a model labelled "7B", that is seven billion tiny instructions telling the AI how to think. More instructions, smarter model, bigger recipe, more counter it takes up. The whole recipe has to sit on the counter while the chef works. If it fits, the chef works at full speed. The second the recipe is too big for the counter, the chef starts running to the back storeroom for ingredients, and that storeroom, your computer's ordinary system memory, is far slower. Throughput falls off a cliff: from a comfortable forty words a second to two or three, which is unusable.

That is the whole game. Not "how fast is my chef". "How big is my counter".

Quantisation: fitting a big recipe on a normal counter

So how do people run these big recipes on ordinary hardware? They compress them. Instead of writing every instruction out in full, you shrink it down, and the common setting is 4-bit quantisation: the same recipe in a smaller notebook. You lose a sliver of precision and it fits in roughly a quarter of the space. For almost all everyday work you will not notice the loss, and you will very much notice the model actually fitting.

Here is the cheat sheet, for a 4-bit-quantised model, before any conversation is loaded:

- **7 to 8 billion** instructions: about **5 GB** of memory

- **14 billion**: about **10 GB**

- **32 billion**: about **20 GB**

- **70 billion**: about **40 GB**

And that is just the recipe sitting there. The chef has not started cooking. The moment a conversation begins, its memory starts growing too, like dishes piling up on the counter beside the recipe. Longer conversation, more dishes, less room. This is why a model that loads perfectly can still crawl twenty minutes into a chat: the counter ran out of space. Size the memory for the model plus a decent conversation, not just the model.

The three tiers that are actually worth building

You do not need a five-thousand-dollar setup because a Reddit post said so, and you should not cheap out on the counter to save a few hundred. Here are the three real tiers. Prices are a rough guide in US dollars.

Tier one: the affordable start (around US$1,200 to $1,500)

One graphics card with **16 GB of VRAM** is the core. Not the 8 GB version of the same card, which is the trap: 8 GB fills the moment you load a real model and the conversation spills immediately. Around it, a mid-range processor (it matters far less than you think for this), 64 GB of system memory as the back storeroom for overflow, a 2 TB SSD as the pantry where models are stored, a decent power supply and good airflow.

What it runs: 7 and 8 billion models very comfortably. These are not toys. They handle a real coding assistant, document summaries, private chat and light agent loops. You can push a 14 billion model with some trade-offs: shorter conversations before the counter fills, slightly slower output, still usable.

If you are already in the Apple world, a MacBook Pro or Mac Mini with 16 GB of unified memory lands in the same tier. Apple is different in a useful way: there is no separate storeroom, the graphics and the main system share one pool, so all 16 GB is usable counter. It is a touch slower than a dedicated Nvidia card, but the simplicity is hard to beat.

Tier two: the sweet spot for serious work

This is where most teams doing real local work should aim. Two paths.

Path one: a current-generation card with **16 GB** and faster "chef hands", better for agent-style loops where the model thinks, acts, and thinks again.

Path two, the move a lot of experienced local users make: a used **RTX 3090**. It is an older card, but remember the rule, the card barely matters, the VRAM is everything. The 3090 has **24 GB**. At 24 GB you can run a 32 billion model quantised and still keep room for a long conversation. Models in this class start genuinely rivalling cloud quality for most everyday use. On the Apple side, a Mac Mini M4 Pro with 64 GB of unified memory gets you here too, loading a 32 billion model with breathing room, quietly, sipping power, at a comfortable ten to twelve words a second.

A quick word on that measure: output speed is quoted in tokens per second, roughly words per second. Ten to fifteen feels like a person typing fast. Thirty-plus feels instant. Eleven to twelve is a comfortable working pace, not blazing, perfectly usable.

Tier three: high end, only if the workflow demands it

Do not buy this because it sounds cool. The centrepiece is a top consumer card with **24 GB**, a high-end processor, 128 GB of system memory for overflow, and a serious power supply. It runs 32 billion models like butter, with long conversations and complex agent chains. You can experiment with 70 billion models at heavy compression, though you will be covering the whole counter with recipe pages and paying for it in conversation length.

The Apple equivalent is a Mac Studio with 96 GB of unified memory. That much shared counter loads several models at once, a reasoning model, an embedding model, a coding model, all sitting there together, and it idles under 100 watts while a high-end desktop card draws five to ten times that under load. Power and heat are real line items at this tier.

Above all this sits the boss level: 32 GB cards, dual-GPU rigs at 64 GB-plus, ten thousand dollars and up. That is for people training their own models, not running them. Almost nobody reading this needs it.

One note on the Raspberry Pi, because someone always asks: love the Pi, but it is not a local AI daily driver. It is the garage workshop, brilliant for edge experiments and sandboxed agent execution, wrong for running a real language model for chat or coding. For that you want one of the three tiers above.

The software, kept tight

Two tools dominate for getting models running.

**Ollama** is a command-line tool. One command downloads and loads a model, and it is running. Slightly higher learning curve, very simple once you are in, and it works on Mac, Windows and Linux.

**LM Studio** is the same idea with a visual, ChatGPT-style interface. If the command line makes you nervous, start here. Both handle downloading, detecting your graphics card, and serving the model locally so your other tools can talk to it.

One detail that saves real frustration: model files come in different packaging formats, like the same film on a DVD or a Blu-ray. **GGUF** runs best on Macs. **AWQ** is built for Nvidia cards and, on the right hardware, tends to give faster responses and better quality than GGUF on the same card. Most people grab whatever model has the most downloads and, on the wrong format for their machine, quietly leave speed on the table. Match the format to the hardware.

For agent workflows, a local model served this way plugs into orchestration tools for automation and multi-agent setups, which is its own topic for another day.

Be honest: local is not a total replacement for the cloud

A lot of local-AI content oversells this, so here is the straight version. A local model is not, today, a full replacement for the frontier cloud models. The most powerful reasoning still lives in the cloud, and when a genuinely hard problem lands, that is what you reach for.

But the smartest setup in 2026 and beyond is not one or the other, it is hybrid. Local for the daily work, cloud for the heavy lifting. Local AI is your home gym: always open, always private, handles maybe 80 percent of what you do, never a queue. The cloud is the specialist equipment downtown you walk to occasionally. You use both.

Where this fits a business, not just a hobbyist

Everything above is the hobbyist's version of a decision every serious business now faces: how much of our AI runs on infrastructure we control. The kitchen is the same, the stakes are higher. The 80 percent that is private, repetitive and sensitive wants to run in-house, on hardware sized to the workload, under your own governance. The 20 percent that needs frontier reasoning reaches out to the cloud.

The piece that makes the hybrid actually work is the layer underneath both: a [company knowledge graph and operating layer](/llm-infrastructure) that gives an agent the same governed context whether it answers on your own hardware or calls a frontier model. Without that, local and cloud are two disconnected brains. With it, they are one system that happens to run in two places.

The summary you came for

- Buy the counter, not the hand speed. **VRAM is the number.** Everything else is secondary.

- Budget the whole kitchen, not just the chef.

- If you are starting, tier one at US$1,200 to $1,500 is real and capable, and you can upgrade the card later.

- Match the model format to your hardware, and quantise to fit.

- Go hybrid: local for the daily 80 percent, cloud for the frontier 20 percent.

If you are weighing sovereign or on-premise AI for your business and want to know which of your workflows should run in-house, that is exactly what our [AI Opportunity Audit](/ai-opportunity-audit) maps. It is free, runs in your browser, and takes about two minutes to show you where your context lives and what it would take to run it on hardware you own.