Why I Moved My AI Experiments Local

By late 2023 the public AI tooling ecosystem had exploded — tens of thousands of new GitHub projects touching large language models appeared in the space of months, and the pace has not really slowed since. That explosion was exciting to watch, but for anyone advising SMEs on where their data goes, it also raised an obvious question before the shiny-new-toy question: who else can see what we are about to type into this thing?

The cloud AI tools available at the time were genuinely impressive. But the economics and the data-handling trade-offs were impossible to ignore once you looked past the demo. Every prompt sent to a hosted model travels outside your network, usually to a jurisdiction and a vendor data-processing agreement you have not read carefully. For personal experimentation that is a minor concern. For a business handling client records, financial data, or anything covered by a confidentiality clause, it is a governance decision, not a technical one.

So I started moving experiments into a local homelab, and the project ended up teaching me more about workload-to-location mapping than any amount of reading would have.

Which workloads actually suit local inference

Not everything benefits from running locally, and pretending otherwise wastes effort. In practice the split looks something like this:

• Local is fine for: classification, tagging, summarising internal documents, drafting first-pass text, simple data extraction, and anything where the input is sensitive but the reasoning required is shallow.
• Cloud frontier models earn their place for: complex multi-step reasoning, anything needing the largest available context window, and tasks where the output quality gap between a 7B local model and a frontier model is still wide enough to matter.
• Neither is right for: anything regulated that neither you nor the vendor has a clear data-handling answer for. If you cannot say where the data sits and who can access it, that is the answer.

A smaller local model run on your own hardware will not match a frontier model on genuinely hard reasoning. But a surprising amount of day-to-day business AI use — internal note-taking, first drafts, routine classification — does not need that ceiling. It needs predictable cost and a clear answer to "where does this go."

The rough cost framing

Cloud inference is billed per token, which is fine until usage climbs and nobody is watching the meter. A local setup flips the economics: a one-off hardware cost (or hardware you already own), ongoing electricity, and no per-request bill. For low-to-moderate, steady workloads, that trade generally favours local once you are past the first few months. For spiky or occasional use, cloud's pay-as-you-go model is usually cheaper and much less hassle.

The useful exercise for an SME is not picking a side. It is working out, task by task, which side of that line each workload actually sits on — and then asking whether anyone is tracking the spend at all, because "nobody knows what we're spending on AI tools" is the pattern I still see most.

The data-boundary questions an SME should actually ask

Before any department adopts an AI tool, three questions are worth putting in writing:

1. Where does the data go? Not "which vendor" — which country, which data centre, and whether that is contractually fixed or can change.
2. Who can see it, and for how long? Ask directly whether your prompts and outputs are used for model training or retained for support purposes, and get the answer in the contract, not a sales call.
3. What happens if we need to prove where the data was? If you cannot produce an audit trail showing what left the network and when, you do not have a policy — you have a hope.

None of this requires sophistication. It requires someone asking the question before the tool is rolled out to twenty people, rather than after.

What the local setup actually looked like

The early build was deliberately basic: a GPU-backed VM, local model storage, and simple monitoring from day one so I could see what was actually being asked of the system. That was enough to prove the idea before investing further.

The lesson

Once you run even a handful of workflows locally, the shape of the problem changes. Privacy becomes something you can reason about rather than take on trust. Costs become predictable instead of a surprise on the invoice. And you end up writing down routing rules — which workload goes where — because those rules turn out to be the valuable part, not the model itself.

That was the start of a longer shift in how I think about AI infrastructure for clients, but the important bit was simple: treating "local or cloud" as a deliberate, per-workload decision rather than a default made the whole system easier to trust.

If your organisation is weighing up where its own AI workloads should sit, that is exactly the kind of question I help fractional IT and security clients work through — have a look at my services page or get in touch to talk through your specific setup.