If you want to run an AI model on your own server, the first question is always: which AI hardware for business do you actually need? The answer depends on how large the model is, how many people use it at once, and how fast the answers have to come back. This guide explains how to make that choice without wasting thousands of euros on the wrong kit.
The core idea is simple: a language model is a large file full of numbers, and to make that file compute quickly you need a graphics card (GPU) with enough memory. The rest of the server, such as the processor, working memory, and storage, is secondary. The GPU, and above all its memory, decides what you can handle.
Why does an AI model run on a GPU instead of a normal processor?
A language model predicts the next piece of text one word at a time. To do that, it performs billions of multiplications for every word. A normal processor (CPU) handles that kind of math neatly, a few at a time, one after another. A GPU is built to run thousands of calculations in parallel, which is exactly what a language model needs. That's why a model runs ten to fifty times faster on a GPU than on a CPU.
You can run small models on a modern CPU, and for a test setup or a light background task that's perfectly fine. But as soon as you want to serve several people at once or expect quick answers, a GPU becomes essential. To understand which models you can run in the first place, start with our guide to open-source AI models for SMBs, because the model choice determines the hardware, not the other way around.
What is VRAM and why is it the most important factor?
VRAM is the memory that sits directly on the GPU. It's the fast working memory the model lives in while it computes. And this is the single most important rule in this guide: the model has to fit in the VRAM, otherwise it runs slowly, or not at all.
How much VRAM you need depends on two things: the size of the model and how much it's compressed.
Model size is expressed in parameters, usually in billions (marked with a B). A 7B model has 7 billion parameters, a 70B model has 70 billion. More parameters generally means smarter answers, but also more memory.
Compression is called quantization in AI jargon. By default, each parameter is a number that takes up 16 bits. With quantization you shrink that to, say, 8 or 4 bits per parameter, which makes the model two to four times smaller. You give up a little quality, but at 4-bit that loss is often barely noticeable for business use. Quantization is therefore the main lever for fitting a larger model onto more modest hardware.
A handy rule of thumb for the minimum VRAM you need: take the number of billions of parameters and multiply it by roughly 0.5 to 1 GB for a 4-bit model, or by roughly 2 GB for an uncompressed 16-bit model. Then add some room for the context (the text you feed in) and for multiple simultaneous users. A 7B model at 4-bit fits comfortably in 8 GB of VRAM, whereas the same model uncompressed quickly needs 16 GB.
Which hardware matches which model size?
The table below links model sizes to a hardware tier and a typical use. The VRAM figures assume 4-bit quantization, which is the sensible default for most business use. The amounts mentioned are indicative and meant to show orders of magnitude, not to serve as a quote.
| Model tier | VRAM (4-bit) | Type of hardware | Typical use |
|---|---|---|---|
| Small (1B–3B) | 2–4 GB | Consumer GPU or modern CPU | Classification, tagging, simple summaries |
| Mid (7B–13B) | 8–16 GB | Consumer GPU (mid/high end) | Internal chatbot, drafting emails, document Q&A |
| Large (30B–34B) | 20–32 GB | Datacenter GPU or dual consumer cards | More complex reasoning, RAG over a knowledge base |
| Very large (70B) | 40–48 GB | Datacenter GPU | Quality close to top cloud models, demanding tasks |
The takeaway: for the vast majority of SMB use cases, a model in the 7B to 13B tier is more than enough. An internal assistant that drafts emails, summarizes documents, or answers questions about your manuals runs perfectly well on a single good GPU with 16 GB to 24 GB of VRAM. Only when your model needs complex reasoning or legally precise answers do you climb into the larger tiers, and that's where datacenter GPUs come in.
[ TIME SAVED ]
Save 12 hours per week on manually drafting and answering standard questions
Should you buy a server or rent a private GPU?
Once you know which tier you need, the second big choice arrives: do you buy hardware yourself, or rent compute from a European provider? Both work, and the answer depends on your volume and your requirements around sovereignty.
Your own server (on-premise) means you buy the hardware and place it in your own building or datacenter. An entry-level server with a consumer GPU of 16 GB to 24 GB of VRAM costs, indicatively, a few thousand euros as a one-off. A server rack with a datacenter GPU for the larger models can quickly run into the tens of thousands. In return, after purchase you use it without limit and without a monthly bill, and your data literally never leaves your building. For organizations with hard compliance requirements, that's the deciding argument.
Renting a private GPU means you reserve a GPU server that runs exclusively for you at a European hosting provider such as Hetzner, Scaleway, or OVHcloud. You share the card with no one, your data stays in the EU, but you don't have to manage any hardware yourself. Costs are, indicatively, in the range of a few hundred euros per month for an entry card, rising for the heavier datacenter GPUs. For many businesses this is the practical middle ground: you test quickly, you scale up or down easily, and you're not tied to a large upfront investment.
A simple rule of thumb: if you run AI lightly or irregularly, renting is cheaper and more flexible. If you run AI intensively and daily, your own server wins over time because the fixed cost per use ends up low. To make that calculation properly, our AI cost overview for SMBs helps you place rental and purchase costs honestly side by side.
For most businesses we recommend starting with a rented private GPU. That way you first prove the application delivers value before you buy a server rack. If the volume turns out to be high and structural, you move to your own hardware. This approach also fits the broader theme of digital sovereignty for Dutch businesses, where you gradually pull more control into your own environment.
Which mistakes are best avoided?
The most expensive mistake is starting too big. Many businesses assume they need the largest model, buy a pricey datacenter GPU, and then discover a 7B model does the job just as well. Start small, check whether the quality is good enough, and only scale up when you have a concrete reason.
A second pitfall is ignoring quantization. Without it you need two to four times as much VRAM, and therefore much more expensive hardware, while the quality difference at 4-bit is usually negligible for business use. Always size your hardware based on a quantized model.
A third mistake is forgetting to account for how many people use the system at once. One user asking the occasional question needs very little, but ten people working simultaneously demand extra VRAM and a faster GPU. Map that usage pattern in advance and you'll buy exactly what fits.
If you'd rather not make these calls alone, we're happy to think along through our local AI solutions, where we work out together which model and which hardware fit your situation. Choosing the right size often saves thousands of euros, both at purchase and in monthly rental.
If you want to get serious about AI on your own infrastructure, start with the basics of local AI for business and then map out your concrete use case. From there, choosing the right hardware is a matter of arithmetic, not guesswork.