What server do you need to run a local AI model?
Choosing a server for AI that runs a local language model comes down to one number more than any other: how much memory the model needs to sit in while it answers. Everything else (CPU, disk, network) matters, but memory decides whether the model runs at all. This guide gives you concrete sizing rules so you can pick hardware before you commit.
The one rule that decides everything: memory
A model's weights must fit into memory. On a GPU that means VRAM; on a CPU-only box that means system RAM. Quantization shrinks the weights so they fit on smaller hardware. A useful rule of thumb for 4-bit (Q4) models is roughly 0.6 GB of memory per billion parameters, plus overhead for the context window.
- 7B-8B model, Q4: about 6-8 GB VRAM. Runs on a single entry GPU, or slowly on a CPU with 16 GB RAM.
- 13B-14B model, Q4: about 10-12 GB VRAM.
- 30B-34B model, Q4: about 20-24 GB VRAM.
- 70B model, Q4: about 40-48 GB VRAM, usually two 24 GB GPUs.
Add headroom for the context window. A long 32k-token context can add several gigabytes on top of the weights.
GPU or CPU?
A GPU is not strictly required, but it changes the experience completely. On CPU, a 7B model may produce a few tokens per second: fine for background jobs, painful for interactive chat. A modern GPU with enough VRAM produces dozens of tokens per second. If people will type and wait for answers, budget for a GPU. If you only run batch summarisation or overnight processing, a strong CPU with plenty of RAM is workable.
The rest of the machine
- System RAM: at least the size of the model, ideally double, even when the model runs on the GPU.
- Storage: NVMe SSD. Model files are large (4 GB for a small model, 40 GB or more for a large one) and load faster from fast disk.
- CPU: matters most for CPU inference and for feeding the GPU. Modern cores with AVX support help.
- Bandwidth: the first download is big; inference itself is light on network.
How to check what a machine can do
On a Linux server, confirm the GPU and its memory, then pull a model with Ollama and watch memory use:
nvidia-smi # GPU model and VRAM free -h # system RAM curl -fsSL https://ollama.com/install.sh | sh ollama pull llama3.1:8b # download an 8B model ollama run llama3.1:8b # start a chat nvidia-smi # check VRAM used while it answers
If nvidia-smi shows the model fits with room to spare, you can move to a larger model or a longer context. If it spills into system RAM, responses slow down sharply.
Which CloudHosting option fits
For experiments and small 7B-8B models, a well-specified VPS with enough RAM is a sensible start. For interactive use, larger models, or several users at once, a dedicated physical server with a GPU in our Riga data centre gives predictable performance and full control. Because the model and your data stay on your own EU-hosted machine, you keep GDPR and NIS2 alignment and full data sovereignty, which is the main reason to self-host rather than call an external API. If you would rather not assemble the stack yourself, OpenClaw is our self-hosted AI package that ships ready to run on this kind of hardware.
Practical takeaway
Size the memory first: 0.6 GB per billion parameters for a Q4 model, plus context headroom. Match that to VRAM if you need speed, or to RAM if you can wait. Put the files on NVMe, keep RAM at least equal to the model, and test with ollama run and nvidia-smi before you scale up.