How to choose a server for AI workloads: what actually matters
This guide explains how to size a server for AI workloads without overspending. The right answer depends entirely on what you run: calling external APIs, hosting agents, or running open models on your own hardware, and the difference between those setups is 2 GB of RAM versus a GPU server. It is written for developers and sysadmins who want working numbers, not buzzwords.
Start with the workload, not the hardware
An AI server is not one thing. In practice there are three distinct workloads, each with a different bottleneck:
- Calling external AI APIs: your code sends prompts to a hosted model. The bottleneck is network and reliability.
- Running agents and automations: schedulers, chat memory, headless browsers, queues. The bottleneck is RAM and disk.
- Self-hosting open models: inference on your own CPU or GPU. The bottleneck is memory bandwidth and raw compute.
Most sizing mistakes come from buying for the third case when you are actually in the first.
Calling external APIs: almost any VPS works
If your application sends requests to OpenAI, Anthropic or a similar provider, the heavy compute happens on their side. Your server does HTTP, JSON and a bit of queueing. A machine with 1-2 vCPU and 2 GB RAM comfortably handles thousands of API calls per day.
What actually matters here:
- Uptime and stable outbound networking. A flaky uplink means failed webhooks and duplicate retries, which spend API tokens twice.
- Timeouts and retry logic. A model response can take 30-120 seconds. Set client timeouts accordingly and make handlers idempotent.
- Not latency. Inference time dominates: 40 ms of extra round trip is invisible next to a 15 second generation.
Quick sanity check from your server:
curl -o /dev/null -s -w "connect: %{time_connect}s total: %{time_total}s\n" https://api.anthropic.com/
Agents and automations: RAM and disk fill up first
An agent stack looks light on paper and heavy in practice. The Python or Node runtime, a vector store, Redis, a database and especially headless browsers add up fast: one headless Chromium instance takes 400 MB to 1 GB on its own. Plan 4-8 GB RAM and 2-4 vCPU for a small production agent setup, and more if agents run browser sessions in parallel.
Disk fills up quietly. Agents log every step, store conversation context and cache downloads. A busy automation can write several GB of logs per month. Take 40-80 GB of NVMe and configure rotation on day one:
free -h du -sh /var/log /home/agent/.cache journalctl --vacuum-size=500M
Swap is worth having as insurance against OOM kills, but if agents regularly touch swap, move up a RAM tier: swapping context mid-run makes agents time out in ugly, hard to debug ways.
Self-hosting small open models on CPU
You can run models with 3-8B parameters on a VPS without a GPU using llama.cpp or Ollama. Two things decide whether this is usable: RAM and AVX support.
Quantization basics. Model weights are normally stored in 16 bits. Quantization stores them in 4-5 bits with a small quality loss, cutting RAM needs roughly in four. The usual sweet spot is Q4_K_M. Rule of thumb: a Q4 model needs about 0.7 GB of RAM per billion parameters, plus 1-2 GB for the OS and context. An 8B model therefore wants around 7-8 GB total, so a 16 GB plan runs it with room to spare.
Check the CPU first. Without AVX2 inference runs, but painfully:
grep -o 'avx[0-9_]*' /proc/cpuinfo | sort -u
Expect honest numbers: on 4-8 modern vCPU with AVX2, a 7-8B Q4 model generates roughly 3-8 tokens per second. That is fine for background summarization, classification, or processing data that must not leave your infrastructure. It is too slow for interactive chat with waiting users. Test it in five minutes:
curl -fsSL https://ollama.com/install.sh | sh ollama run llama3.1:8b "Summarize this in one sentence: ..."
Sizing cheat sheet
Numbers that work in practice, per workload:
Workload vCPU RAM Disk Notes API calls only 1-2 2 GB 20 GB network reliability first Agents and automations 2-4 4-8 GB 40-80 GB NVMe, log rotation 1-3B model, Q4 quantized 4 8 GB 40 GB AVX2 required 7-8B model, Q4 quantized 6-8 16 GB 60 GB 3-8 tokens/s, batch jobs 13B+ or interactive inference GPU server or external API
These figures assume the model is the main tenant. If the same machine also runs your application and database, add their requirements on top instead of sharing.
When a GPU server or the API is honestly cheaper
Run the numbers before buying hardware. Small hosted models cost around 0.1-0.5 EUR per million tokens. If you process under roughly 1 million tokens per day, the API bill is a few EUR per month: no server you can rent beats that. Self-hosting on CPU wins only when data must stay on your infrastructure, or when you have steady batch volume and latency does not matter.
A GPU makes sense when you need models above 13B, interactive latency, fine-tuning, or sustained throughput that would cost hundreds of EUR per month in API fees. The catch is utilization: a GPU that is busy 2 hours a day is expensive, and the economics only work when it runs near capacity, so either consolidate workloads onto it or stay with the API.
A reasonable path: start on a VPS for API-driven workloads and agents, measure your real token volume and RAM usage for a month, then move to a dedicated server with the CPU or GPU configuration your measurements justify. Sizing from data beats sizing from launch-day optimism.