Why AI needs a GPU
At its core, artificial intelligence consists of huge numbers of matrix calculations. Graphics cards (GPUs) have thousands of small compute cores that perform exactly these calculations in parallel – many times faster than a regular processor. Without a GPU, training is practically impossible and even generating answers from a language model (inference) is painfully slow.
The key metric: graphics memory (VRAM)
For a model to run fast, it must fit completely into the graphics card's memory. Rules of thumb for language models (LLMs):
- 16-bit (full quality): approx. 2 GB VRAM per billion parameters
- 8-bit quantised: approx. 1 GB per billion parameters
- 4-bit quantised: approx. 0.5–0.6 GB per billion parameters
- Plus headroom for the context (conversation history) and multiple concurrent users
An 8-billion-parameter model therefore runs on small GPUs. A 70-billion model needs around 40 GB in 4-bit and about 140 GB in 16-bit – so one large GPU or several. Training and fine-tuning also require memory for gradients and optimiser state, often several times the size of the model itself.
RTX PRO 6000, H100 or H200?
| GPU | VRAM | Best for |
|---|---|---|
| NVIDIA RTX PRO 6000 (Blackwell) | 96 GB GDDR7 | Inference, mid-sized LLMs, image generation, rendering – excellent value |
| NVIDIA H100 | 80 GB HBM3 | Training and fine-tuning, high throughput with many users |
| NVIDIA H200 | 141 GB HBM3e | Large models and long contexts, maximum memory bandwidth |
For many businesses that “only” want to provide a language model internally – such as a chatbot based on their own documents – the RTX PRO 6000 is the best entry point: lots of memory at a much lower price than data centre GPUs with HBM memory. If you train your own models or need to serve very many concurrent requests, H100 or H200 are the better choice.
Cloud GPU or your own GPU server?
Cloud GPUs are billed by the hour. That is ideal for trying something out or for a one-off training run. But if your model runs continuously, the hourly prices quickly add up to a multiple of the monthly rent for your own GPU server. Then there is data protection: with a rented server in Austria or Germany you know exactly where your data is and who has access.
- Cloud: short experiments, irregular load, quick trials
- Own GPU server: continuous operation, predictable costs, sensitive data, GDPR
What else to look out for
- CPU and RAM: for data preprocessing and loading models, the server needs enough system memory – at least as much as VRAM.
- Fast storage: NVMe SSDs considerably shorten the loading of large models and datasets.
- Connectivity: bandwidth matters when many users access the server or large amounts of data are transferred.
- Software stack: drivers, CUDA and inference servers such as vLLM or Ollama should be set up properly.
Our offer
We configure GPU servers with NVIDIA RTX PRO 6000, H100 and H200 individually for your use case – hosted in Austria or Germany. On request we set up the complete AI environment and support integration into your software.
FAQ
Can I run ChatGPT-like models myself?
Yes. Open models such as Llama, Mistral or Qwen can run on your own GPU server and be connected to your company data – without data flowing to an external AI provider.
Is a gaming graphics card enough for AI?
For first experiments with small models, yes. For professional continuous operation, gaming cards lack graphics memory and data centre suitability. Professional GPUs such as the RTX PRO 6000 or H100 are designed for 24/7 operation.