Note: this article provides a technical overview and does not replace legal advice.
The problem with public AI services
Anyone who copies texts, emails or contracts into a public AI chat passes this data to an external provider – often based in the USA. For personal data, trade secrets or confidential customer documents this is delicate: you need a legal basis, have to assess the data transfer and often don't know exactly how long and for what purpose the data is stored.
The alternative: your own language model
Powerful language models are now also available as freely usable open-weight models, for example from the Llama, Mistral, Qwen or gpt-oss families. They can run on your own GPU server. Your employees then use a chat interface that feels similar to ChatGPT – except that all requests and documents stay on your server in Austria or Germany.
Typical use cases
- Internal knowledge assistant: answer questions about manuals, policies and project documents – with sources.
- Texts and emails: write drafts, summarise, translate.
- Customer support: suggested replies based on your previous tickets and product documentation.
- Document processing: extract and structure invoices, contracts or forms.
- Programming: a code assistant for your development team without source code leaving the company.
How does it work technically?
A private AI solution essentially consists of four building blocks:
GPU server
The hardware the model runs on – depending on model size, for example with NVIDIA RTX PRO 6000, H100 or H200.
Inference server
Software such as vLLM or Ollama serves the model and answers requests – including from several users at once.
Knowledge connection (RAG)
Your documents are made searchable. For each question the AI looks up the relevant passages and answers on that basis.
Interface & integration
A chat interface for employees with user management – or an interface (API) for your own software.
Which GPU you need depends mainly on the graphics memory the model requires. Details are explained in our guide GPU servers for AI.
What to consider for data protection
- Server location in the EU and a European contract partner
- Data processing agreement with the hosting provider
- Access rights: who may query which documents via the AI?
- Decide deliberately on logging – what is stored and for how long?
- Inform employees which data may go into which system
When using AI, the requirements of the EU AI Act also apply, for example on transparency and AI literacy of staff. Have the legal questions for your specific use case reviewed.
What does private AI cost?
The biggest cost factors are the GPU server (depending on model size and number of users) and the one-time setup. In return there are no fees per user or per request – with many employees or intensive use, your own server is therefore often cheaper than subscriptions for every workstation. We are happy to provide a non-binding estimate.
How to get started
- Choose one concrete use case – for example the knowledge assistant for one team.
- Start with a mid-sized model on a single GPU server.
- Test with real users and collect feedback.
- Only then expand: larger models, more documents, more teams.
We support you from the hardware to the finished solution: GPU servers in Austria or Germany, setting up the model and interface and – as software developers – integrating it into your existing systems.
FAQ
Is a private language model as good as ChatGPT?
For most business tasks – texts, summaries, questions about your own documents – current open models deliver very good results. For very complex specialised tasks, the largest commercial models may still have advantages.
Will the AI be trained on my data?
No, not without your involvement. With the RAG approach your documents are only used to answer questions; the model itself stays unchanged. Fine-tuning on your own data is possible, but a deliberate decision.
Do I need my own IT experts for this?
No. We set up the server, model and interface and, on request, also handle ongoing operation and updates.