Running LLMs fully offline: a practical guide for privacy-sensitive businesses

Some data cannot leave your building. Law firms, clinics, banks, manufacturers with trade secrets — for them, sending prompts to a cloud API is not a risk assessment, it is a hard no. The good news: open-weight models crossed the quality threshold a while ago. A well-deployed offline LLM in 2026 handles summarization, extraction, classification, and grounded Q&A at a level that would have required a frontier API two years ago. Here is how we scope these deployments.

Pick the smallest model that passes your eval

Model choice drives everything downstream — hardware, latency, cost. The mistake we see most is teams reaching for a 70B-parameter model when a 8B model fine-tuned on their domain does the job. Our rule of thumb from production deployments:

• Llama 3.1 8B or Mistral 7B class: document classification, entity extraction, routing, short summaries. Runs comfortably on a single 24GB GPU.
• Qwen 2.5 32B or Llama 3.3 70B class: long-document summarization, multi-step reasoning, RAG over large internal corpora. This is the sweet spot for most business assistants.
• Mixtral 8x7B / mixture-of-experts models: high throughput with many concurrent users, at the cost of much higher VRAM (the full model must be resident even though only a fraction activates per token).

Build a small eval set first — 50 to 100 real inputs from your business with expected outputs — and test candidate models against it before buying any hardware. This takes days, not weeks, and routinely saves clients five figures in GPUs.

Quantization: free VRAM, small quality cost

A 70B model at full precision needs roughly 140GB of memory. Quantization compresses the weights: Q4 (4-bit) cuts that to about 40GB, Q8 to about 75GB. In our measurements, Q4_K_M GGUF quantization costs 1-3% on typical benchmarks and is usually invisible on extraction and summarization tasks. Q8 is our default when quality matters more than VRAM. Below 4 bits, quality falls off a cliff — do not go there for production.

Hardware sizing with real numbers

• Single user or small team, 8B model at Q4: one RTX 4090 (24GB), ~50-80 tokens/sec. Total machine cost around 3,500 EUR.
• Department assistant, 70B model at Q4: 2x A100 40GB or a single H100 80GB. Expect 15-25 tokens/sec for one stream, scaling well with batching.
• High-concurrency API (50+ simultaneous users): a 4x H100 node running vLLM with continuous batching, or a cluster of smaller GPUs serving a quantized 32B model — often cheaper and easier to operate.

Do not forget CPU offload as a budget option: llama.cpp can run a 70B Q4 model on 64GB of RAM with a fraction on GPU, at 3-6 tokens/sec. Too slow for chat, fine for overnight batch jobs like document processing pipelines.

vLLM vs llama.cpp

These are the two serving stacks we deploy. vLLM is the production choice for concurrent traffic: PagedAttention and continuous batching give 3-5x the throughput of naive serving, it exposes an OpenAI-compatible API, and it handles tensor parallelism across GPUs. llama.cpp is the portability choice: it runs anywhere (CPU, Metal, CUDA), starts in seconds, and is ideal for edge boxes, demos, and low-volume internal tools. A typical llama.cpp launch looks like this:

bashllama-server \
  --model llama-3.3-70b-Q4_K_M.gguf \
  --ctx-size 32768 \
  --n-gpu-layers 99 \
  --port 8080

Both speak the OpenAI API dialect, so your application code never changes whether it points at a local server or a cloud endpoint. We standardize on that interface for every deployment.

When offline beats the cloud

Offline wins when compliance forbids third-party processing, when latency to a distant API matters, when you process huge volumes at a fixed cost (a GPU doing 24/7 batch extraction costs less than per-token API billing past a few million tokens a day), and when connectivity is unreliable. Cloud APIs still win for frontier reasoning quality, zero ops burden, and spiky low-volume workloads. Many of our clients run both: cloud for the hard cases, offline for everything sensitive.

This is exactly the kind of system we build at SLASH AI — model selection, quantization, serving stack, and fine-tuning included, deployed on your hardware with no data leaving your network. If you are scoping an offline LLM project, get in touch and we will tell you honestly whether offline is the right call for your case.