What is vLLM and why is it used for self-hosted AI inference?

vLLM is an open-source inference engine for large language models, chosen over alternatives like LMDeploy, SGLang, and TensorRT-LLM for its performance, broad model support, extensive documentation, and built-in multi-model type routing. Its production stack includes an OpenAI-compatible API router with prefix-aware routing, GPU worker nodes with native tensor parallelism support, KV cache storage via LMCache for multi-worker environments, and Prometheus/Grafana observability. It ships with infrastructure diagrams that can be extended for custom multi-model setups.

What is the recommended approach to self-hosting AI on-premise?

A phased approach is recommended to avoid locking into expensive hardware prematurely. Phase 1: use managed services like AWS Bedrock to test model options and find the right size and reasoning capability before committing to infrastructure. Phase 2: use cloud-managed Kubernetes (e.g. AWS EKS) to prototype your multi-model setup, benchmark GPU requirements, and understand TPM/RPM demands. Phase 3: once you understand your usage patterns, migrate your existing Kubernetes cluster to an on-premise deployment. Starting directly with on-premise hardware represents significant operational risk without the data from Phases 1 and 2.

How much does it cost to self-host an LLM?

Costs vary significantly by model size. Small models (7B–8B parameters, e.g. Llama 3 8B, Mistral 7B) suitable for customer support and simple Q&A run on a single G6e instance at approximately $400–600/month. Medium models (70B parameters, e.g. Llama 3 70B, Qwen 72B) suitable for internal knowledge bases and RAG require a multi-GPU setup at approximately $3,000–8,000/month. Large models (100B+ parameters, e.g. GPT-OSS-120B, DeepSeek-R1) for complex reasoning, code refactoring, and advanced agents require p5e.48xlarge instances with 8×H200 GPUs at approximately $30,000+/month. These are single-environment estimates — multi-environment enterprise setups multiply these figures quickly.

What LLM optimization techniques have the biggest impact on inference performance?

The six optimization techniques with the biggest practical impact are: quantization (reducing weight precision from 16-bit to 8-bit or 4-bit to shrink model memory footprint, directly affecting which models fit on available hardware); automatic prefix caching (worker-level KV state caching that avoids reprocessing repeated context like long documents, improving throughput and reducing latency); distributed caching via LMCache (cluster-wide offloading to CPU memory, disk, or Redis for cheaper tiered storage); tensor parallelism (splitting tensors across multiple GPUs — essential for large models, dependent on fast NVLink interconnects); speculative decoding (pairing a large model with a fast small model that predicts tokens verified by the large model in a single pass, multiplying generation speed); and disaggregated prefilling (separating compute-bound prefill from memory-bandwidth-bound decode onto different GPUs, enabling independent scaling of each stage).

What metrics should you benchmark when evaluating a self-hosted LLM setup?

The key benchmarking metrics for a self-hosted LLM inference setup are: Median TTFT (Time to First Token) — user-perceived responsiveness from prompt submission to first generated token; Median TPOT (Time Per Output Token) — latency per subsequent token; Median ITL (Inter-Token Latency) — gap between consecutive tokens affecting streaming smoothness; Output token throughput — tokens generated per second across all concurrent users; Total token throughput — combined rate for prompt processing and generation; Request throughput — complete requests resolved per second; and Max request concurrency — peak simultaneous requests handled during the test. Generic benchmarking tools like LLMfit are a starting point, but you should benchmark in your own environment to surface hardware-specific traps — for example, L40S GPUs communicating over PCIe rather than NVLink can see communication overhead cancel out additional compute from adding more GPUs.

Author at Infinum

Author

Vjekoslav Drakšić

GPU server rack in a dark blue data centre environment, representing self-hosted AI model inference infrastructure

Self-Hosting AI Models: A Practical Guide to Building Your Own Stack

Infrastructure decisions, model selection tradeoffs, and performance optimization techniques we encountered while building a self-hosted multi-model inference stack.