Confidential AI company · NDA
Self-hosted AI Platform
vLLMSGLangk3s16 GPUs
- Role
- Confidential AI company · NDA
- Year
- 2026
- Stack
- vLLM · SGLang · llama.cpp · Docker · k3s · Incus
Designed, built and ran a self-hosted AI platform on a multi-node cluster of 16 NVIDIA GPUs (about 464 GB VRAM, including an RTX PRO 6000 Blackwell).
- Served open-weight LLMs (vLLM, SGLang, llama.cpp) with tensor parallelism and FP8/NVFP4/INT4 quantization.
- Per-user AI agents, one Docker container each, with tool calling, memory, RAG, speech and OCR.
- VPN-only access, SSO, encrypted secrets; backups and monitoring on Docker, k3s and Incus.