Running private LLMs is far from free - they are too big for enterprise DCs, so the invoice shifts from a cloud vendor to your NeoCloud one.
Between high-end GPU based systems, surging electricity and cooling bills, server maintenance, and the specialized engineering talent required to keep inference pipelines optimized, self-hosting carries, in real life, massive capital and operational expenses. And in the EU it gets harder: high-end GPU systems are scarce, and the sovereign AI promise is still an illusion - capacity you did not reserve is capacity someone else already holds.
Metering and billing internal teams by the token is essential: it creates accountability against wasteful compute loops and directly amortizes those upfront infrastructure, pipelines, and staffing investments. And your token counts tell you why you must reserve on a NeoCloud - and what: GPU systems with the memory and interconnect your model demands, held for the window your workloads need, in a market where on-demand EU capacity may simply not exist.
This page is an add-on to Local AI is not CHEAP - the same EDU AI LAB tokenomics, scaled from a small ~3B lab node to very large models on reserved NeoCloud GPUs (GitHub).
With Hybrid AI you can easily spend $1 million on fully private datacenter inference (renting one infra slice with GPU cost with long term reservation commitments $4-$10 per hour each), so '$1 000 000 / 365 / 24 = ~$114 per hour' say 12 GPU slices and storage.
See more in: howto.hybridai.click - Automating HybridAI on NeoCloud with Agentic Infra approach
Open weight models (open source + open science) are growing from large ~300B to huge 1.6T and now to giants around 2.8T.
6 models to explore as sizes are growing:
OK, Qwen3.8-Flash-Next at ~176B total looks small enough - but can you run it yourself?
Running Qwen3.8-Flash-Next myself - a story of hundreds of gigabytes. Model weights alone are huge, and quantization magic decides how huge: 335 GB in BF16, 173 GB in FP8, 130 GB in NVFP4 (Blackwell-only quant). "Just download the model" quickly becomes a storage and memory planning exercise.
Note: sizes from the vLLM & SGLang day-0 recipes (Aug 26). Weights alone - plus ≥51 GB host CPU RAM for the FP8 quantized N-gram table.
Wuff Wuff Wuff - 4 data-center cards?
More than model weights: serving multiple users effectively needs room for KV cache, activations, and concurrency - and the hardware must fit the precision, like NVFP4 running only on Blackwell. The official recipes call it a data-center deployment: NVFP4 default 4× B200 (TP4), FP8 minimum 2× GB300 (TP2), recommended full tray 4× GB300/B200 (TP4), and on Hopper 8× H200 in TEP8 - plain TP8 won't even load it.
InferenceX SemiAnalysis - We can already explore and Contrast & Compare some configs
Total tokens per $1 TCO vs P90 interactivity is the serving trilemma measured. The H200 SGLang TEP4 FP8 point: MTP speculative decoding, ~94% chip cache hit rate, ~50.4M tokens per $1 TCO.
InferenceX SemiAnalysis - Every point has a story
Click any point to view server metrics and logs - cache hit rates, queue depth, and the full request timeline for that run.
Let's make together deepdive into stories (perf graphs and configs) for Qwen 3.8 flash next and MiniMax M3.
If you are new to DC Interference, start with the main principles below first. Without them it will be difficult for you to follow the deep dive and understand the jargon used there.
This is a tricky architectural and engineering challenge with a lot of tradeoffs, ideal EDU AI LAB material.
Here I share minimal setups distilled from the larger ones I use in enterprise workshops, to demonstrate the complexity of local AI inference engineering in practice and the role of caching.
Tokenomics - cost is more than just tokens. It spans hardware and power, pipeline complexity, observability (can you find and debug it?), reproducibility, and manageability. Every form of spend needs to be metered at the end.
Hurray, until we see Vera Rubin systems online, models will not grow further, because ~1.6T is reasonable maximum which can still run effectively on 8xB300 system.
CC (Contrast & Compare) - in AI there is always more than one way to do things. A controlled environment with enough complexity lets you pick and test approaches, finding the right fit across tradeoffs and levels of integration. The EDU AI LAB contrasts and compares three inference engines on the same small ~3B model: llama.cpp, vLLM, and SGLang.
The whole fight of getting maximum performance in AI is an optimization GAME.
Roofline diagrams (for GPUs), like the one above, help identify whether a program's performance is bottlenecked by compute power or memory bandwidth.
Serving a large LLM is not one problem but twelve, and they pull in different directions at once. Mostly these are not simple tradeoffs, but genuine trilemmas: three goals where winning any two means giving up the third (see Trilemma).
The first four are what users feel: tokens that matter, waits they notice, and the speed-cost-quality puzzles behind them. The other eight are what the hardware forces: fitting giant models onto GPUs, feeding them long contexts, and keeping every GPU you pay for busy on tokens people actually want.
1. Goodput - the metric that actually matters. Raw throughput looks great on vendor slides, but it counts every token your GPUs burn - even the ones nobody wanted.
What counts is the tokens that actually serve users: time to first token (how long users wait before anything appears), tokens per second while streaming, and staying stable when agentic workloads hammer the system with unpredictable, multi-turn loops. Each one pulls the hardware in a different direction: first tokens are bound by compute (FLOPs), streaming by memory bandwidth (HBM), and agentic stability by cache reuse - no single knob wins all three.
Delivering them is what drives real engineering: fast first tokens mean taming quadratic prefill compute (chunked prefill, KV caching), smooth streaming is a memory-bandwidth fight (FlashAttention, PagedAttention, speculative decoding), and agentic stability needs cache-aware scheduling (Radix Attention). Optimize for goodput, and batching, caching, parallelism - the whole serving stack - falls into place.
2. Latency percentiles - the average hides the pain. Half your users may see decent response times, but the average says nothing about the rest.
P50, P90, P95 tell you how the slowest users are doing - and the tail is exactly what users feel and complain about. Tail latency is lost goodput: seconds wasted waiting are tokens never delivered.
3. The AI serving trilemma - serving is a balancing act between speed, cost, and quality. Push one corner and another gives way: faster responses usually mean more GPUs, cheaper serving usually costs quality.
Every optimization trades one against another. Goodput is the score that tells you if the trade was worth it.
4. The AI deploying trilemma - before serving even starts, you have to decide where the model lives: your own GPUs, a cloud vendor, or something in between.
Each option shifts the speed, cost, and quality tradeoffs differently - and with high-end GPUs scarce in the EU, deployment choices lock in your goodput ceiling before a single token is served.
5. Blackwell is a family - "Blackwell" is one brand, not one chip: datacenter "Fathers" (B200, GB200, B300) and consumer/workstation "Sons" (RTX 5090, RTX PRO 6000, DGX Spark) share the name but not the same chip architecture.
Kernels tuned for one side do not run unmodified on the other, and new capabilities land datacenter-first. Which Blackwell you reserve decides what software can run on it - the details are all in the picture.
6. Model parallelism - one big model does not fit on one GPU, so it gets split across many. Tensor parallelism slices the weights so GPUs work on the same layer together, pipeline parallelism turns layers into an assembly line, and sequence parallelism splits a huge prompt.
Splitting well keeps every GPU busy - idle GPUs are pure waste, and good utilization means more goodput per GPU you pay for.
7. Long context is not free - the longer the prompt, the more compute it takes just to read it. Prompt processing grows fast with context size, and the bill lands straight on time-to-first-token and GPU memory.
Cache hierarchies and smarter memory use push the memory wall back, but every extra token of context still has a price.
8. Knowledge engrams - instead of stacking ever bigger KV tensors across RAM/VRAM tiers, bypass traditional KV caching by offloading context into standalone "knowledge engrams" that feed engram-aware layers directly.
Massive context without the VRAM bill - the price is extra compute spent retrieving and injecting engrams during inference: a memory limit traded for a processing overhead.
9. Cache offloading - KV cache lives in expensive, scarce HBM, and long contexts fill it fast. Offloading cache tiers to CPU RAM and SSDs buys back memory without buying more GPUs.
Two designs show the range: NVIDIA Dynamo dynamically spills cache from GPU HBM to local CPU RAM and SSDs to cut cost, while Kimi Mooncake pools cache in centralized remote memory over RDMA - the design behind 1M+ token context windows.
10. Prefill-decode disaggregation - reading the prompt and writing the answer are two very different jobs, so why force them onto the same GPUs? In the classic setup one long prompt blocks everyone else's answers in the queue.
Splitting prefill and decode onto separate GPU pools means no more head-of-line blocking, and each pool gets hardware that fits its job - predictable latency, scalable goodput.
11. Mixture-of-Experts serving - the trick behind today's huge models: the total model is enormous, but each token only wakes up a small subset of "experts".
That sparsity is why big models can stay (relatively) cheap to serve - massive capacity on paper, a fraction of the compute per token, and more goodput for the GPUs you already have.
12. Multi-token prediction - normally the model generates one token per step, which caps your streaming speed. MTP-style decoding predicts more than one token per step, so each GPU pass delivers more tokens.
Fewer steps for the same answer means faster tokens-per-second for users - a direct lift on TPOT and goodput.
Send reasoning back just one time: Trace as State: Reasoning Traces as Conditional States for Long-Context Transformers
Based on current discussions in AI architecture (often referred to as Recurrent Depth or Looped Transformers):
OpenAI Astra and Recurrent Depth / Looped Transformers (Sebastian Raschka):
GPT 6 Astra: The End of Chain-of-Thought? (Cloud Codes):
Deep dive materials about Inference Engineering - for self-study.
Inference Engineering - by Philip Kiely (Baseten). Now in its 3rd edition and free as a PDF.
A book for engineers who want to understand the technologies that power every AI company and application in the world.
Covers the inference stack end to end - GPU kernels and the memory wall, KV-cache and prefix caching, quantization, batching, and serving in production - the same tradeoffs the EDU AI LAB demonstrates live.
First Principles
The Engineering Behind LLM Inference - From First Principles
Must-study before serving any AI model to really grasp the tradeoffs.
PY (@thecommitlog) has excellent deep-dive videos (with sources) which can help you better understand the tradeoffs in Inference Engineering - they are great companions to the book by Philip Kiely.
The Engineering Behind LLM Inference From First Principles - YouTube playlist:
Future of LLM Inference is diversity by specialization
How will your future inference stack look?
The Entire AI Chip War Explained: Nvidia vs Everyone