Constrained Decoding for LLMs: Grammar-Guided Generation with XGrammar, Outlines and llguidance
Constrained decoding for LLMs explained: token masking, finite-state machines, pushdown grammars, XGrammar, Outlines and llguidance, with overhead.
Speculative Decoding Explained: EAGLE-3 vs Medusa vs Lookahead for LLM Inference Speedup
Speculative decoding explained: draft-and-verify math, EAGLE-3 vs Medusa vs lookahead, acceptance rates, vLLM and SGLang support, and when it hurts.
GPU Memory Physics: DRAM vs SRAM, Kernel Fusion and FlashAttention (Series Part 3)
GPU memory physics: DRAM vs SRAM latency and bandwidth, tiling, kernel fusion, and why FlashAttention made LLM inference faster. Series finale.
llm-d on Kubernetes: Prefill/Decode Disaggregation for LLM Inference
llm-d explained: disaggregated prefill and decode, KV-cache-aware routing and vLLM on Kubernetes, with cost math, an install path and where it beats a plain vLLM deployment.
SGLang 0.5.18 vs 0.5.15: What Changed and How to Upgrade
SGLang 0.5.16-0.5.18 add DSpark speculative decoding, a Rust server and Breakable CUDA Graph. Breaking flags, cache moves and a step-by-step upgrade path.
OpenVINO 2026.4 vs 2025.4: What Changed for Edge LLMs and NPUs
OpenVINO 2026.4 vs 2025.4: EAGLE-3 and MTP speculative decoding, MoE disk offload, FP8 in NNCF, NPU changes, and the APIs removed in 2026.
What a ChatGPT Query Actually Costs in Energy and Water: Every Number, Traced to Source
The 500ml and 2.9 Wh figures come from real papers that said something narrower. Here is every published measurement of AI energy and water per query, with its scope.
MLPerf Edge Agentic Inference: How TensorRT Edge-LLM Beat llama.cpp 6.4x
MLPerf Inference v6.1 added an Edge Agentic benchmark. NVFP4, FP8 KV cache, 96% cache reuse and tree MTP explain the 6.4x gap over the llama.cpp reference.
Flink 2.2 ML_PREDICT and VECTOR_SEARCH: Running Model Inference Inside a Stream
Flink 2.2 adds VECTOR_SEARCH and Table API ML_PREDICT for in-stream inference. How they work, where they break, and when to keep inference outside.
ExecuTorch 1.5 On-Device LLM Serving (2026): Batched Scheduling, Cancellation and Off-Graph KV Cache
ExecuTorch 1.5 shipped multi-method export, batched request scheduling, bounded cancellation and off-graph KV cache. What it changes for on-device LLM apps.