Flink 2.2 ML_PREDICT and VECTOR_SEARCH: Running Model Inference Inside a Stream
Flink 2.2 adds VECTOR_SEARCH and Table API ML_PREDICT for in-stream inference. How they work, where they break, and when to keep inference outside.
ExecuTorch 1.5 On-Device LLM Serving (2026): Batched Scheduling, Cancellation and Off-Graph KV Cache
ExecuTorch 1.5 shipped multi-method export, batched request scheduling, bounded cancellation and off-graph KV cache. What it changes for on-device LLM apps.
vLLM vs SGLang vs TensorRT-LLM in 2026: The Serving Engine Pick
vLLM, SGLang, and TensorRT-LLM compared for 2026 production LLM serving: throughput, KV-cache handling, and which to pick by workload.
Diffusion LLMs: How Text Diffusion Models Work (2026)
Diffusion LLMs explained: how text diffusion models replace next-token prediction with iterative denoising and parallel decoding - masked discrete diffusion, remasking, throughput trade-offs vs autoregressive models in 2026.
Constrained Decoding: Architecture for Guaranteed-Valid LLM Output (2026)
How constrained decoding guarantees valid LLM output: grammars, FSAs, token masking, JSON-schema enforcement, and where structured generation breaks in production.
On-Device SLM Inference: A 2026 Edge GPU Benchmark
A 2026 benchmark methodology for small language models on edge GPUs — latency, tokens/sec, memory, and cost for Phi, Gemma, and Qwen on Jetson-class hardware.
vLLM Cost Economics: 2026 Deep Dive on $/Million Tokens
A practical 2026 deep dive on vLLM cost economics — KV cache, paged attention, speculative decoding, and dollar-per-million-tokens math.
KV Cache Optimization for LLM Inference: A Deep Dive
KV cache optimization for LLM inference — PagedAttention, quantization, prefix caching, and eviction, with the memory math behind each technique.