-
Hopper GEMM in CUTLASS C++ and the CuTe DSL, part 2, the kernel [W4A8]
Part 2 of a series that reads the CUTLASS 3.x C++ Hopper GEMM stack from the source and puts each piece next to its CuTe DSL equivalent, as the base for porting vLLM's W4A8 kernel to the CuTe DSL. This part covers the warp-specialized cooperative kernel, the persistent tile scheduler, and clusters with TMA multicast
-
Expert Parallel MoE in Decode - From a Padded Grid to a Persistent Grid
How an expert-parallel MoE layer runs in decode on one GPU with DeepSeek-V4-Flash shapes. It covers DeepEP's padded buffers and masked_m, the T x topk bound, the flat row list and its prefix sum, and why a persistent grid launches about 1,184 CTAs. A padded grid launches 24,576.
-
Mixture of Experts in vLLM - From First Principles to fused_moe
How a mixture-of-experts layer works, built up one step at a time with Qwen3-30B-A3B shapes, from one expert and one token to the (token, expert) pairs of a batch, the per-expert GEMM shapes, and how vLLM's fused_moe runs the whole layer with two index lists and no copies of the input
-
Hopper GEMM in CUTLASS C++ and the CuTe DSL, part 1, the layers and the builder [W4A8]
Part 1 of a series that reads the CUTLASS 3.x C++ Hopper GEMM stack from the source and puts each piece next to its CuTe DSL equivalent, as the base for porting vLLM's W4A8 kernel to the CuTe DSL. This part covers the five levels of CUTLASS 3.x, how a kernel is assembled from them, and what the CollectiveBuilder chooses
-
Large-Scale LLM Training Systems - How and Why
How large-scale LLM training systems are designed, from the 16 bytes per parameter and mixed precision to collectives, ZeRO and FSDP, tensor, sequence, context, expert, and pipeline parallelism, 4D composition, MFU, and what breaks at scale