From bare-metal microcontroller registers to large-scale LLM inference pipelines. We obsess over every microsecond, every cache miss, every memory access pattern.
$ perf stat -e cycles,instructions,cache-misses ./inference
# Performance counter stats:
42,891,234,567 cycles
98,234,567,890 instructions # 2.29 IPC
1,234,567 cache-misses # 0.001%
✓ 3.2x throughput improvement Complete treatment of atomic operations and reduction patterns in CUDA. Covers atomicAdd contention and throughput, warp-level reductions using __shfl_xor_sync, block-level reductions with shared memory, multi-block reductions with global memory coordination, and full implementations of a histogram kernel and parallel prefix sum.
Complete technical guide to CUDA Cooperative Groups — thread_block, thread_block_tile, coalesced_groups, grid_group, and multi_grid_group. Covers tiled partitions for warp-level programming, cooperative kernel launches, grid-wide synchronization, and benchmarked implementations of reductions and scans using cooperative groups.
Complete treatment of CUDA Graphs for LLM inference optimization. Covers graph capture mechanics, replay for single-launch kernel sequences, memory pre-allocation constraints, handling dynamic shapes via multi-graph caching, vLLM's graph capture strategy for different batch sizes, and implementation of a forward pass captured as a CUDA graph.
Step-by-step construction of a custom multi-head attention CUDA kernel — from the naive O(N^2) materialization approach, through tiled attention with shared memory, to the online softmax technique used in FlashAttention. Each version benchmarked for memory usage and throughput on an A100.
Comprehensive guide to implementing high-performance GEMM kernels using NVIDIA CUTLASS. Covers the CUTLASS template hierarchy (Gemm, threadblock, warp, instruction tiles), tile scheduling strategies, epilogue functors, mixed-precision GEMM configuration, the relationship between CUTLASS templates and hardware tensor core instructions, and performance tuning methodology with Nsight Compute profiling.
Comprehensive guide to debugging CUDA programs — compute-sanitizer for memory errors and race conditions, cuda-gdb for interactive debugging, common error codes and their root causes, race condition detection with racecheck, synchronization errors, and systematic debugging workflows for production CUDA code.
A thorough technical guide to CUDA Dynamic Parallelism — device-side kernel launches, nested execution, memory visibility, synchronization semantics, performance characteristics, and the narrow set of problems where device-side launches outperform host-driven alternatives.
A systematic treatment of CUDA kernel fusion patterns for LLM inference. Covers why fusion eliminates HBM round-trips, elementwise fusion (bias+activation+dropout in one kernel), reduction fusion (LayerNorm as a single kernel), GEMM epilogue fusion (bias+activation after matmul), attention fusion (FlashAttention), and a complete implementation of a fused bias+GELU kernel.
Step-by-step optimization of matrix transpose in CUDA — naive global memory, coalesced read with scattered write, tiled shared memory, bank-conflict-free padding, vectorized loads, and diagonal block ordering. Each version is benchmarked to show exactly where each optimization gains throughput.