The GPU Memory Hierarchy: Registers to VRAM

Five Levels from Registers to VRAM

GPUs expose a multi-level memory hierarchy that programmers often manage explicitly. Understanding each level is essential for writing efficient compute kernels and shaders.

1. Registers: per-thread private storage, ~1 cycle latency
2. Shared memory / L1: on-chip scratchpad shared within a thread block, ~20 cycles
3. L2 cache: shared across all SMs, ~50-100 cycles
4. Global memory (VRAM): off-chip DRAM, ~400-800 cycles
5. Constant / texture caches: read-only paths optimized for broadcast and 2D locality

If a kernel uses too many registers, the compiler spills them to local memory backed by VRAM. That spill is orders of magnitude slower than keeping values in registers. This tradeoff is called register pressure.

Latency Hiding, Not Just Bandwidth

GPUs have enormous bandwidth but terrible latency. The scheduler hides latency by switching to other ready warps while one warp waits on a global load. For example, a kernel that performs one global read per thread with low occupancy may stall frequently because too few warps are available to cover the ~400 cycle VRAM round trip.

cCopy
Loading…

Shared memory requires explicit allocation and copy. L1 cache is automatic but smaller. The key insight: maximize active warps and minimize global traffic through shared memory staging and coalesced access.

For This Exercise

You will model latency and bandwidth at each hierarchy level and compute how many warps a kernel needs to hide global memory stalls. This task asks you to print per-level access costs and explain when register spills would hurt occupancy. The numbers you compute here connect directly to the coalescing and occupancy exercises that follow.