How PyTorch runs the same operations on a GPU

pgmesa · · 40 min read

Part of Rebuilding PyTorch (8 of 8)
  1. Why I rebuilt PyTorch's core from scratch
  2. What a PyTorch tensor really is
  3. Interlude: What the CPU does when your code reads memory
  4. How PyTorch walks a tensor of any shape
  5. How PyTorch prepares the loop of every operation
  6. How a PyTorch kernel uses the vector registers of every CPU
  7. How PyTorch sums a tensor, and splits the work between threads
  8. How PyTorch computes gradients
  9. How PyTorch runs the same operations on a GPU

Every operation in this series so far ran on the CPU. Move the tensors to a GPU with x.cuda(), and a + b gives the same result through the same Python call, the same TensorIterator, the same broadcasting and type promotion. Only the last step changes, and it changes a lot: the loop that walked the elements becomes a kernel that tens of thousands of GPU threads run at once, on a device with its own memory, behind a bus. This article follows a + b from the call to that kernel and back, and the pieces PyTorch adds around it so that the GPU is not left waiting: launches that return at once, an allocator that almost never asks CUDA for memory, a division turned into a multiplication, and copies from the CPU that the CPU does not have to wait for.

Before you start. This article builds on the CPU side of the series: How PyTorch walks a tensor of any shape (strides, and the one division of its section 9), How PyTorch prepares the loop of every operation (the iterator, its merged dimensions and its strides in bytes) and How a PyTorch kernel uses the vector registers of every CPU (DispatchStub and the kernels compiled per CPU). What a tensor is introduced the allocators and pinned memory; here is how they work. No GPU programming is assumed: each term is explained where it first appears. The code is C++ and CUDA C++, which is C++ with a few additions for the GPU.

This is the path the article follows:

  1. What is different on a GPU.
  2. From a + b to a CUDA kernel.
  3. Launches return before the work is done.
  4. Memory: a cache in front of cudaMalloc.
  5. A thread per element.
  6. Any strides: an offset per thread.
  7. A division made a multiplication.
  8. How fast is it?.
  9. Getting the data there: pinned memory.
  10. What this article leaves out.

And at the end, the key takeaways.

The numbers come from raxiscuda.py, run in November 2026 with PyTorch 2.14.1 (CUDA 13.0) and Raxis 0.5.0, on Windows 11 25H2 with an RTX 5070 Ti (driver 610.88) next to the Ryzen 7 9700X of the other articles. The GPU numbers are from Windows only.

1. What is different on a GPU

A CPU core, as the interlude described it, runs one stream of instructions very fast, and has caches and a branch predictor to keep that one stream from waiting. A GPU makes the opposite trade. The RTX 5070 Ti has 70 streaming multiprocessors (SMs), each of which can keep up to 1,536 threads in flight, over a hundred thousand threads in total. Each thread is slow and simple; what makes the GPU fast is that when some threads wait for memory, others are ready to run. For an operation like a + b, where every element is independent, that is a perfect fit.

The code describes the work in three levels:

The code chooses only the grid and the block: kernel<<<blocks, threads per block>>>(...). Everything after that is done by the hardware, with no operating system involved. When a block reaches an SM, the hardware splits it into warps of 32 consecutive threads, the unit the SM really runs: a block of 128 threads is four warps, and block sizes are chosen as multiples of 32. The 32 threads of a warp run the same instruction at the same time, each on its own data, much like the lanes of a SIMD register on the CPU. When they read 32 neighbouring floats, the hardware merges the reads into a few wide memory transactions. That is called a coalesced access, and it is the GPU’s version of reading whole cache lines.

Inside the SM the warps are spread once more. The SM has four partitions, each with a warp scheduler, 32 lanes that compute, and 64 KiB of registers. Each warp of a block goes to one partition and stays there, with its registers in that partition’s register file, so a partition holds up to 12 warps (48 per SM, 1,536 threads), and an SM of this card holds at most 24 blocks. Every cycle, each scheduler picks one of its warps that is not waiting for anything and issues that warp’s next instruction, which the 32 lanes run, one per thread. The next cycle it can pick another warp: their registers are all there already, so switching costs nothing. That is how the GPU waits for memory: a warp that waits for a load leaves the lanes to the other warps of its partition.

So each level of the code lands on a level of the chip:

The codeThe chip
gridthe whole GPU: a queue of blocks and the unit that hands them out
blockone SM, whole, until it ends
warp (formed by the hardware)one partition and its scheduler
threadone lane

Compared with the CPU of the interlude, the pieces map like this, with differences that matter:

The GPU also has its own memory: 16 GiB of GDDR7 on this card, with about 896 GB/s of bandwidth, more than ten times what the CPU gets from its RAM. And between the two there is a bus, PCIe 5.0 with 16 lanes here, about 64 GB/s in each direction. So a GPU tensor lives in a different memory, the CPU cannot read it directly, and every byte that moves between the two crosses that bus.

Between the SMs and that memory there are caches, as on the CPU, with a few differences. Each SM has 128 KiB of fast memory that it splits between its L1 cache and the shared memory of its blocks, a memory that the threads of a block share and manage by hand: up to 100 KiB of it on this card, and a kernel like a + b, which uses none, leaves almost all of it to the L1. The L2, 48 MiB, is one for the whole chip, and everything the SMs read from the GDDR7 goes through it. Lines are 128 bytes in both, where the CPU’s are 64, but they are filled in sectors of 32 bytes: when a warp needs 4 bytes, 32 come from memory, not the whole line. And every level takes much longer to answer. This is the same chain of dependent reads as in the interlude, walked by a single GPU thread (raxislatency.cu):

Where the chain livesGPU (one thread)CPU of the interlude
L115 ns0.72 ns
L2121 ns2.53 ns
L3(none)9 to 10 ns
Memory289 ns81 to 100 ns

The chain fits in the L1 up to 80 KiB, in the L2 up to 32 or 40 MiB depending on the run, and from 48 MiB on it comes from the GDDR7. The GPU’s clock is half the CPU’s here, 2.9 GHz against 5.5, but that does not explain the gap: in cycles, the L1 answers in 44 instead of 4, and a read from the L2 takes longer than one from the CPU’s RAM. Reading the chain in order is exactly as slow as reading it at random, at every size, so there is no prefetcher bringing the next lines early, as the CPU had. A single GPU thread is slow at everything, memory included. The GPU hides the wait with numbers instead: while one warp waits 289 ns, the schedulers of the paragraph above run the others, and the memory serves hundreds of reads at the same time. That is how slow answers add up to 896 GB/s, and section 5 comes back to it.

Three questions follow, and they give this article its structure: how does an operation on a GPU tensor find GPU code; how is GPU memory managed; and how do the data get there without making the GPU wait.

How CUDA threads are grouped, and the GPU they run onTop: a kernel launch for a + b on 4096 by 4096 floats is a grid of 16,384 blocks; block 5 is drawn larger as 128 threads in four warps of 32; thread 1 of warp 0 loads elements 5124 to 5127 and 5636 to 5639. Bottom: the RTX 5070 Ti has 70 streaming multiprocessors. One SM is drawn larger: four partitions, each with a warp scheduler, 32 lanes and 64 KiB of registers that hold its 12 resident warps, one of them highlighted as the warp the scheduler issues for in this cycle, and an L1 cache with up to 100 KiB of shared memory. An SM keeps up to 48 warps, 1,536 threads, in at most 24 blocks. About 840 blocks of 128 threads fit on the GPU at once; the rest wait in a queue. All SMs share a 48 MiB L2 cache, in front of 16 GiB of GDDR7 at about 896 GB/s, and the card reaches the CPU through PCIe 5.0 x16 at about 64 GB/s each way.What a launch asks for: a + b on 4096 x 4096 floatsGrid: 16,384 blocks; about 840 fiton the SMs at once, the rest wait in a queuewarp 0warp 1warp 2warp 3Block 5: 128 threads, 4 warps of 32Warp 032 threads, oneinstruction at a time,each on its own dataThread 1elements 5124-5127and 5636-5639Where it runs: the RTX 5070 Ti and its 70 SMsOne SM (of 70): 4 x 12 = 48 warps resident1,536 threads in at most 24 blocks, each until it endswarp scheduler32 lanesregisters: 12 warpswarp scheduler32 lanesregisters: 12 warpswarp scheduler32 lanesregisters: 12 warpswarp scheduler32 lanesregisters: 12 warpsL1 cache and shared memory(up to 100 KiB shared by a block's threads)L2 cache, 48 MiB, shared by all SMsGDDR7, 16 GiB, about 896 GB/sPCIe 5.0 x16, about 64 GB/s each wayCPU and its RAM
The code describes the work as a grid of blocks of threads; the hardware runs each block on one SM, whose schedulers each issue one instruction per cycle, for one of their resident warps (the highlighted one), and the 32 lanes run it for the warp's 32 threads. Hardware numbers fromtorch.cuda.get_device_properties(0) on the RTX 5070 Ti.

And the same machinery in motion, on a model small enough to follow: a launch of 16 blocks, the queue, two SMs, and every cycle of their schedulers.

A kernel launch: the queue of blocks and two SMs, cycle by cycleAn animation of a scaled model. A launch of 16 blocks of 4 warps waits in a queue; two SMs take two blocks each. Each SM has two partitions with four warp places; every cycle, each partition's scheduler issues one instruction for one ready warp, while the warps that issued their loads wait 16 cycles for memory. When all the warps of a block are done, the block leaves its SM and the next block of the queue takes its place.issuing this cyclereadywaiting for memorydonefree placeadd_kernel<<<16 blocks, 128 threads>>>(a, b, out)Scaled model: 2 SMs of 2 partitions, 4 warp places each; a load takes 16 cycles. B = block, w = warp.The queue of blocksB0B1B2B3B4B5B6B7B8B9B10B11B12B13B14B15SM 0: blocks 0 and 2scheduler: B0 w2 addB0 w0doneB0 w2addB2 w0waitingB2 w2waitingscheduler: B0 w3 addB0 w1doneB0 w3addB2 w1waitingB2 w3waitingSM 1: blocks 1 and 3scheduler: B1 w2 addB1 w0doneB1 w2addB3 w0waitingB3 w2waitingscheduler: B1 w3 addB1 w1doneB1 w3addB3 w1waitingB3 w3waitingcycle 22 blocks done 0 of 16 scheduler cycles used 91%
A launch, cycle by cycle. The blocks wait in the queue until an SM has room, and a block that has started stays until all its warps are done. Each warp issues its loads one per cycle and then waits for memory; meanwhile its scheduler issues for the other warps of its partition, and it idles only when every one of them is waiting.

2. From a + b to a CUDA kernel

On the CPU, a + b ends in a call through a DispatchStub, the object of the kernels article that holds one function pointer per CPU instruction set and picks the right one at runtime (raxisbinary_ops.cpp):

add_stub(iter.device_type(), iter);

The first argument was there from the start: the device of the iterator’s tensors. The stub holds a second kind of slot, for CUDA (pytorchDispatchStub.h, raxisdispatch_stub.h):

std::atomic<void*> cpu_dispatch_ptr{nullptr};
void* cuda_dispatch_ptr = nullptr;

and the call picks the slot by device (raxisdispatch_stub.cpp). On the CPU it picks among the per-instruction-set kernels; on CUDA it takes the one CUDA kernel, or raises “add_stub: no CUDA kernel yet” if the slot is empty. Everything before that call is shared: the iterator has already broadcast the shapes, promoted the types, merged the dimensions and allocated the output, on the output’s device. The CUDA kernel receives the same TensorIterator the CPU kernel would.

Who fills the CUDA slot? The CUDA kernels live in their own library. In PyTorch that is libtorch_cuda, next to libtorch_cpu; in Raxis it is libraxis_cuda. Each .cu file registers its kernels with a line that looks exactly like the CPU one (raxisbinary_ops_kernel.cu):

void add_kernel_cuda(TensorIterator& iter) {
    iter.common_dtype().dispatch_all_types([&]<typename scalar_t>(std::type_identity<scalar_t>) {
        cuda::gpu_kernel_with_scalars(iter, AddFunctor<scalar_t>());
    });
}
REGISTER_DISPATCH(add_stub, &add_kernel_cuda);

When the CUDA compiler builds the file, REGISTER_DISPATCH becomes REGISTER_CUDA_DISPATCH (pytorchDispatchStub.h, raxisdispatch_stub.h): a static object whose constructor writes the kernel’s address into the stub’s CUDA slot. C++ runs the constructors of static objects when a library is loaded, so loading libraxis_cuda is what fills every CUDA slot. Nobody calls a registration function; the act of loading is the registration. The CUDA allocator joins the same way, with REGISTER_ALLOCATOR(DeviceType::CUDA, ...), which puts it in a table of allocators by device type that the factories read (raxisfactories.cpp); and so do the few things the CPU library needs to ask CUDA, such as “is a GPU available?” or “synchronize”, through an interface with a default implementation that answers “no CUDA” (pytorchCUDAHooksInterface.h, raxiscuda_hooks.h).

This is why the CPU library does not depend on CUDA at all, and why Raxis can ship its CUDA support as a separate package. pip install raxis installs the CPU library; pip install raxis-cuda13 adds a package that contains only libraxis_cuda, and import raxis loads it if it is there (raxis__init__.py). JAX does the same with its CUDA plugins; PyTorch instead builds one wheel per CUDA version, each with both libraries inside.

3. Launches return before the work is done

add_kernel_cuda ends in a kernel launch: the CPU asks the GPU to run a kernel with a given grid of blocks. How long does a + b take? It depends on what you measure:

MicrosecondsPyTorchRaxis
1024 floats: until a + b returns5.54.9
1024 floats: until the GPU has finished8.97.5
4096 x 4096 floats: until a + b returns5.44.8
4096 x 4096 floats: until the GPU has finished261260

For 16 million elements, the call comes back in 5 microseconds, while the GPU needs 260 to do the work. A launch does not wait for the kernel: it puts it in a queue and returns. That queue is a stream: the kernels and copies of one stream run on the GPU one after another, in the order they were queued, while the CPU goes on with the Python program. In a training step, the CPU queues the forward, the backward and the optimizer update, hundreds of kernels, while the GPU is still working on the first ones.

The CPU has to wait when it needs a result: loss.item(), x.cpu(), print(x), or a copy to the CPU. Each of these synchronizes: it waits until the stream has run everything queued before it. A training loop that prints the loss at every step, once per batch, synchronizes once per step, which is usually fine. A Python loop that calls .item() on each sample of a batch, to count the correct predictions one by one for example, synchronizes hundreds of times per step and throws the asynchrony away.

The small case shows the other side of it. For 1024 floats, the GPU work is a few microseconds and the cost is the launch itself. Called a thousand times in a row, each a + b costs 7.3 microseconds in PyTorch and 6.6 in Raxis, almost all of it the CPU’s work: the Python call, the iterator, the launch. A GPU kept busy by many tiny operations is limited by the CPU that launches them, which is why torch.compile and CUDA graphs, which merge or record launches, help most there.

4. Memory: a cache in front of cudaMalloc

The output of a + b needs GPU memory. CUDA’s own functions, cudaMalloc and cudaFree, are slow, and cudaFree also waits for the whole GPU to finish, because the memory could still be in use by a queued kernel. The script measures it: allocating an empty CUDA tensor, with the allocator’s cache emptied first so that the allocation has to come from cudaMalloc, and with the cache as it normally is:

Microseconds per empty(...) on the GPUPyTorchRaxis
64 MiB, from the cache1.71.8
64 MiB, from cudaMalloc9081
4 KiB, from the cache1.81.9
4 KiB, from cudaMalloc4340

A training step allocates and frees thousands of tensors. At 40 to 90 microseconds each, plus a full synchronization at every free, the GPU would spend more time waiting for the allocator than computing. So PyTorch never gives memory back to CUDA during normal work. Its caching allocator keeps freed memory and hands it to the next tensor that fits (pytorchCUDACachingAllocator.cpp, raxiscuda_caching_allocator.cpp). Raxis ports a simplified version of it, and the rules are few:

Follow three allocations on a fresh device, with sizes in MiB because the allocator works in powers of two. A tensor of 3 MiB, 3,145,728 bytes, is already a multiple of 512; it is over 1 MiB and under 10 MiB, so the allocator asks CUDA for a 20 MiB segment, takes the first 3 MiB and leaves a free block of 17 MiB. A second tensor of 5 MiB finds that block, the best fit, and splits it again: 5 MiB taken, 12 MiB free. Free the first one: its block has no neighbour on its left, and on its right is the 5 MiB tensor, so nothing merges, and the 3 MiB block waits in the pool. A third tensor of 2 MiB finds it as the best fit, and takes all of it: a large block is only split when more than 1 MiB would be left, so that the pool does not fill with slivers nobody can use, and here exactly 1 MiB would be. No cudaMalloc after the first one, and in a training loop, where every step allocates the same sizes as the step before, every tensor finds a block of the right size.

Three allocations in one 20 MiB segment of the caching allocatorFour bars of the same 20 MiB segment, to scale. First, a 3 MiB tensor: the allocator gets a 20 MiB segment from cudaMalloc, takes 3 MiB and leaves a free block of 17 MiB. Second, a 5 MiB tensor takes the 17 MiB block, the best fit, and leaves 12 MiB free. Third, the first tensor is freed: its block has no free neighbour, so nothing merges. Fourth, a 2 MiB tensor takes the free 3 MiB block whole, because the 1 MiB that would remain is too little to split; that 1 MiB stays unused inside the block.1. A 3 MiB tensorcudaMalloc of a 20 MiB segment, splitfirst, 3free, 17 MiB2. A 5 MiB tensorbest fit: the 17 MiB block, split againfirst, 3second, 5free, 12 MiB3. The first tensor is freedno free neighbour, nothing mergesfree, 3second, 5free, 12 MiB4. A 2 MiB tensorbest fit: the 3 MiB block, taken whole (1 MiB left is too little to split)third, 2second, 5free, 12 MiB0 MiB3 MiB8 MiB20 MiBblock of a tensorfree block, kept in the poolinside a tensor's block, unused
The example of this section, to scale: one cudaMalloc serves all three tensors. When a block is freed, it waits in the pool for the next tensor it fits.

This is also why PyTorch reports two numbers. torch.cuda.memory_allocated() counts the bytes in blocks handed to tensors; memory_reserved() counts the segments taken from CUDA, allocated or cached. nvidia-smi shows something closer to the second, so a program that freed all its tensors still appears to hold memory. For finding leaks, memory_allocated() is the honest number, and it is what the autograd article used to show a reference cycle keeping a graph alive.

Streams add one rule, in PyTorch and in Raxis alike. A freed block is only reused on the stream it was allocated on: anything queued there later runs after everything that used the block, so the reuse needs no waiting. A tensor that another stream also used must say so with record_stream; when it is freed, the allocator records an event on that stream, and the block waits until the event completes. What Raxis leaves out of PyTorch’s allocator: expandable segments, the pools of CUDA graphs, and the allocator’s settings.

5. A thread per element

Now the kernel itself. On the CPU, cpu_kernel gave each thread a long chunk of the elements and walked it. On the GPU, each thread handles a handful of elements, and there are as many threads as needed to cover all of them. The function that does this is gpu_kernel, and it receives the same lambda-like object as its CPU counterpart: AddFunctor<float>, whose operator() returns a + b (pytorchLoops.cuh, raxisloops.cuh). It chooses between two kernels (pytorchCUDALoops.cuh, raxisloops.cuh):

The vectorized kernel first. With NN = 16,777,216 floats (4096 x 4096), and three operands of 4 bytes, PyTorch gives each thread 8 elements and each block 128 threads, so a block covers 1,024 elements, and the grid has 16,384 blocks. Each thread moves its elements with vector loads and stores of 16 bytes, four floats per instruction, much as a CPU kernel fills a SIMD register (pytorchCUDALoops.cuh, raxisloops.cuh):

const int block_start = kBlockWork * blockIdx.x;          // 1024 * block number
for (int i = 0; i < kLoops; i++) {                        // 2 vectors per thread
    vec_t v = from[threadIdx.x + i * kNumThreads];        // vector t, then t + 128
    ...
}

blockIdx.x and threadIdx.x are CUDA’s names for “which block am I” and “which thread am I in my block”. These are the elements that some threads of block 5 load from a:

ThreadFirst vectorElementsSecond vectorElements
005120 to 51231285632 to 5635
115124 to 51271295636 to 5639
225128 to 51311305640 to 5643
31315244 to 52471595756 to 5759

Each thread’s 8 elements are its two vectors of four. Only the loads and stores are 16 bytes wide: the thread then adds its 8 pairs one by one, in an unrolled loop, because a GPU thread has no instruction that adds four floats at once. The arithmetic is vectorized across threads instead: one add issued for a warp is 32 adds, one per lane. What the wide loads save is instructions, one load for four floats instead of four.

The vectors are numbered from the start of the block, at element 5 x 1024 = 5120. Why does thread 1 take vector 1 and not vector 2, which would keep each thread’s 8 elements together? Because of the warp: threads 0 to 31 execute each load at the same time, and with this layout they read 32 consecutive vectors, 512 consecutive bytes, which the hardware serves as four 128-byte cache lines. If each thread took two neighbouring vectors, the warp’s first load would touch every other vector, spread over eight lines instead of four, and the second load would come back to the same lines for the other half: the same data for twice the traffic between the caches and the threads. Consecutive threads on consecutive addresses is the rule of every fast GPU kernel.

The vectors one warp loads, with two ways of giving vectors to threadsA strip of 64 vectors of 16 bytes, 1024 bytes in eight 128-byte lines. With PyTorch's layout, thread t of the warp loads vector t: the first load of the 32 threads covers vectors 0 to 31, four whole lines, and the next 32 vectors belong to the next warp. If instead thread t loaded vectors 2t and 2t + 1, the first load would cover the even vectors 0 to 62, spread over all eight lines with half of each line unused, and the second load would read the same eight lines again for the odd vectors.PyTorch: thread t loads vector t, then vector t + 128line 0line 1line 2line 3line 4line 5line 6line 7t0t1t2t31first load: 4 lines, all usedA worse layout: thread t loads vectors 2t and 2t + 1line 0line 1line 2line 3line 4line 5line 6line 7t0t1t2t31first load: 8 lines, half of each; the second reads them again0 B256 B512 B768 B1024 Bthe warp's first loadits second loadloaded by the next warp
32 threads, one load each: with PyTorch's layout they read 512 consecutive bytes, four whole 128-byte lines. With the worse layout, the same load is spread over eight lines and uses half of each, and the second load reads them again.

The last block may not be full: NN does not have to be a multiple of 1,024. That block falls back to one element per thread with a bounds check, the GPU version of the scalar tail of a CPU loop. And the vectors are only as wide as the addresses allow: a 16-byte load must start at a multiple of 16 bytes, so the kernel checks every pointer first, and uses vectors of two floats, or one, when an operand does not start on that boundary (raxisloops.cuh).

Why 128 threads and 8 elements? The block size is a multiple of 32, so no warp runs with idle lanes, and 128 fills an SM on every recent NVIDIA GPU: here 12 blocks reach the 1,536 threads, well within the limit of 24 blocks per SM, while blocks of 32 threads would hit that limit at 768 threads and leave half of the SM’s warp places empty. The 8 elements per thread are about memory. A load from the GDDR7 takes about 300 nanoseconds (section 1), so to keep 896 GB/s flowing the GPU needs 896 GB/s x 289 ns, about 260 KB, requested at any moment, and more under load, when the latency grows, so each thread issues the loads of all its elements before it needs the first one. PyTorch’s code says it plainly, in a comment for a newer GPU: “Target 128 bytes of input per thread to keep roughly 128 KiB in flight per SM” (pytorchCUDALoops.cuh). More elements per thread would mean more registers per thread and fewer blocks to spread. The grid is then NN divided by the 1,024 elements of a block, rounded up: PyTorch does not look at how many SMs the card has, and leaves the spreading to the queue.

6. Any strides: an offset per thread

When the operands are not all contiguous, the vectorized kernel cannot run: a 16-byte load needs four contiguous floats, as a SIMD load on the CPU does. The generic kernel loads one element at a time, from any address. On the CPU that is the scalar loop, which is why PyTorch vectorizes there only strides of one element or zero; on the GPU it costs no extra instructions, because one load of a warp is already 32 loads from 32 addresses, one per lane. What the strides change is how many lines those 32 addresses touch.

Take x + row, with x of shape [3, 4] and row of shape [4]. The iterator broadcasts row with a stride of 0 and stores the dimensions with the fastest one first, as the TensorIterator article explained. In bytes, for float32:

SizesOutput stridesx stridesrow strides
Dimension 04444
Dimension 1316160

Each thread of the generic kernel computes 4 elements, interleaved as in the vectorized kernel: thread i of the first block takes elements i, i + 128, i + 256 and i + 384. With only 12 elements, thread i computes element i. To find where that element is in each operand, it splits i into one index per dimension, from the fastest one up, and adds index times stride: this is the OffsetCalculator (pytorchOffsetCalculator.cuh, raxisoffset_calculator.cuh):

for (int dim = 0; dim < dims; ++dim) {
    auto divmod = sizes_[dim].divmod(linear_idx);    // quotient and remainder
    linear_idx = divmod.div;
    for (int arg = 0; arg < NARGS; arg++) {
        offsets[arg] += divmod.mod * strides_[dim][arg];
    }
}

For thread 7: 7 divided by 4 is 1, remainder 3, so its index in dimension 0 is 3; then 1 divided by 3 is 0, remainder 1, so its index in dimension 1 is 1. Its offsets are 3×4+1×16=283 \times 4 + 1 \times 16 = 28 bytes in the output and in x, and 3×4+1×0=123 \times 4 + 1 \times 0 = 12 bytes in row: element [1, 3] of x plus element [3] of row.

ThreadIndex in dim 0Index in dim 1Offset in x and the outputOffset in row
00000
3301212
401160
7312812
11324412

This is exactly the unravelling that the article on n-dimensional loops measured as the slow way, a division and a remainder per dimension for every element. The CPU avoided it with a counter that carries from one dimension to the next, and only divides once per thread, where its chunk starts (section 9 of that article). A GPU thread handles one to eight elements, so its chunk start is almost every element: the division cannot be avoided. It can be made cheap, though.

The same kernel also handles operands whose dtype differs from the functor’s: it reads each one through a switch on its runtime dtype and converts it, instead of compiling the functor for every combination of input and output types (raxisloops.cuh).

7. A division made a multiplication

An integer division is one of the slowest things a GPU thread can do: there is no division instruction for integers, and the compiler expands it into a sequence of tens of instructions. But here every divisor is a size of the tensor, known on the CPU before the launch. For a fixed divisor dd, division can be replaced by a multiplication and a shift, with a “magic number” mm and a shift ss computed once (pytorchIntegerDivider.cuh, raxisinteger_divider.cuh):

s=⌈log⁡2d⌉,m=⌊232 (2s−d)d⌋+1,⌊nd⌋=(⌊n m232⌋+n)≫ss = \lceil \log_2 d \rceil, \qquad m = \left\lfloor \frac{2^{32}\,(2^s - d)}{d} \right\rfloor + 1, \qquad \left\lfloor \frac{n}{d} \right\rfloor = \left( \left\lfloor \frac{n\,m}{2^{32}} \right\rfloor + n \right) \gg s

⌊n m/232⌋\lfloor n\,m / 2^{32} \rfloor is the upper half of the 64-bit product of two 32-bit numbers, which the GPU computes in one instruction, __umulhi; then an addition and a shift. The idea is from Torbjorn Granlund and Peter Montgomery’s 1994 paper, “Division by Invariant Integers using Multiplication”, and compilers use it for every division by a constant. These are some divisors as the script computes them:

ddssmmnn⌊nm/232⌋\lfloor nm/2^{32} \rfloorquotientnn // dd
321,431,655,766100333333
1,00010103,079,216123,4562,962123123
4,09612110,000,00002,4412,441
4,097134,292,870,65610,000,0009,995,1182,4402,440

For dd = 1,000: (2,962+123,456)≫10=126,418/1,024=123.45(2{,}962 + 123{,}456) \gg 10 = 126{,}418 / 1{,}024 = 123.45, and the shift keeps 123. For a power of two, mm is 1, the upper half is 0, and what is left is a plain shift. The script also checks the formula against integer division for 270,000 values of nn per divisor, up to 2312^{31}.

Why up to 2312^{31}? Because the formula is exact only while nn and the offsets fit in 31 bits. That is why every CUDA kernel of PyTorch indexes with 32-bit integers, which are also cheaper for the address arithmetic, and why gpu_kernel first checks whether the iterator fits: a tensor with more than 2312^{31} elements, or offsets past 2 GB, is split in halves along its largest dimension, and each half is launched on its own (raxisloops.cuh).

8. How fast is it?

The script runs a + b on 4096 x 4096 float32, 64 MiB per tensor, in four ways, and counts the bytes each one must move:

a + b, 4096 x 4096 float32PyTorch, msGB/sRaxis, msGB/s
contiguous0.2567870.256786
both inputs one element off the 16-byte alignment0.2567870.256787
a plus a broadcast row of 40960.1757670.175768
a plus b.t(), transposed0.3885180.355567

The contiguous addition moves 192 MiB in 0.256 ms, 787 GB/s, 88% of the card’s 896 GB/s: there is little left to gain in the kernel, and PyTorch and Raxis run the same one. The broadcast row reads only a and writes the output, 128 MiB, through the generic kernel with its offsets and divisions, and still gets 767 GB/s. Two things in the table I did not expect:

9. Getting the data there: pinned memory

So far the data were already on the GPU. In training, every batch starts on the CPU and has to cross the PCIe bus. Whatever can be created on the GPU, random initial weights or a mask, should be created there, with device="cuda", and never crosses it. The GPU copies with its own copy engine, by DMA (direct memory access): it reads the CPU’s memory itself, without the CPU taking part. For that, the memory must stay at the same physical address during the whole copy. Normal memory does not promise that: the operating system can move a page, or swap it to disk. Pinned (page-locked) memory is CPU memory that the operating system has promised not to move, as the first article introduced.

When the source is normal, pageable memory, the driver copies it in two steps: the CPU first copies the data into a pinned buffer of the driver’s own, a staging buffer, and the copy engine sends it from there. The script copies 64 MiB to the GPU, and measures how long the call takes to return:

Milliseconds until the call returns, 64 MiBPyTorchRaxis
pageable, blocking3.4453.340
pinned, blocking1.1861.214
pageable, non_blocking=True3.2303.298
pinned, non_blocking=True0.0050.004
pin_memory(), block from the cache3.1793.228
pin_memory(), cache emptied first7.6327.621
clone() on the CPU, for scale3.1743.127

From pinned memory, the copy runs at 64 MiB in 1.19 ms, 56 GB/s, close to what PCIe 5.0 x16 can carry. From pageable memory it takes three times as long. The driver stages the data in chunks, and the copy engine sends each chunk while the CPU copies the next one, so the two steps overlap; but the CPU copies at the speed of a clone(), and the copy engine spends most of the time waiting for it. The non_blocking=True rows are the important ones. From pinned memory, the call queues the copy on the stream and returns in 5 microseconds, and the CPU can do something else while the copy engine works. From pageable memory, non_blocking changes almost nothing: the call can only return once all the data are in the staging buffer, because until then the program could change or free them, and all it saves is the wait for the last chunk to cross the bus.

Where each kind of memory lives, which cache holds it, and who moves the bytes in each of these copies:

Where the memory of a copy to the GPU lives, and who moves itTwo boxes joined by the PCIe bus. On the CPU side, in the RAM: a pageable tensor, a pinned block of the host caching allocator, and the driver's staging buffer, which is pinned too but not visible to the program. On the GPU side: the SMs and the L2 cache, the copy engine, and the GDDR7 memory, divided into segments of the CUDA caching allocator, split into blocks of tensors. Arrow 1, pin_memory: the CPU copies the pageable tensor into the pinned block. Arrow 2: the copy engine reads the pinned block across the bus and writes into a block in the GDDR7. Arrow 3: from pageable memory the CPU first copies into the staging buffer, and the copy engine reads from there.CPUcores: a CPU copy is a memcpyRAMpageablea tensor, as usualpinned blockhost caching allocatorstaging bufferthe driver'sPinned and staging: RAM the operating system will notmove. The program reads a pinned block like any memory;only the driver uses the staging buffer.PCIe 5.0 x16~64 GB/seach wayGPU70 SMsL2 cache, 48 MiBcopyengine(DMA)GDDR7, 16 GiBCUDA caching allocator:segments from cudaMalloc,split into tensor blocksthe destination block132the CPU copiesthe copy engine copies, by DMA1x.pin_memory()the CPU copies x into a pinned block: 3.2 ms for 64 MiB2pinned x.to("cuda")the copy engine reads the block by DMA: 1.19 ms; with non_blocking the call returns in 5 microseconds3pageable x.to("cuda")the CPU copies into the staging buffer in chunks, the copy engine sends each one: 3.4 ms
A copy to the GPU touches two caches, one on each side of the bus: pinned blocks in the CPU's RAM, and blocks of the GDDR7. Only the copy engine crosses the bus; what the CPU does is copy within its own RAM, into a pinned block or into the driver's staging buffer.

But pinning has a cost too: pin_memory() is a copy into pinned memory, the same 3 ms as the staging. The gain is not in the number of copies, which is the same, but in who does it and when. For a single copy, pinning first is even slower: 3.2 ms to pin plus 1.19 ms to copy makes 4.4 ms, against 3.4 ms straight from pageable memory, where the driver overlaps its staging with the copy engine, chunk by chunk. Pinning pays when its copy is out of the way: done ahead of time by another thread, as the DataLoader does, or not needed at all, when the tensor is created in pinned memory or one pinned buffer is refilled and sent again and again. And pinned memory is itself slow to get from the system, 7.6 ms instead of 3.2 ms when the allocator has no block to reuse, which is why PyTorch keeps a second caching allocator, for pinned memory (pytorchCachingHostAllocator.h, raxiscuda_host_allocator.cpp). It rounds sizes to powers of two and keeps freed blocks in a list per size, with one twist. With non_blocking=True, the Python code can drop the pinned tensor while the copy engine is still reading it; if its block went straight back to the list, the next batch could be written into it before the copy ends. So the copy records its stream on the block (pytorchCopy.cu, raxiscopy_kernel.cu), freeing it places an event, a marker that the GPU signals when the stream reaches it, and the block is reused only after the event has completed.

PyTorch’s DataLoader puts all of this together. With num_workers=2, two worker processes load batches in turn, prefetch_factor batches each ahead of the training loop (2 by default); with pin_memory=True, a thread in the main process copies each batch into pinned memory as it arrives (pytorchdataloader.py). The training loop then only has to call x.to("cuda", non_blocking=True), which returns at once, and launch the step. As timelines, with durations in the proportions measured above:

A training loop without and with DataLoader workers and pinned memoryTwo timelines on the same time scale. Without workers or pinned memory, the main thread loads each batch, waits for the GPU to finish the previous step, stages the copy into the driver's buffer and launches the kernels; the GPU is idle while the next batch loads, and a step takes 9 units. With two worker processes and a pin thread, the batches are loaded and pinned while the GPU computes; the main thread only queues an asynchronous copy and launches the kernels, and the GPU runs one step after another, about 7 units each.load a batchCPU waitsCPU stages the copypin the batchlaunch kernelscopy to the GPUkernelsqueue the copy, returns at onceWithout: num_workers=0, pageable memoryMain threadCopy engineGPU kernelsidleidleidleload 1stagecopystep 1load 2waitstagecopystep 2load 3waitstagecopystep 3load 4waitstagecopyWith: num_workers=2, pin_memory=TrueWorker 0Worker 1Pin threadMain threadCopy engineGPU kernelsload 1load 3load 5load 2load 4load 6pinpinpinpinpinpinwaitstep 1step 2step 3step 4time
Without workers the GPU waits for every batch to be loaded and staged. With them, loading and pinning overlap the GPU's work, and the main thread is back from each copy in microseconds. The durations are illustrative, with the proportions measured in this section.

Without workers, each step is a chain: load the batch, wait for the GPU to finish the previous step (the copy is queued behind it on the same stream), stage, launch, and only then does the GPU start, so it sits idle while the next batch loads. With workers and pinned memory, loading and pinning happen on other cores while the GPU computes, and the GPU goes from one step to the next. Raxis has the pieces, pin_memory(), pin_memory=True in the factories and non_blocking, but no DataLoader; its nanoGPT example pins each batch itself before copying it.

10. What this article leaves out

The CUDA backend of Raxis covers every operation of the library, and most of it follows the same pattern as a + b: a stub, a kernel registered from libraxis_cuda, the same iterator. What does not fit in one article:

Reductions on the GPU. A sum over a GPU tensor splits the work across threads, warps and blocks, adds within a warp by passing values between its threads (warp shuffles), and combines blocks through a second pass. It is the GPU version of the reductions article, and a possible topic for a later one.

User streams. With one stream, a copy and a kernel never overlap on the GPU, as the timeline shows: the copy of the next batch waits for the current step. A second stream for copies lets them overlap. Raxis has them as PyTorch does, raxis.cuda.Stream, Event and record_stream:

copy_stream = raxis.cuda.Stream()
with raxis.cuda.stream(copy_stream):
    x = batch.to("cuda", non_blocking=True)    # batch is pinned
raxis.cuda.current_stream().wait_stream(copy_stream)
x.record_stream(raxis.cuda.current_stream())

wait_stream makes the work queued next on the default stream wait for the copy, and record_stream tells the allocator that the default stream uses x, a block of the copy stream (section 4).

Also left out: the attention, convolution and matrix multiplication kernels; CUDA graphs; and the packaging of the CUDA wheels, with their machine code for several GPU generations.

Key takeaways

Further reading

If something is not clear, or you think I got something wrong, leave a comment.