How PyTorch runs the same operations on a GPU
pgmesa · · 40 min read
Part of Rebuilding PyTorch (8 of 8)
- Why I rebuilt PyTorch's core from scratch
- What a PyTorch tensor really is
- Interlude: What the CPU does when your code reads memory
- How PyTorch walks a tensor of any shape
- How PyTorch prepares the loop of every operation
- How a PyTorch kernel uses the vector registers of every CPU
- How PyTorch sums a tensor, and splits the work between threads
- How PyTorch computes gradients
- How PyTorch runs the same operations on a GPU
Every operation in this series so far ran on the CPU. Move the tensors to a GPU with x.cuda(), and a + b gives the same result through the same Python call, the same TensorIterator, the same broadcasting and type promotion. Only the last step changes, and it changes a lot: the loop that walked the elements becomes a kernel that tens of thousands of GPU threads run at once, on a device with its own memory, behind a bus. This article follows a + b from the call to that kernel and back, and the pieces PyTorch adds around it so that the GPU is not left waiting: launches that return at once, an allocator that almost never asks CUDA for memory, a division turned into a multiplication, and copies from the CPU that the CPU does not have to wait for.
Before you start. This article builds on the CPU side of the series: How PyTorch walks a tensor of any shape (strides, and the one division of its section 9), How PyTorch prepares the loop of every operation (the iterator, its merged dimensions and its strides in bytes) and How a PyTorch kernel uses the vector registers of every CPU (
DispatchStuband the kernels compiled per CPU). What a tensor is introduced the allocators and pinned memory; here is how they work. No GPU programming is assumed: each term is explained where it first appears. The code is C++ and CUDA C++, which is C++ with a few additions for the GPU.
This is the path the article follows:
- What is different on a GPU.
- From
a + bto a CUDA kernel. - Launches return before the work is done.
- Memory: a cache in front of
cudaMalloc. - A thread per element.
- Any strides: an offset per thread.
- A division made a multiplication.
- How fast is it?.
- Getting the data there: pinned memory.
- What this article leaves out.
And at the end, the key takeaways.
The numbers come from raxiscuda.py, run in November 2026 with PyTorch 2.14.1 (CUDA 13.0) and Raxis 0.5.0, on Windows 11 25H2 with an RTX 5070 Ti (driver 610.88) next to the Ryzen 7 9700X of the other articles. The GPU numbers are from Windows only.
1. What is different on a GPU
A CPU core, as the interlude described it, runs one stream of instructions very fast, and has caches and a branch predictor to keep that one stream from waiting. A GPU makes the opposite trade. The RTX 5070 Ti has 70 streaming multiprocessors (SMs), each of which can keep up to 1,536 threads in flight, over a hundred thousand threads in total. Each thread is slow and simple; what makes the GPU fast is that when some threads wait for memory, others are ready to run. For an operation like a + b, where every element is independent, that is a perfect fit.
The code describes the work in three levels:
- A thread runs the kernel once. Every thread runs the same code and finds its own data from its index: in the simplest kernel for
a + b, threadiaddsa[i]andb[i]. - A block is a group of threads, up to 1,024, that run on the same SM at the same time, can share a small fast memory and can wait for each other at a barrier. PyTorch’s elementwise kernels use blocks of 128 threads.
- A grid is all the blocks of one kernel launch. A kernel for 16 million elements has thousands of blocks, far more than fit on the GPU at once. The hardware places blocks on an SM while it has room, and the rest wait in a queue until a block finishes and frees its place.
The code chooses only the grid and the block: kernel<<<blocks, threads per block>>>(...). Everything after that is done by the hardware, with no operating system involved. When a block reaches an SM, the hardware splits it into warps of 32 consecutive threads, the unit the SM really runs: a block of 128 threads is four warps, and block sizes are chosen as multiples of 32. The 32 threads of a warp run the same instruction at the same time, each on its own data, much like the lanes of a SIMD register on the CPU. When they read 32 neighbouring floats, the hardware merges the reads into a few wide memory transactions. That is called a coalesced access, and it is the GPU’s version of reading whole cache lines.
Inside the SM the warps are spread once more. The SM has four partitions, each with a warp scheduler, 32 lanes that compute, and 64 KiB of registers. Each warp of a block goes to one partition and stays there, with its registers in that partition’s register file, so a partition holds up to 12 warps (48 per SM, 1,536 threads), and an SM of this card holds at most 24 blocks. Every cycle, each scheduler picks one of its warps that is not waiting for anything and issues that warp’s next instruction, which the 32 lanes run, one per thread. The next cycle it can pick another warp: their registers are all there already, so switching costs nothing. That is how the GPU waits for memory: a warp that waits for a load leaves the lanes to the other warps of its partition.
So each level of the code lands on a level of the chip:
| The code | The chip |
|---|---|
| grid | the whole GPU: a queue of blocks and the unit that hands them out |
| block | one SM, whole, until it ends |
| warp (formed by the hardware) | one partition and its scheduler |
| thread | one lane |
Compared with the CPU of the interlude, the pieces map like this, with differences that matter:
- A partition is like a small core, and its warps are its hardware threads. A resident warp is what a hardware thread is on the CPU: the state of a running program kept on the chip, its registers and its next instruction. A CPU core keeps two and runs instructions from both in the same cycle, out of order; a partition keeps twelve, runs each warp’s instructions in order, and issues for one warp per cycle. With no out-of-order execution to hide a wait, it relies on having another warp ready.
- A CUDA thread is a lane, closer to a SIMD lane than to a CPU thread. Unlike a SIMD lane it has its own registers, can read from any address, and can take its own branch: when the threads of a warp take different branches, the warp runs both, each with the lanes that did not take it switched off, as a CPU SIMD loop does with masks. NVIDIA calls this SIMT, single instruction, multiple threads.
- A block is the nearest thing to a software thread: a unit of work that waits for a place to run. The difference is what happens once it runs. The operating system takes turns between software threads on a core, saving the registers of one in memory and loading those of the next, many times a second. A block that has started stays on its SM until it ends and is not paused for another, and a block still in the queue has no registers yet, so there is nothing to save. On the GPU, work waits before it starts, never in the middle.
The GPU also has its own memory: 16 GiB of GDDR7 on this card, with about 896 GB/s of bandwidth, more than ten times what the CPU gets from its RAM. And between the two there is a bus, PCIe 5.0 with 16 lanes here, about 64 GB/s in each direction. So a GPU tensor lives in a different memory, the CPU cannot read it directly, and every byte that moves between the two crosses that bus.
Between the SMs and that memory there are caches, as on the CPU, with a few differences. Each SM has 128 KiB of fast memory that it splits between its L1 cache and the shared memory of its blocks, a memory that the threads of a block share and manage by hand: up to 100 KiB of it on this card, and a kernel like a + b, which uses none, leaves almost all of it to the L1. The L2, 48 MiB, is one for the whole chip, and everything the SMs read from the GDDR7 goes through it. Lines are 128 bytes in both, where the CPU’s are 64, but they are filled in sectors of 32 bytes: when a warp needs 4 bytes, 32 come from memory, not the whole line. And every level takes much longer to answer. This is the same chain of dependent reads as in the interlude, walked by a single GPU thread (raxislatency.cu):
| Where the chain lives | GPU (one thread) | CPU of the interlude |
|---|---|---|
| L1 | 15 ns | 0.72 ns |
| L2 | 121 ns | 2.53 ns |
| L3 | (none) | 9 to 10 ns |
| Memory | 289 ns | 81 to 100 ns |
The chain fits in the L1 up to 80 KiB, in the L2 up to 32 or 40 MiB depending on the run, and from 48 MiB on it comes from the GDDR7. The GPU’s clock is half the CPU’s here, 2.9 GHz against 5.5, but that does not explain the gap: in cycles, the L1 answers in 44 instead of 4, and a read from the L2 takes longer than one from the CPU’s RAM. Reading the chain in order is exactly as slow as reading it at random, at every size, so there is no prefetcher bringing the next lines early, as the CPU had. A single GPU thread is slow at everything, memory included. The GPU hides the wait with numbers instead: while one warp waits 289 ns, the schedulers of the paragraph above run the others, and the memory serves hundreds of reads at the same time. That is how slow answers add up to 896 GB/s, and section 5 comes back to it.
Three questions follow, and they give this article its structure: how does an operation on a GPU tensor find GPU code; how is GPU memory managed; and how do the data get there without making the GPU wait.
torch.cuda.get_device_properties(0) on the RTX 5070 Ti.And the same machinery in motion, on a model small enough to follow: a launch of 16 blocks, the queue, two SMs, and every cycle of their schedulers.
2. From a + b to a CUDA kernel
On the CPU, a + b ends in a call through a DispatchStub, the object of the kernels article that holds one function pointer per CPU instruction set and picks the right one at runtime (raxisbinary_ops.cpp):
add_stub(iter.device_type(), iter);
The first argument was there from the start: the device of the iterator’s tensors. The stub holds a second kind of slot, for CUDA (pytorchDispatchStub.h, raxisdispatch_stub.h):
std::atomic<void*> cpu_dispatch_ptr{nullptr};
void* cuda_dispatch_ptr = nullptr;
and the call picks the slot by device (raxisdispatch_stub.cpp). On the CPU it picks among the per-instruction-set kernels; on CUDA it takes the one CUDA kernel, or raises “add_stub: no CUDA kernel yet” if the slot is empty. Everything before that call is shared: the iterator has already broadcast the shapes, promoted the types, merged the dimensions and allocated the output, on the output’s device. The CUDA kernel receives the same TensorIterator the CPU kernel would.
Who fills the CUDA slot? The CUDA kernels live in their own library. In PyTorch that is libtorch_cuda, next to libtorch_cpu; in Raxis it is libraxis_cuda. Each .cu file registers its kernels with a line that looks exactly like the CPU one (raxisbinary_ops_kernel.cu):
void add_kernel_cuda(TensorIterator& iter) {
iter.common_dtype().dispatch_all_types([&]<typename scalar_t>(std::type_identity<scalar_t>) {
cuda::gpu_kernel_with_scalars(iter, AddFunctor<scalar_t>());
});
}
REGISTER_DISPATCH(add_stub, &add_kernel_cuda);
When the CUDA compiler builds the file, REGISTER_DISPATCH becomes REGISTER_CUDA_DISPATCH (pytorchDispatchStub.h, raxisdispatch_stub.h): a static object whose constructor writes the kernel’s address into the stub’s CUDA slot. C++ runs the constructors of static objects when a library is loaded, so loading libraxis_cuda is what fills every CUDA slot. Nobody calls a registration function; the act of loading is the registration. The CUDA allocator joins the same way, with REGISTER_ALLOCATOR(DeviceType::CUDA, ...), which puts it in a table of allocators by device type that the factories read (raxisfactories.cpp); and so do the few things the CPU library needs to ask CUDA, such as “is a GPU available?” or “synchronize”, through an interface with a default implementation that answers “no CUDA” (pytorchCUDAHooksInterface.h, raxiscuda_hooks.h).
This is why the CPU library does not depend on CUDA at all, and why Raxis can ship its CUDA support as a separate package. pip install raxis installs the CPU library; pip install raxis-cuda13 adds a package that contains only libraxis_cuda, and import raxis loads it if it is there (raxis__init__.py). JAX does the same with its CUDA plugins; PyTorch instead builds one wheel per CUDA version, each with both libraries inside.
3. Launches return before the work is done
add_kernel_cuda ends in a kernel launch: the CPU asks the GPU to run a kernel with a given grid of blocks. How long does a + b take? It depends on what you measure:
| Microseconds | PyTorch | Raxis |
|---|---|---|
1024 floats: until a + b returns | 5.5 | 4.9 |
| 1024 floats: until the GPU has finished | 8.9 | 7.5 |
4096 x 4096 floats: until a + b returns | 5.4 | 4.8 |
| 4096 x 4096 floats: until the GPU has finished | 261 | 260 |
For 16 million elements, the call comes back in 5 microseconds, while the GPU needs 260 to do the work. A launch does not wait for the kernel: it puts it in a queue and returns. That queue is a stream: the kernels and copies of one stream run on the GPU one after another, in the order they were queued, while the CPU goes on with the Python program. In a training step, the CPU queues the forward, the backward and the optimizer update, hundreds of kernels, while the GPU is still working on the first ones.
The CPU has to wait when it needs a result: loss.item(), x.cpu(), print(x), or a copy to the CPU. Each of these synchronizes: it waits until the stream has run everything queued before it. A training loop that prints the loss at every step, once per batch, synchronizes once per step, which is usually fine. A Python loop that calls .item() on each sample of a batch, to count the correct predictions one by one for example, synchronizes hundreds of times per step and throws the asynchrony away.
The small case shows the other side of it. For 1024 floats, the GPU work is a few microseconds and the cost is the launch itself. Called a thousand times in a row, each a + b costs 7.3 microseconds in PyTorch and 6.6 in Raxis, almost all of it the CPU’s work: the Python call, the iterator, the launch. A GPU kept busy by many tiny operations is limited by the CPU that launches them, which is why torch.compile and CUDA graphs, which merge or record launches, help most there.
4. Memory: a cache in front of cudaMalloc
The output of a + b needs GPU memory. CUDA’s own functions, cudaMalloc and cudaFree, are slow, and cudaFree also waits for the whole GPU to finish, because the memory could still be in use by a queued kernel. The script measures it: allocating an empty CUDA tensor, with the allocator’s cache emptied first so that the allocation has to come from cudaMalloc, and with the cache as it normally is:
Microseconds per empty(...) on the GPU | PyTorch | Raxis |
|---|---|---|
| 64 MiB, from the cache | 1.7 | 1.8 |
64 MiB, from cudaMalloc | 90 | 81 |
| 4 KiB, from the cache | 1.8 | 1.9 |
4 KiB, from cudaMalloc | 43 | 40 |
A training step allocates and frees thousands of tensors. At 40 to 90 microseconds each, plus a full synchronization at every free, the GPU would spend more time waiting for the allocator than computing. So PyTorch never gives memory back to CUDA during normal work. Its caching allocator keeps freed memory and hands it to the next tensor that fits (pytorchCUDACachingAllocator.cpp, raxiscuda_caching_allocator.cpp). Raxis ports a simplified version of it, and the rules are few:
- Sizes are rounded up to 512 bytes, so that blocks are reusable by tensors of slightly different sizes.
- Memory comes from CUDA in segments, one
cudaMalloceach, of fixed sizes: 2 MiB for allocations up to 1 MiB, shared by many small tensors; 20 MiB for allocations under 10 MiB; and a segment of its own, rounded to 2 MiB, for anything larger (pytorchAllocatorConfig.h, raxiscuda_caching_allocator.cpp). - A segment is split into blocks. A new allocation takes the smallest free block that fits, the best fit, found in a
std::setordered by stream and size, and splits off the rest as a new free block (raxiscuda_caching_allocator.cpp). - A freed block merges with its free neighbours in the same segment, so the segment can become whole again (raxis
cuda_caching_allocator.cpp). - Small and large blocks live in separate pools, so that the many short-lived small tensors do not cut up the segments where the large ones live.
- Memory goes back to CUDA only with
torch.cuda.empty_cache(), or whencudaMallocfails: then the allocator frees every cached segment that is whole and tries once more before raising “CUDA out of memory”.
Follow three allocations on a fresh device, with sizes in MiB because the allocator works in powers of two. A tensor of 3 MiB, 3,145,728 bytes, is already a multiple of 512; it is over 1 MiB and under 10 MiB, so the allocator asks CUDA for a 20 MiB segment, takes the first 3 MiB and leaves a free block of 17 MiB. A second tensor of 5 MiB finds that block, the best fit, and splits it again: 5 MiB taken, 12 MiB free. Free the first one: its block has no neighbour on its left, and on its right is the 5 MiB tensor, so nothing merges, and the 3 MiB block waits in the pool. A third tensor of 2 MiB finds it as the best fit, and takes all of it: a large block is only split when more than 1 MiB would be left, so that the pool does not fill with slivers nobody can use, and here exactly 1 MiB would be. No cudaMalloc after the first one, and in a training loop, where every step allocates the same sizes as the step before, every tensor finds a block of the right size.
cudaMalloc serves all three tensors. When a block is freed, it waits in the pool for the next tensor it fits.This is also why PyTorch reports two numbers. torch.cuda.memory_allocated() counts the bytes in blocks handed to tensors; memory_reserved() counts the segments taken from CUDA, allocated or cached. nvidia-smi shows something closer to the second, so a program that freed all its tensors still appears to hold memory. For finding leaks, memory_allocated() is the honest number, and it is what the autograd article used to show a reference cycle keeping a graph alive.
Streams add one rule, in PyTorch and in Raxis alike. A freed block is only reused on the stream it was allocated on: anything queued there later runs after everything that used the block, so the reuse needs no waiting. A tensor that another stream also used must say so with record_stream; when it is freed, the allocator records an event on that stream, and the block waits until the event completes. What Raxis leaves out of PyTorch’s allocator: expandable segments, the pools of CUDA graphs, and the allocator’s settings.
5. A thread per element
Now the kernel itself. On the CPU, cpu_kernel gave each thread a long chunk of the elements and walked it. On the GPU, each thread handles a handful of elements, and there are as many threads as needed to cover all of them. The function that does this is gpu_kernel, and it receives the same lambda-like object as its CPU counterpart: AddFunctor<float>, whose operator() returns a + b (pytorchLoops.cuh, raxisloops.cuh). It chooses between two kernels (pytorchCUDALoops.cuh, raxisloops.cuh):
- Contiguous operands of the functor’s own types, the common case after the iterator has merged dimensions: a vectorized kernel.
- Anything else, broadcast or permuted operands, or a dtype that has to be converted: a generic kernel that computes each element’s position in every operand.
The vectorized kernel first. With = 16,777,216 floats (4096 x 4096), and three operands of 4 bytes, PyTorch gives each thread 8 elements and each block 128 threads, so a block covers 1,024 elements, and the grid has 16,384 blocks. Each thread moves its elements with vector loads and stores of 16 bytes, four floats per instruction, much as a CPU kernel fills a SIMD register (pytorchCUDALoops.cuh, raxisloops.cuh):
const int block_start = kBlockWork * blockIdx.x; // 1024 * block number
for (int i = 0; i < kLoops; i++) { // 2 vectors per thread
vec_t v = from[threadIdx.x + i * kNumThreads]; // vector t, then t + 128
...
}
blockIdx.x and threadIdx.x are CUDA’s names for “which block am I” and “which thread am I in my block”. These are the elements that some threads of block 5 load from a:
| Thread | First vector | Elements | Second vector | Elements |
|---|---|---|---|---|
| 0 | 0 | 5120 to 5123 | 128 | 5632 to 5635 |
| 1 | 1 | 5124 to 5127 | 129 | 5636 to 5639 |
| 2 | 2 | 5128 to 5131 | 130 | 5640 to 5643 |
| 31 | 31 | 5244 to 5247 | 159 | 5756 to 5759 |
Each thread’s 8 elements are its two vectors of four. Only the loads and stores are 16 bytes wide: the thread then adds its 8 pairs one by one, in an unrolled loop, because a GPU thread has no instruction that adds four floats at once. The arithmetic is vectorized across threads instead: one add issued for a warp is 32 adds, one per lane. What the wide loads save is instructions, one load for four floats instead of four.
The vectors are numbered from the start of the block, at element 5 x 1024 = 5120. Why does thread 1 take vector 1 and not vector 2, which would keep each thread’s 8 elements together? Because of the warp: threads 0 to 31 execute each load at the same time, and with this layout they read 32 consecutive vectors, 512 consecutive bytes, which the hardware serves as four 128-byte cache lines. If each thread took two neighbouring vectors, the warp’s first load would touch every other vector, spread over eight lines instead of four, and the second load would come back to the same lines for the other half: the same data for twice the traffic between the caches and the threads. Consecutive threads on consecutive addresses is the rule of every fast GPU kernel.
The last block may not be full: does not have to be a multiple of 1,024. That block falls back to one element per thread with a bounds check, the GPU version of the scalar tail of a CPU loop. And the vectors are only as wide as the addresses allow: a 16-byte load must start at a multiple of 16 bytes, so the kernel checks every pointer first, and uses vectors of two floats, or one, when an operand does not start on that boundary (raxisloops.cuh).
Why 128 threads and 8 elements? The block size is a multiple of 32, so no warp runs with idle lanes, and 128 fills an SM on every recent NVIDIA GPU: here 12 blocks reach the 1,536 threads, well within the limit of 24 blocks per SM, while blocks of 32 threads would hit that limit at 768 threads and leave half of the SM’s warp places empty. The 8 elements per thread are about memory. A load from the GDDR7 takes about 300 nanoseconds (section 1), so to keep 896 GB/s flowing the GPU needs 896 GB/s x 289 ns, about 260 KB, requested at any moment, and more under load, when the latency grows, so each thread issues the loads of all its elements before it needs the first one. PyTorch’s code says it plainly, in a comment for a newer GPU: “Target 128 bytes of input per thread to keep roughly 128 KiB in flight per SM” (pytorchCUDALoops.cuh). More elements per thread would mean more registers per thread and fewer blocks to spread. The grid is then divided by the 1,024 elements of a block, rounded up: PyTorch does not look at how many SMs the card has, and leaves the spreading to the queue.
6. Any strides: an offset per thread
When the operands are not all contiguous, the vectorized kernel cannot run: a 16-byte load needs four contiguous floats, as a SIMD load on the CPU does. The generic kernel loads one element at a time, from any address. On the CPU that is the scalar loop, which is why PyTorch vectorizes there only strides of one element or zero; on the GPU it costs no extra instructions, because one load of a warp is already 32 loads from 32 addresses, one per lane. What the strides change is how many lines those 32 addresses touch.
Take x + row, with x of shape [3, 4] and row of shape [4]. The iterator broadcasts row with a stride of 0 and stores the dimensions with the fastest one first, as the TensorIterator article explained. In bytes, for float32:
| Sizes | Output strides | x strides | row strides | |
|---|---|---|---|---|
| Dimension 0 | 4 | 4 | 4 | 4 |
| Dimension 1 | 3 | 16 | 16 | 0 |
Each thread of the generic kernel computes 4 elements, interleaved as in the vectorized kernel: thread i of the first block takes elements i, i + 128, i + 256 and i + 384. With only 12 elements, thread i computes element i. To find where that element is in each operand, it splits i into one index per dimension, from the fastest one up, and adds index times stride: this is the OffsetCalculator (pytorchOffsetCalculator.cuh, raxisoffset_calculator.cuh):
for (int dim = 0; dim < dims; ++dim) {
auto divmod = sizes_[dim].divmod(linear_idx); // quotient and remainder
linear_idx = divmod.div;
for (int arg = 0; arg < NARGS; arg++) {
offsets[arg] += divmod.mod * strides_[dim][arg];
}
}
For thread 7: 7 divided by 4 is 1, remainder 3, so its index in dimension 0 is 3; then 1 divided by 3 is 0, remainder 1, so its index in dimension 1 is 1. Its offsets are bytes in the output and in x, and bytes in row: element [1, 3] of x plus element [3] of row.
| Thread | Index in dim 0 | Index in dim 1 | Offset in x and the output | Offset in row |
|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 |
| 3 | 3 | 0 | 12 | 12 |
| 4 | 0 | 1 | 16 | 0 |
| 7 | 3 | 1 | 28 | 12 |
| 11 | 3 | 2 | 44 | 12 |
This is exactly the unravelling that the article on n-dimensional loops measured as the slow way, a division and a remainder per dimension for every element. The CPU avoided it with a counter that carries from one dimension to the next, and only divides once per thread, where its chunk starts (section 9 of that article). A GPU thread handles one to eight elements, so its chunk start is almost every element: the division cannot be avoided. It can be made cheap, though.
The same kernel also handles operands whose dtype differs from the functor’s: it reads each one through a switch on its runtime dtype and converts it, instead of compiling the functor for every combination of input and output types (raxisloops.cuh).
7. A division made a multiplication
An integer division is one of the slowest things a GPU thread can do: there is no division instruction for integers, and the compiler expands it into a sequence of tens of instructions. But here every divisor is a size of the tensor, known on the CPU before the launch. For a fixed divisor , division can be replaced by a multiplication and a shift, with a “magic number” and a shift computed once (pytorchIntegerDivider.cuh, raxisinteger_divider.cuh):
is the upper half of the 64-bit product of two 32-bit numbers, which the GPU computes in one instruction, __umulhi; then an addition and a shift. The idea is from Torbjorn Granlund and Peter Montgomery’s 1994 paper, “Division by Invariant Integers using Multiplication”, and compilers use it for every division by a constant. These are some divisors as the script computes them:
| quotient | // | |||||
|---|---|---|---|---|---|---|
| 3 | 2 | 1,431,655,766 | 100 | 33 | 33 | 33 |
| 1,000 | 10 | 103,079,216 | 123,456 | 2,962 | 123 | 123 |
| 4,096 | 12 | 1 | 10,000,000 | 0 | 2,441 | 2,441 |
| 4,097 | 13 | 4,292,870,656 | 10,000,000 | 9,995,118 | 2,440 | 2,440 |
For = 1,000: , and the shift keeps 123. For a power of two, is 1, the upper half is 0, and what is left is a plain shift. The script also checks the formula against integer division for 270,000 values of per divisor, up to .
Why up to ? Because the formula is exact only while and the offsets fit in 31 bits. That is why every CUDA kernel of PyTorch indexes with 32-bit integers, which are also cheaper for the address arithmetic, and why gpu_kernel first checks whether the iterator fits: a tensor with more than elements, or offsets past 2 GB, is split in halves along its largest dimension, and each half is launched on its own (raxisloops.cuh).
8. How fast is it?
The script runs a + b on 4096 x 4096 float32, 64 MiB per tensor, in four ways, and counts the bytes each one must move:
a + b, 4096 x 4096 float32 | PyTorch, ms | GB/s | Raxis, ms | GB/s |
|---|---|---|---|---|
| contiguous | 0.256 | 787 | 0.256 | 786 |
| both inputs one element off the 16-byte alignment | 0.256 | 787 | 0.256 | 787 |
a plus a broadcast row of 4096 | 0.175 | 767 | 0.175 | 768 |
a plus b.t(), transposed | 0.388 | 518 | 0.355 | 567 |
The contiguous addition moves 192 MiB in 0.256 ms, 787 GB/s, 88% of the card’s 896 GB/s: there is little left to gain in the kernel, and PyTorch and Raxis run the same one. The broadcast row reads only a and writes the output, 128 MiB, through the generic kernel with its offsets and divisions, and still gets 767 GB/s. Two things in the table I did not expect:
- The misaligned inputs are as fast. One element off, no operand starts on a 16-byte boundary, so the kernel uses loads of one float instead of four, and the time does not change. I also tried uint8 and tensors small enough to stay in the GPU’s 48 MiB L2 cache, and found no difference either. Vector loads issue fewer instructions for the same bytes, and here the memory is the limit, not the instructions; I did not find a case on this GPU where they change the time.
- The transposed input is slower, and Raxis is faster there than PyTorch. In
a + b.t(), consecutive threads readbwith a stride of 4096 floats, so a warp’s 32 reads touch 32 different 128-byte segments instead of one: the reads are not coalesced. The caches absorb part of it, but it still costs a third of the bandwidth. I have not looked into why Raxis’s generic kernel does better than PyTorch’s here.
9. Getting the data there: pinned memory
So far the data were already on the GPU. In training, every batch starts on the CPU and has to cross the PCIe bus. Whatever can be created on the GPU, random initial weights or a mask, should be created there, with device="cuda", and never crosses it. The GPU copies with its own copy engine, by DMA (direct memory access): it reads the CPU’s memory itself, without the CPU taking part. For that, the memory must stay at the same physical address during the whole copy. Normal memory does not promise that: the operating system can move a page, or swap it to disk. Pinned (page-locked) memory is CPU memory that the operating system has promised not to move, as the first article introduced.
When the source is normal, pageable memory, the driver copies it in two steps: the CPU first copies the data into a pinned buffer of the driver’s own, a staging buffer, and the copy engine sends it from there. The script copies 64 MiB to the GPU, and measures how long the call takes to return:
| Milliseconds until the call returns, 64 MiB | PyTorch | Raxis |
|---|---|---|
| pageable, blocking | 3.445 | 3.340 |
| pinned, blocking | 1.186 | 1.214 |
pageable, non_blocking=True | 3.230 | 3.298 |
pinned, non_blocking=True | 0.005 | 0.004 |
pin_memory(), block from the cache | 3.179 | 3.228 |
pin_memory(), cache emptied first | 7.632 | 7.621 |
clone() on the CPU, for scale | 3.174 | 3.127 |
From pinned memory, the copy runs at 64 MiB in 1.19 ms, 56 GB/s, close to what PCIe 5.0 x16 can carry. From pageable memory it takes three times as long. The driver stages the data in chunks, and the copy engine sends each chunk while the CPU copies the next one, so the two steps overlap; but the CPU copies at the speed of a clone(), and the copy engine spends most of the time waiting for it. The non_blocking=True rows are the important ones. From pinned memory, the call queues the copy on the stream and returns in 5 microseconds, and the CPU can do something else while the copy engine works. From pageable memory, non_blocking changes almost nothing: the call can only return once all the data are in the staging buffer, because until then the program could change or free them, and all it saves is the wait for the last chunk to cross the bus.
Where each kind of memory lives, which cache holds it, and who moves the bytes in each of these copies:
But pinning has a cost too: pin_memory() is a copy into pinned memory, the same 3 ms as the staging. The gain is not in the number of copies, which is the same, but in who does it and when. For a single copy, pinning first is even slower: 3.2 ms to pin plus 1.19 ms to copy makes 4.4 ms, against 3.4 ms straight from pageable memory, where the driver overlaps its staging with the copy engine, chunk by chunk. Pinning pays when its copy is out of the way: done ahead of time by another thread, as the DataLoader does, or not needed at all, when the tensor is created in pinned memory or one pinned buffer is refilled and sent again and again. And pinned memory is itself slow to get from the system, 7.6 ms instead of 3.2 ms when the allocator has no block to reuse, which is why PyTorch keeps a second caching allocator, for pinned memory (pytorchCachingHostAllocator.h, raxiscuda_host_allocator.cpp). It rounds sizes to powers of two and keeps freed blocks in a list per size, with one twist. With non_blocking=True, the Python code can drop the pinned tensor while the copy engine is still reading it; if its block went straight back to the list, the next batch could be written into it before the copy ends. So the copy records its stream on the block (pytorchCopy.cu, raxiscopy_kernel.cu), freeing it places an event, a marker that the GPU signals when the stream reaches it, and the block is reused only after the event has completed.
PyTorch’s DataLoader puts all of this together. With num_workers=2, two worker processes load batches in turn, prefetch_factor batches each ahead of the training loop (2 by default); with pin_memory=True, a thread in the main process copies each batch into pinned memory as it arrives (pytorchdataloader.py). The training loop then only has to call x.to("cuda", non_blocking=True), which returns at once, and launch the step. As timelines, with durations in the proportions measured above:
Without workers, each step is a chain: load the batch, wait for the GPU to finish the previous step (the copy is queued behind it on the same stream), stage, launch, and only then does the GPU start, so it sits idle while the next batch loads. With workers and pinned memory, loading and pinning happen on other cores while the GPU computes, and the GPU goes from one step to the next. Raxis has the pieces, pin_memory(), pin_memory=True in the factories and non_blocking, but no DataLoader; its nanoGPT example pins each batch itself before copying it.
10. What this article leaves out
The CUDA backend of Raxis covers every operation of the library, and most of it follows the same pattern as a + b: a stub, a kernel registered from libraxis_cuda, the same iterator. What does not fit in one article:
Reductions on the GPU. A sum over a GPU tensor splits the work across threads, warps and blocks, adds within a warp by passing values between its threads (warp shuffles), and combines blocks through a second pass. It is the GPU version of the reductions article, and a possible topic for a later one.
User streams. With one stream, a copy and a kernel never overlap on the GPU, as the timeline shows: the copy of the next batch waits for the current step. A second stream for copies lets them overlap. Raxis has them as PyTorch does,
raxis.cuda.Stream,Eventandrecord_stream:copy_stream = raxis.cuda.Stream() with raxis.cuda.stream(copy_stream): x = batch.to("cuda", non_blocking=True) # batch is pinned raxis.cuda.current_stream().wait_stream(copy_stream) x.record_stream(raxis.cuda.current_stream())
wait_streammakes the work queued next on the default stream wait for the copy, andrecord_streamtells the allocator that the default stream usesx, a block of the copy stream (section 4).
Also left out: the attention, convolution and matrix multiplication kernels; CUDA graphs; and the packaging of the CUDA wheels, with their machine code for several GPU generations.
Key takeaways
- A GPU tensor takes the CPU’s path until the last step (section 2): the stub’s CUDA slot is filled when
libraxis_cudais loaded. - A launch returns in about 5 microseconds here (section 3), and the CPU waits only when it needs a result. The caching allocator keeps the 40 to 90 microseconds of
cudaMallocout of the way (section 4). - Consecutive threads read consecutive addresses (section 5), and the divisions of the offsets are multiplications (section 7):
a + bruns at 88% of the RTX 5070 Ti’s bandwidth (section 8). - Pinned memory takes the CPU out of the copy (section 9): 56 GB/s here, and
non_blocking=Truereturns at once.
Further reading
- NVIDIA’s CUDA Programming Guide, from the source: its programming model (SMs, threads, blocks, grids and warps) and asynchronous execution (streams, events and synchronization).
- The PyTorch notes on CUDA semantics (asynchronous execution, streams, the caching allocator and its settings), and the tutorial A guide on good usage of non_blocking and pin_memory().
- Zachary DeVito, A guide to PyTorch’s CUDA Caching Allocator (2022), by one of its authors, with the stream rules this article skips.
- Torbjorn Granlund and Peter L. Montgomery, “Division by Invariant Integers using Multiplication”, PLDI 1994: the magic numbers of section 7, and how compilers use them.
- Edward Yang’s PyTorch developer podcast, the episodes “Just enough CUDA to be dangerous” and “Asynchronous versus synchronous execution” (2021).
If something is not clear, or you think I got something wrong, leave a comment.