How a PyTorch kernel uses the vector registers of every CPU
pgmesa · · 25 min read
Part of Rebuilding PyTorch (5 of 8)
- Why I rebuilt PyTorch's core from scratch
- What a PyTorch tensor really is
- Interlude: What the CPU does when your code reads memory
- How PyTorch walks a tensor of any shape
- How PyTorch prepares the loop of every operation
- How a PyTorch kernel uses the vector registers of every CPU
- How PyTorch sums a tensor, and splits the work between threads
- How PyTorch computes gradients
- How PyTorch runs the same operations on a GPU
The last two articles built everything around a kernel: TensorIterator prepares the shape, the types and the strides, and the loop walks them in 2-D blocks without a single division. What is left is the code that runs inside each block, the kernel itself, and that is where the last factor of the speed is decided. An addition is one instruction on eight numbers at once, if the code is written for it. An exp is not an instruction at all, and someone has to write it so that it runs on eight numbers at once too. And all of that has to work on the CPU that the program finds when it starts, which may have 128, 256 or 512-bit registers.
Before you start. This article builds on How PyTorch walks a tensor of any shape, especially its section on the compiler, SIMD and
VectorizedLoop2d, and on How PyTorch prepares the loop of every operation. What SIMD registers are, and which ones each CPU has, is in the interlude on the hardware. Here there is more C++ than in the previous ones: templates, lambdas and a few SIMD intrinsics. Each piece is explained, and the tables say what it does with real values.
This is the path the article follows:
And at the end, the key takeaways.
All the timings are from a Ryzen 7 9700X, a Zen 5 with AVX-512, on one thread, with a million float32 numbers and outputs allocated beforehand (out=). They come from raxiskernels.py, which runs every operation in NumPy, PyTorch and Raxis, and in PyTorch and Raxis once per instruction set, forcing it as section 6 explains. I ran it in November 2026, with NumPy 2.5.4, PyTorch 2.14.1 and Raxis 0.5.0, on Windows 11 25H2 (MSVC 19.44) and Ubuntu 24.04 under WSL2 (GCC 13.3), on the same machine; the conclusions are the same, and the few numbers that differ a lot are pointed out.
1. What a kernel is
Back to the end of the last article: add built its TensorIterator and then called add_stub(iter.device_type(), iter). The stub is a table with one function per device and per instruction set, section 6 explains how it is filled, and the function it calls is the kernel. This is Raxis’s, for every type (raxisbinary_ops_kernel.cpp):
void add_kernel(TensorIterator& iter) {
iter.common_dtype().dispatch_all_types([&]<typename scalar_t>(std::type_identity<scalar_t>) {
cpu_kernel_vec(iter,
[](scalar_t a, scalar_t b) -> scalar_t { return a + b; }, // op
[](Vectorized<scalar_t> a, Vectorized<scalar_t> b) { return a + b; }); // vop
});
}
Three things happen here, and two of them we have already seen:
dispatch_all_typesturns the common dtype of the iterator into a C++ type,scalar_t, with aswitch. That was in the first article.cpu_kernel_vecgives the iterator a loop,VectorizedLoop2d, andfor_eachruns it on 2-D blocks, on as many threads as it decides. That was the second.- The two lambdas are new: what to do with one element,
op, and what to do with one vector of elements,vop. That is all an operation writes.
There is also a cpu_kernel, with op alone (pytorchLoops.h, raxisloops.h). It runs the plain loop of the next section on every block, and leaves any vectorization to the compiler. Kernels use it when a vector version is not worth writing, or not possible: a comparison that writes bools from floats, an operation that mixes types, or one with branches that would make a vector version slower than the scalar one.
2. One lambda for any number of inputs
add takes two inputs, exp one and where three. The loops are written once for all of them, so they have to find out from the lambda how many inputs it has, and of what types. That is the job of function_traits (pytorchFunctionTraits.h, raxisfunction_traits.h), a template that takes the type of the lambda’s operator() apart:
auto op = [](float a, float b) -> float { return a + b; };
using traits = function_traits<decltype(op)>;
traits::arity // 2
traits::result_type // float
traits::arg<0>::type // float
With that, the plain loop can load each argument from its own pointer, with its own stride and its own type, and call the lambda. This is the heart of basic_loop, simplified (pytorchLoops.h, raxisloops.h):
// From basic_loop and dereference, simplified: data[0] is the output,
// data[1..] the inputs, each with its stride in bytes
for (; i < n; i++) {
auto args = std::make_tuple(
load<arg<0>::type>(data[1] + i * strides[1]),
load<arg<1>::type>(data[2] + i * strides[2])); // one per argument, generated
*(result_type*)(data[0] + i * strides[0]) = std::apply(op, args);
}
The tuple is not written by hand: a std::index_sequence of the argument positions expands into one load per argument when the template is instantiated. So for add the compiler sees exactly the loop from the last article, with three pointers and three strides, and no tuple left in the generated code. This is also the loop that runs when the strides are not contiguous, the one the compiler does not vectorize.
3. The vectorized loop, and its tail
The last article ended inside VectorizedLoop2d: when every inner stride is the size of an element, or all but one input that has a stride of 0, each row goes through vectorized_loop (pytorchLoops.h, raxisloops.h). Here is its body, almost word for word:
Vec opt_scalar = Vec(S > 0 ? load<scalar_t>(data[S]) : scalar_t(0)); // the stride-0 input
int64_t i = 0;
for (; i <= n - 2 * Vec::size(); i += 2 * Vec::size()) {
auto args1 = dereference_vec<traits>(&data[1], opt_scalar, S, i);
auto args2 = dereference_vec<traits>(&data[1], opt_scalar, S, i + Vec::size());
auto out1 = std::apply(vop, std::move(args1));
auto out2 = std::apply(vop, std::move(args2));
out1.store(data[0] + i * sizeof(scalar_t));
out2.store(data[0] + (i + Vec::size()) * sizeof(scalar_t));
}
if (i < n) {
basic_loop(data, strides, i, n, op); // the tail, one element at a time
}
Three details explain most of its speed:
- Two vectors per iteration. A vector instruction takes a few cycles to produce its result, but the CPU can start another one every cycle, as long as it does not depend on the first. Two independent vectors per iteration give it two chains of work to overlap, which is enough for a simple operation like an addition to be limited by memory and not by the instructions.
- The stride-0 input is loaded once.
opt_scalarholds its value repeated in every lane of a register, anddereference_vechands it tovopin its position instead of loading from memory. This is the broadcast of the last two articles, with no extra work per element. - The tail. What does not fill two vectors at the end of the row goes through
basic_loop, withop, one element at a time.
With rows of 1000 floats, the size of the examples of the series:
| Instruction set | Floats per vector | Per iteration | Vector iterations | Tail |
|---|---|---|---|---|
| AVX2 | 8 | 16 | 62 (992 floats) | 8 |
| AVX-512 | 16 | 32 | 31 (992 floats) | 8 |
The tail is small here. It is not small when the rows are short, which is one of the reasons that the rows of 4 from the last article cost so much: 4 floats never fill a vector, so the whole row was a tail. And it is the reason the two lambdas must compute the same thing: the same row mixes elements computed by vop and by op, and a user should not be able to tell which was which.
And a row with any other inner stride, like the 2 of x[::2], never gets here: VectorizedLoop2d sends it whole to basic_loop, one element at a time. That is not a case PyTorch forgot. A vector load reads consecutive floats only, and the instructions that read scattered ones, the gathers, cost about as much as reading them one by one, as the interlude on the hardware explains. That is most of the factor of three that x[::2] costs in the stride table of the same article.
4. Vectorized<T>: one register as a type
vop works on Vectorized<float>, and the last article only showed one of its operators. It is worth seeing the whole idea, because it is what lets the same kernel compile for every CPU.
The generic version is a plain array, with loops for every operation (pytorchvec_base.h, raxisvec_base.h):
template <typename T>
struct Vectorized {
static constexpr int kSize = VECTOR_WIDTH / sizeof(T); // 32 bytes: 8 floats, 4 doubles
alignas(VECTOR_WIDTH) T values_[kSize];
// operator+ adds values_[i] in a loop, exp() calls std::exp on each, ...
};
It works for any type and any CPU, and the compiler may or may not turn its loops into vector instructions. For the types and instruction sets that matter there are specializations, which replace the array with a real register and every operation with the instruction that does it. These are the three versions of Vectorized<float> in Raxis, and PyTorch has the same three, among others:
| Header | When it is used | Register | Floats | operator+ |
|---|---|---|---|---|
vec256/vec256_float.h | AVX2 build | __m256 | 8 | _mm256_add_ps |
vec512/vec512_float.h | AVX-512 build | __m512 | 16 | _mm512_add_ps |
vec128/vec128_float.h | ARM64 (NEON) | float32x4_t | 4 | vaddq_f32 |
(raxisvec256_float.h, raxisvec512_float.h, raxisvec128_float.h; PyTorch’s AVX2 one is pytorchvec256_float.h.)
Which one a file gets is decided by the preprocessor: vec.h includes the AVX2 headers only when the file is compiled with CPU_CAPABILITY_AVX2, the AVX-512 ones with CPU_CAPABILITY_AVX512, and the NEON ones on ARM64. In section 6 we will see that each kernel file is compiled several times, once with each of those definitions. The kernel’s source never changes: a + b on two Vectorized<float> becomes _mm256_add_ps in one build and _mm512_add_ps in another.
Two more things the specializations need, and both are visible in vec256_float.h (raxisvec256_float.h, pytorchvec256_float.h):
- Loads and stores of fewer elements,
loadu(ptr, count)andstore(ptr, count), through a small aligned buffer. Some kernels use them to process their own tails withvop. - Comparisons return masks, not
bools.a > bon two registers gives a register whose lanes are all ones where it is true and all zeros where it is false, and a “blend” instruction picks, lane by lane, from one register or another with that mask. That is how vector code does anifwithout branching: it computes both sides and keeps one, per lane.
5. Functions that are not one instruction
An addition is one instruction. exp is not: no x86 or ARM CPU has an instruction that computes the exponential of eight floats. So Vectorized<float>::exp() has to be written with additions, multiplications and bit tricks, for eight lanes at a time and without branches, because each lane may need a different path and a vector has only one.
The trick behind almost every implementation is the same: split into a part that powers of two handle exactly, and a small remainder that a polynomial handles well.
costs nothing: a float is stored as a sign, an exponent and a mantissa, so is the number with exponent and mantissa zero, built by writing into the exponent bits. And on the small interval of , a polynomial of degree 5 approximates to the precision of a float. PyTorch’s exp_u20 does exactly that, with 20 ULP of error at most (pytorchvec256_float.h, raxisvec256_float.h). An ULP, unit in the last place, is the distance between a float and the next one, so 20 ULP means that the last few bits may be wrong:
// From exp_u20, simplified (each line runs on 8 lanes at once)
x = min(max(x, ln(FLT_MIN)), ln(FLT_MAX)); // clamp instead of branching
n = floor(x * log2(e) + 0.5); // round(x / ln 2)
r = x - n * ln2; // the small remainder
p = 1 + r * (c1 + r * (c2 + r * (c3 + r * (c4 + r * c5)))); // e^r, 5 FMAs
two_n = (int(n) + 127) << 23; // 2^n, written in the exponent bits
return p * as_float(two_n);
Follow it with :
| Step | Value |
|---|---|
| 4.328 | |
| 4 | |
| 0.227411 | |
| polynomial, | 1.255335 |
| , exponent bits | 16 |
| result, | 20.0854 |
and . (The real code computes and multiplies by 2 at the end, so that does not overflow the exponent, and it puts a 0 in the lanes where was too small, with a blend: the mask is the if.)
Every line is a vector instruction with no branches, so the 8 lanes, or 16 with AVX-512, go through it together. A scalar exp from the C library does a similar computation for one number, with branches for the special cases and more care for the last bit.
Who implements these functions. PyTorch, and Raxis with it, takes them from three places:
- SLEEF, a library of vectorized math functions for every instruction set, with versions named by their error:
Sleef_expf8_u10isexpfor 8 floats with at most 1.0 ULP of error.Vectorized<float>::exp()calls it (pytorchvec256_float.h, raxisvec256_float.h).- Its own polynomials, when speed matters more than the last bit:
exp_u20for the softmax inside flash attention, or a polynomialerffor GELU.- MKL’s VML, when PyTorch is built with it: whole contiguous arrays at once (
vsExp), with MKL choosing its own instruction set.torch.exp,tanhorsinon the CPU go there.
Now the cost. sigmoid is , and its vop uses Vectorized::exp(), so it shows the whole difference between a scalar exp and a vector one. In the generic Vectorized of the DEFAULT build, exp() calls std::exp lane by lane; in the AVX2 and AVX-512 builds it is SLEEF with 8 and 16 lanes:
| A million floats, Windows | DEFAULT | AVX2 | AVX-512 |
|---|---|---|---|
PyTorch sigmoid | 2.37 ms | 0.47 ms | 0.22 ms |
Raxis sigmoid | 2.53 ms | 0.49 ms | 0.21 ms |
PyTorch exp (MKL VML) | 0.46 ms | 0.46 ms | 0.47 ms |
Raxis exp (MKL VML) | 0.46 ms | 0.46 ms | 0.46 ms |
From the scalar exp to 8 lanes, 5 times faster; from 8 to 16 lanes, 2.2 times more. This is the kind of operation where the width of the registers pays in full, because the work is in the instructions and the memory has time to keep up. And exp does not change at all with the instruction set of PyTorch’s build, because MKL picks its own.
On Linux, with GCC and glibc, the DEFAULT column changes. PyTorch’s sigmoid takes 1.26 ms there, because the scalar expf of Linux’s C library, glibc, is faster than the one on Windows, while Raxis stays at 2.6 ms, because its generic exp() converts each float to double and calls the double version (PyTorch’s calls the float one). It only matters on x86 CPUs without AVX2. The other difference is NumPy, which has its own vector versions of these functions: on Linux its tanh takes 0.12 ms, against 1.58 ms for MKL’s in PyTorch and Raxis, and its exp 0.25 ms against 0.46. On Windows the gap is smaller (tanh 0.74 ms in NumPy). I have not looked into where MKL loses that time on this CPU, and it is the first thing I would look at to make these operations faster.
6. One binary for every CPU
So the same kernel is worth 5 times more with AVX2 than without, and twice more again with AVX-512 on some operations. But a program compiled with AVX2 instructions crashes on a CPU that does not have them, with an “illegal instruction” error, and a wheel on PyPI has to run on any x86-64 CPU, from the first ones of the 2000s to those of 2026. Compiling for the minimum wastes the registers of every recent CPU, and compiling for the newest leaves out the older ones.
PyTorch’s answer has three pieces.
Each kernel file is compiled several times. The build system takes every file of kernels and compiles it once per instruction set, with that set’s compiler flags and a definition with its name: /arch:AVX2 or -mavx2 -mfma and CPU_CAPABILITY=AVX2, /arch:AVX512 or -mavx512f -mavx512bw -mavx512vl -mavx512dq and CPU_CAPABILITY=AVX512, and no flags for CPU_CAPABILITY=DEFAULT (pytorchCodegen.cmake, raxisDispatchCompile.cmake). Raxis does it with one small generated file per kernel and per set, which only #includes the real one (raxisDispatchCompile.cmake):
set(wrapper "${dispatch_dir}/${name}.${CPU_CAPABILITY}.cpp") # binary_ops_kernel.AVX2.cpp
file(WRITE "${wrapper}" "#include \"${abs_src}\"\n") # the same source
set_source_files_properties(${wrapper} PROPERTIES
COMPILE_FLAGS "${FLAGS} -DCPU_CAPABILITY=${CPU_CAPABILITY} -DCPU_CAPABILITY_${CPU_CAPABILITY}")
Three copies of add_kernel in the same library would clash, so every kernel lives in inline namespace CPU_CAPABILITY { ... }, which the definition turns into namespace AVX2, namespace AVX512 or namespace DEFAULT. That is the inline namespace CPU_CAPABILITY at the top of loops.h and of every Vectorized header.
Each copy registers itself. At the end of the kernel file, REGISTER_DISPATCH(add_stub, &add_kernel) stores the address of that copy in the slot of its set: in the AVX2 build it fills add_stub’s AVX2 slot, in the DEFAULT build the DEFAULT one (pytorchDispatchStub.h, raxisdispatch_stub.h). These are static variables initialized when the library loads, which is why the static build of Raxis needs to link the whole archive: otherwise the linker drops these files, which nothing references, and every stub is empty.
The stub picks one, once. The first call to add_stub asks which instruction set the CPU has, takes the best slot that is filled, and keeps the pointer for every later call (pytorchDispatchStub.cpp, raxisdispatch_stub.cpp). Asking the CPU takes two instructions:
cpuidtells what the CPU supports: AVX2 and FMA in leaf 7 and 1, and for AVX-512 its foundation (F) and the extensions PyTorch requires (DQ, BW, VL).xgetbvtells what the operating system supports. A CPU can have AVX-512 and still not be usable if the OS does not save the 512-bit registers when it switches threads, so the check reads the register XCR0 and requires the bits of the YMM state for AVX2, and of the ZMM and mask registers for AVX-512.
PyTorch asks through cpuinfo, a library that does the same and more; Raxis writes the two calls. Either way, you can see the result, and you can force a lower set with an environment variable before the first operation, which is how the tables of this article were made:
>>> import torch, raxis
>>> torch.backends.cpu.get_cpu_capability(), raxis.backends.cpu.get_cpu_capability()
('AVX512', 'AVX512')
# ATEN_CPU_CAPABILITY=avx2 python ... and RAXIS_CPU_CAPABILITY=avx2 python ...
('AVX2', 'AVX2')
The cost of all this is a larger library, since every kernel is there three times, and one indirect call per operation, which is nothing next to a loop over thousands of elements. Compiling with -march=native would give the best code for one machine, and is fine for a program you build yourself, but it cannot go into a wheel.
7. AVX-512 only where it pays
If AVX-512 makes sigmoid twice as fast, why not use it for everything? Here is the table with add, which is limited by memory:
| A million floats, Windows | DEFAULT | AVX2 | AVX-512 |
|---|---|---|---|
PyTorch add | 0.111 ms | 0.101 ms | 0.102 ms |
PyTorch sigmoid | 2.37 ms | 0.47 ms | 0.22 ms |
add does not change: at 12 MB moved per call, the time goes in reading and writing memory, and the width of the registers does not matter. It is memory-bound, and sigmoid, with dozens of instructions per element, is compute-bound. Even the DEFAULT build is close: the loops of the generic Vectorized are simple enough for the compiler to vectorize them with SSE2, the minimum of every x86-64 CPU. So for add there is no AVX-512 kernel at all. In the AVX-512 build, PyTorch’s REGISTER_DISPATCH stores nullptr, and the stub falls back to the AVX2 kernel. A kernel that wants AVX-512 has to ask for it with ALSO_REGISTER_AVX512_DISPATCH (pytorchDispatchStub.h, raxisdispatch_stub.h), and in PyTorch the ones that do are those limited by computation: the transcendentals, sigmoid, GELU, softmax, attention.
There are reasons beyond “it does not help”:
- Clock speed. On some Intel server CPUs, running 512-bit instructions lowers the clock frequency of the core for a while, which can make the code around them slower. Recent CPUs, like the Zen 5 of these measurements, do not have that problem, but a library that runs everywhere has to be careful.
- The same results. A kernel that sums with 16 lanes adds the numbers in a different order than one with 8, and floats give slightly different results in different orders. Keeping the memory-bound kernels on AVX2 keeps their results equal on every x86 CPU with AVX2, whether it has AVX-512 or not, and Raxis relies on that to compare its results with PyTorch’s bit by bit: it registers AVX-512 for the same kernels as PyTorch, and its own few where it measured a gain.
The reductions, sums and maxima are the clearest case, and the next article measures them.
8. One a + b, all the way down
With the stub, every step of an a + b on the CPU is in place: the iterator of the TensorIterator article, the threads and the DimCounter of the article on n-dimensional loops, and the kernel, its loops and its registers from this one. Here is the example those articles carried, a of shape (1000, 1000) plus b of shape (1000, 1), followed from the Python call to the vaddps instructions:
raxis.add(a, b) on the 8 threads of this CPU. Highlighted: what one thread runs; the other seven run the same on their own rows at the same time. PyTorch goes through its dispatcher and autograd first, and then takes the same steps from build() down.9. The other CPUs
Everything so far was x86. The same machinery serves the rest, with other names:
| CPU | PyTorch builds | Raxis builds |
|---|---|---|
| x86-64 | DEFAULT, AVX2, AVX512 | DEFAULT, AVX2, AVX512 |
| ARM64 (Apple M, Graviton, Raspberry Pi 4/5) | DEFAULT with NEON, SVE128, SVE256 | DEFAULT with NEON |
| POWER | VSX | none |
| IBM Z | ZVECTOR | none |
On ARM64, NEON is part of the architecture: every 64-bit ARM CPU has its 128-bit registers, so the DEFAULT build already uses them, with the vec128 specializations from section 4. SVE is the newer vector extension of some ARM servers, whose register width is decided by each CPU, from 128 to 2048 bits; PyTorch builds kernels for 128 and 256, and Raxis does not use it. And the generic Vectorized is what remains for any other CPU, a microcontroller or a RISC-V board: correct everywhere, and as fast as the compiler can make its loops.
Key takeaways
- A kernel is two lambdas (section 1), one for an element and one for a vector of elements, and the loops take the number and the types of the inputs from the lambda itself (section 2).
Vectorized<T>is a register as a type (section 4), so the same kernel code serves AVX2, AVX-512 and NEON.expis not an instruction (section 5): it is a polynomial computed on every lane at once. On the 9700X,sigmoidgets 5 times faster from scalar to 8 lanes, and twice more with 16.- Every kernel file is compiled once per instruction set, and the best one is picked at runtime (section 6). AVX-512 is used only where the computation, not the memory, is the limit (section 7).
Further reading
- The Intel Intrinsics Guide, to look up what each
_mm256_*function does. - SLEEF, the vectorized math library, and its paper: Naoki Shibata and Francesco Petrogalli, “SLEEF: A Portable Vectorized Library of C Standard Mathematical Functions” (2020).
- Agner Fog’s optimization manuals, especially the instruction tables, for the latency and throughput of each instruction on each CPU.
If something is not clear, or you think I got something wrong, leave a comment.