How a PyTorch kernel uses the vector registers of every CPU

pgmesa · · 25 min read

Part of Rebuilding PyTorch (5 of 8)
  1. Why I rebuilt PyTorch's core from scratch
  2. What a PyTorch tensor really is
  3. Interlude: What the CPU does when your code reads memory
  4. How PyTorch walks a tensor of any shape
  5. How PyTorch prepares the loop of every operation
  6. How a PyTorch kernel uses the vector registers of every CPU
  7. How PyTorch sums a tensor, and splits the work between threads
  8. How PyTorch computes gradients
  9. How PyTorch runs the same operations on a GPU

The last two articles built everything around a kernel: TensorIterator prepares the shape, the types and the strides, and the loop walks them in 2-D blocks without a single division. What is left is the code that runs inside each block, the kernel itself, and that is where the last factor of the speed is decided. An addition is one instruction on eight numbers at once, if the code is written for it. An exp is not an instruction at all, and someone has to write it so that it runs on eight numbers at once too. And all of that has to work on the CPU that the program finds when it starts, which may have 128, 256 or 512-bit registers.

Before you start. This article builds on How PyTorch walks a tensor of any shape, especially its section on the compiler, SIMD and VectorizedLoop2d, and on How PyTorch prepares the loop of every operation. What SIMD registers are, and which ones each CPU has, is in the interlude on the hardware. Here there is more C++ than in the previous ones: templates, lambdas and a few SIMD intrinsics. Each piece is explained, and the tables say what it does with real values.

This is the path the article follows:

  1. What a kernel is.
  2. One lambda for any number of inputs.
  3. The vectorized loop, and its tail.
  4. Vectorized<T>: one register as a type.
  5. Functions that are not one instruction.
  6. One binary for every CPU.
  7. AVX-512 only where it pays.
  8. One a + b, all the way down.
  9. The other CPUs.

And at the end, the key takeaways.

All the timings are from a Ryzen 7 9700X, a Zen 5 with AVX-512, on one thread, with a million float32 numbers and outputs allocated beforehand (out=). They come from raxiskernels.py, which runs every operation in NumPy, PyTorch and Raxis, and in PyTorch and Raxis once per instruction set, forcing it as section 6 explains. I ran it in November 2026, with NumPy 2.5.4, PyTorch 2.14.1 and Raxis 0.5.0, on Windows 11 25H2 (MSVC 19.44) and Ubuntu 24.04 under WSL2 (GCC 13.3), on the same machine; the conclusions are the same, and the few numbers that differ a lot are pointed out.

1. What a kernel is

Back to the end of the last article: add built its TensorIterator and then called add_stub(iter.device_type(), iter). The stub is a table with one function per device and per instruction set, section 6 explains how it is filled, and the function it calls is the kernel. This is Raxis’s, for every type (raxisbinary_ops_kernel.cpp):

void add_kernel(TensorIterator& iter) {
    iter.common_dtype().dispatch_all_types([&]<typename scalar_t>(std::type_identity<scalar_t>) {
        cpu_kernel_vec(iter,
            [](scalar_t a, scalar_t b) -> scalar_t { return a + b; },                   // op
            [](Vectorized<scalar_t> a, Vectorized<scalar_t> b) { return a + b; });     // vop
    });
}

Three things happen here, and two of them we have already seen:

There is also a cpu_kernel, with op alone (pytorchLoops.h, raxisloops.h). It runs the plain loop of the next section on every block, and leaves any vectorization to the compiler. Kernels use it when a vector version is not worth writing, or not possible: a comparison that writes bools from floats, an operation that mixes types, or one with branches that would make a vector version slower than the scalar one.

2. One lambda for any number of inputs

add takes two inputs, exp one and where three. The loops are written once for all of them, so they have to find out from the lambda how many inputs it has, and of what types. That is the job of function_traits (pytorchFunctionTraits.h, raxisfunction_traits.h), a template that takes the type of the lambda’s operator() apart:

auto op = [](float a, float b) -> float { return a + b; };
using traits = function_traits<decltype(op)>;
traits::arity                // 2
traits::result_type          // float
traits::arg<0>::type         // float

With that, the plain loop can load each argument from its own pointer, with its own stride and its own type, and call the lambda. This is the heart of basic_loop, simplified (pytorchLoops.h, raxisloops.h):

// From basic_loop and dereference, simplified: data[0] is the output,
// data[1..] the inputs, each with its stride in bytes
for (; i < n; i++) {
    auto args = std::make_tuple(
        load<arg<0>::type>(data[1] + i * strides[1]),
        load<arg<1>::type>(data[2] + i * strides[2]));   // one per argument, generated
    *(result_type*)(data[0] + i * strides[0]) = std::apply(op, args);
}

The tuple is not written by hand: a std::index_sequence of the argument positions expands into one load per argument when the template is instantiated. So for add the compiler sees exactly the loop from the last article, with three pointers and three strides, and no tuple left in the generated code. This is also the loop that runs when the strides are not contiguous, the one the compiler does not vectorize.

3. The vectorized loop, and its tail

The last article ended inside VectorizedLoop2d: when every inner stride is the size of an element, or all but one input that has a stride of 0, each row goes through vectorized_loop (pytorchLoops.h, raxisloops.h). Here is its body, almost word for word:

Vec opt_scalar = Vec(S > 0 ? load<scalar_t>(data[S]) : scalar_t(0));   // the stride-0 input
int64_t i = 0;
for (; i <= n - 2 * Vec::size(); i += 2 * Vec::size()) {
    auto args1 = dereference_vec<traits>(&data[1], opt_scalar, S, i);
    auto args2 = dereference_vec<traits>(&data[1], opt_scalar, S, i + Vec::size());
    auto out1 = std::apply(vop, std::move(args1));
    auto out2 = std::apply(vop, std::move(args2));
    out1.store(data[0] + i * sizeof(scalar_t));
    out2.store(data[0] + (i + Vec::size()) * sizeof(scalar_t));
}
if (i < n) {
    basic_loop(data, strides, i, n, op);    // the tail, one element at a time
}

Three details explain most of its speed:

With rows of 1000 floats, the size of the examples of the series:

Instruction setFloats per vectorPer iterationVector iterationsTail
AVX281662 (992 floats)8
AVX-512163231 (992 floats)8

The tail is small here. It is not small when the rows are short, which is one of the reasons that the rows of 4 from the last article cost so much: 4 floats never fill a vector, so the whole row was a tail. And it is the reason the two lambdas must compute the same thing: the same row mixes elements computed by vop and by op, and a user should not be able to tell which was which.

And a row with any other inner stride, like the 2 of x[::2], never gets here: VectorizedLoop2d sends it whole to basic_loop, one element at a time. That is not a case PyTorch forgot. A vector load reads consecutive floats only, and the instructions that read scattered ones, the gathers, cost about as much as reading them one by one, as the interlude on the hardware explains. That is most of the factor of three that x[::2] costs in the stride table of the same article.

4. Vectorized<T>: one register as a type

vop works on Vectorized<float>, and the last article only showed one of its operators. It is worth seeing the whole idea, because it is what lets the same kernel compile for every CPU.

The generic version is a plain array, with loops for every operation (pytorchvec_base.h, raxisvec_base.h):

template <typename T>
struct Vectorized {
    static constexpr int kSize = VECTOR_WIDTH / sizeof(T);   // 32 bytes: 8 floats, 4 doubles
    alignas(VECTOR_WIDTH) T values_[kSize];
    // operator+ adds values_[i] in a loop, exp() calls std::exp on each, ...
};

It works for any type and any CPU, and the compiler may or may not turn its loops into vector instructions. For the types and instruction sets that matter there are specializations, which replace the array with a real register and every operation with the instruction that does it. These are the three versions of Vectorized<float> in Raxis, and PyTorch has the same three, among others:

HeaderWhen it is usedRegisterFloatsoperator+
vec256/vec256_float.hAVX2 build__m2568_mm256_add_ps
vec512/vec512_float.hAVX-512 build__m51216_mm512_add_ps
vec128/vec128_float.hARM64 (NEON)float32x4_t4vaddq_f32

(raxisvec256_float.h, raxisvec512_float.h, raxisvec128_float.h; PyTorch’s AVX2 one is pytorchvec256_float.h.)

Which one a file gets is decided by the preprocessor: vec.h includes the AVX2 headers only when the file is compiled with CPU_CAPABILITY_AVX2, the AVX-512 ones with CPU_CAPABILITY_AVX512, and the NEON ones on ARM64. In section 6 we will see that each kernel file is compiled several times, once with each of those definitions. The kernel’s source never changes: a + b on two Vectorized<float> becomes _mm256_add_ps in one build and _mm512_add_ps in another.

Two more things the specializations need, and both are visible in vec256_float.h (raxisvec256_float.h, pytorchvec256_float.h):

5. Functions that are not one instruction

An addition is one instruction. exp is not: no x86 or ARM CPU has an instruction that computes the exponential of eight floats. So Vectorized<float>::exp() has to be written with additions, multiplications and bit tricks, for eight lanes at a time and without branches, because each lane may need a different path and a vector has only one.

The trick behind almost every implementation is the same: split xx into a part that powers of two handle exactly, and a small remainder that a polynomial handles well.

ex=2n⋅er,n=round⁡ ⁣(xln⁡2),r=x−nln⁡2,∣r∣≤ln⁡22e^x = 2^n \cdot e^r, \qquad n = \operatorname{round}\!\left(\frac{x}{\ln 2}\right), \qquad r = x - n \ln 2, \quad |r| \le \frac{\ln 2}{2}

2n2^n costs nothing: a float is stored as a sign, an exponent and a mantissa, so 2n2^n is the number with exponent nn and mantissa zero, built by writing n+127n + 127 into the exponent bits. And on the small interval of rr, a polynomial of degree 5 approximates ere^r to the precision of a float. PyTorch’s exp_u20 does exactly that, with 20 ULP of error at most (pytorchvec256_float.h, raxisvec256_float.h). An ULP, unit in the last place, is the distance between a float and the next one, so 20 ULP means that the last few bits may be wrong:

// From exp_u20, simplified (each line runs on 8 lanes at once)
x  = min(max(x, ln(FLT_MIN)), ln(FLT_MAX));   // clamp instead of branching
n  = floor(x * log2(e) + 0.5);                 // round(x / ln 2)
r  = x - n * ln2;                              // the small remainder
p  = 1 + r * (c1 + r * (c2 + r * (c3 + r * (c4 + r * c5))));   // e^r, 5 FMAs
two_n = (int(n) + 127) << 23;                  // 2^n, written in the exponent bits
return p * as_float(two_n);

Follow it with x=3x = 3:

StepValue
xlog⁡2ex \log_2 e4.328
nn4
r=3−4⋅0.693147r = 3 - 4 \cdot 0.6931470.227411
polynomial, ere^r1.255335
242^4, exponent bits 4+127=1314 + 127 = 13116
result, 16⋅1.25533516 \cdot 1.25533520.0854

and e3=20.0855e^3 = 20.0855. (The real code computes 2n−12^{n-1} and multiplies by 2 at the end, so that n=128n = 128 does not overflow the exponent, and it puts a 0 in the lanes where xx was too small, with a blend: the mask is the if.)

Every line is a vector instruction with no branches, so the 8 lanes, or 16 with AVX-512, go through it together. A scalar exp from the C library does a similar computation for one number, with branches for the special cases and more care for the last bit.

Who implements these functions. PyTorch, and Raxis with it, takes them from three places:

  • SLEEF, a library of vectorized math functions for every instruction set, with versions named by their error: Sleef_expf8_u10 is exp for 8 floats with at most 1.0 ULP of error. Vectorized<float>::exp() calls it (pytorchvec256_float.h, raxisvec256_float.h).
  • Its own polynomials, when speed matters more than the last bit: exp_u20 for the softmax inside flash attention, or a polynomial erf for GELU.
  • MKL’s VML, when PyTorch is built with it: whole contiguous arrays at once (vsExp), with MKL choosing its own instruction set. torch.exp, tanh or sin on the CPU go there.

Now the cost. sigmoid is 1/(1+e−x)1 / (1 + e^{-x}), and its vop uses Vectorized::exp(), so it shows the whole difference between a scalar exp and a vector one. In the generic Vectorized of the DEFAULT build, exp() calls std::exp lane by lane; in the AVX2 and AVX-512 builds it is SLEEF with 8 and 16 lanes:

A million floats, WindowsDEFAULTAVX2AVX-512
PyTorch sigmoid2.37 ms0.47 ms0.22 ms
Raxis sigmoid2.53 ms0.49 ms0.21 ms
PyTorch exp (MKL VML)0.46 ms0.46 ms0.47 ms
Raxis exp (MKL VML)0.46 ms0.46 ms0.46 ms

From the scalar exp to 8 lanes, 5 times faster; from 8 to 16 lanes, 2.2 times more. This is the kind of operation where the width of the registers pays in full, because the work is in the instructions and the memory has time to keep up. And exp does not change at all with the instruction set of PyTorch’s build, because MKL picks its own.

On Linux, with GCC and glibc, the DEFAULT column changes. PyTorch’s sigmoid takes 1.26 ms there, because the scalar expf of Linux’s C library, glibc, is faster than the one on Windows, while Raxis stays at 2.6 ms, because its generic exp() converts each float to double and calls the double version (PyTorch’s calls the float one). It only matters on x86 CPUs without AVX2. The other difference is NumPy, which has its own vector versions of these functions: on Linux its tanh takes 0.12 ms, against 1.58 ms for MKL’s in PyTorch and Raxis, and its exp 0.25 ms against 0.46. On Windows the gap is smaller (tanh 0.74 ms in NumPy). I have not looked into where MKL loses that time on this CPU, and it is the first thing I would look at to make these operations faster.

6. One binary for every CPU

So the same kernel is worth 5 times more with AVX2 than without, and twice more again with AVX-512 on some operations. But a program compiled with AVX2 instructions crashes on a CPU that does not have them, with an “illegal instruction” error, and a wheel on PyPI has to run on any x86-64 CPU, from the first ones of the 2000s to those of 2026. Compiling for the minimum wastes the registers of every recent CPU, and compiling for the newest leaves out the older ones.

PyTorch’s answer has three pieces.

Each kernel file is compiled several times. The build system takes every file of kernels and compiles it once per instruction set, with that set’s compiler flags and a definition with its name: /arch:AVX2 or -mavx2 -mfma and CPU_CAPABILITY=AVX2, /arch:AVX512 or -mavx512f -mavx512bw -mavx512vl -mavx512dq and CPU_CAPABILITY=AVX512, and no flags for CPU_CAPABILITY=DEFAULT (pytorchCodegen.cmake, raxisDispatchCompile.cmake). Raxis does it with one small generated file per kernel and per set, which only #includes the real one (raxisDispatchCompile.cmake):

set(wrapper "${dispatch_dir}/${name}.${CPU_CAPABILITY}.cpp")       # binary_ops_kernel.AVX2.cpp
file(WRITE "${wrapper}" "#include \"${abs_src}\"\n")               # the same source
set_source_files_properties(${wrapper} PROPERTIES
    COMPILE_FLAGS "${FLAGS} -DCPU_CAPABILITY=${CPU_CAPABILITY} -DCPU_CAPABILITY_${CPU_CAPABILITY}")

Three copies of add_kernel in the same library would clash, so every kernel lives in inline namespace CPU_CAPABILITY { ... }, which the definition turns into namespace AVX2, namespace AVX512 or namespace DEFAULT. That is the inline namespace CPU_CAPABILITY at the top of loops.h and of every Vectorized header.

Each copy registers itself. At the end of the kernel file, REGISTER_DISPATCH(add_stub, &add_kernel) stores the address of that copy in the slot of its set: in the AVX2 build it fills add_stub’s AVX2 slot, in the DEFAULT build the DEFAULT one (pytorchDispatchStub.h, raxisdispatch_stub.h). These are static variables initialized when the library loads, which is why the static build of Raxis needs to link the whole archive: otherwise the linker drops these files, which nothing references, and every stub is empty.

The stub picks one, once. The first call to add_stub asks which instruction set the CPU has, takes the best slot that is filled, and keeps the pointer for every later call (pytorchDispatchStub.cpp, raxisdispatch_stub.cpp). Asking the CPU takes two instructions:

PyTorch asks through cpuinfo, a library that does the same and more; Raxis writes the two calls. Either way, you can see the result, and you can force a lower set with an environment variable before the first operation, which is how the tables of this article were made:

>>> import torch, raxis
>>> torch.backends.cpu.get_cpu_capability(), raxis.backends.cpu.get_cpu_capability()
('AVX512', 'AVX512')
# ATEN_CPU_CAPABILITY=avx2 python ... and RAXIS_CPU_CAPABILITY=avx2 python ...
('AVX2', 'AVX2')

The cost of all this is a larger library, since every kernel is there three times, and one indirect call per operation, which is nothing next to a loop over thousands of elements. Compiling with -march=native would give the best code for one machine, and is fine for a program you build yourself, but it cannot go into a wheel.

7. AVX-512 only where it pays

If AVX-512 makes sigmoid twice as fast, why not use it for everything? Here is the table with add, which is limited by memory:

A million floats, WindowsDEFAULTAVX2AVX-512
PyTorch add0.111 ms0.101 ms0.102 ms
PyTorch sigmoid2.37 ms0.47 ms0.22 ms

add does not change: at 12 MB moved per call, the time goes in reading and writing memory, and the width of the registers does not matter. It is memory-bound, and sigmoid, with dozens of instructions per element, is compute-bound. Even the DEFAULT build is close: the loops of the generic Vectorized are simple enough for the compiler to vectorize them with SSE2, the minimum of every x86-64 CPU. So for add there is no AVX-512 kernel at all. In the AVX-512 build, PyTorch’s REGISTER_DISPATCH stores nullptr, and the stub falls back to the AVX2 kernel. A kernel that wants AVX-512 has to ask for it with ALSO_REGISTER_AVX512_DISPATCH (pytorchDispatchStub.h, raxisdispatch_stub.h), and in PyTorch the ones that do are those limited by computation: the transcendentals, sigmoid, GELU, softmax, attention.

There are reasons beyond “it does not help”:

The reductions, sums and maxima are the clearest case, and the next article measures them.

8. One a + b, all the way down

With the stub, every step of an a + b on the CPU is in place: the iterator of the TensorIterator article, the threads and the DimCounter of the article on n-dimensional loops, and the kernel, its loops and its registers from this one. Here is the example those articles carried, a of shape (1000, 1000) plus b of shape (1000, 1), followed from the Python call to the vaddps instructions:

The whole path of a + b on the CPUraxis.add(a, b) with a of shape (1000, 1000) and b of shape (1000, 1), float32, on 8 threads. The iterator is built, add_stub picks the AVX2 kernel, for_each splits the million elements into 8 ranges, thread 3 gets rows 375 to 499, each row goes through the vectorized loop 16 floats at a time with b[row] in a register, the last 8 floats of each row go through the scalar loop, and the output is returned.One a + b, from the Python call to the vector registersa (1000, 1000) + b (1000, 1), float32, 8 threads, AVX2 kernelsaddraxis.add(a, b) -> ops::add_out(a, b, out)out is still undefined: the iterator will allocate itbuild()shape (1000, 1000), float32, no temporaries, out like astrides in bytes, the inner dimension first:out [4, 4000] a [4, 4000] b [0, 4]add_stubthe AVX2 copy of add_kernel (add has no AVX-512 one)cpu_kernel_vec(iter, op, vop): op adds 1 float, vop 8for_each1,000,000 elements > 32,768 and 8 threads: parallel_for8 ranges of 125,000 elements, one per thread01,000,000thread 3: [375,000, 500,000)DimCounter375,000 % 1000 = 0 and 375,000 / 1000 = 375: the rangestarts at row 375 and holds 125 whole rowsloop2d(ptrs, strides, 1000, 125)VectorizedLoop2dinner strides out 4, a 4, b 0: not all contiguous,but b is one number per row: vectorized_loop with S = 2,once for each of the 125 rowsvectorized_loopb[row] in the 8 lanes of a register, and per iteration2 x 8 floats of a, 2 vaddps, 2 stores: 62 iterations0-1516-3132-4748-63...976-991992-999tailone row: each box is one iteration, two vectors of 8 floatsbasic_loopfloats 992 to 999, one at a time with op:out[i] = a[i] + b[row]back in addout is float32 already: cast_outputs has nothing to copyout = iter.get_output(0), returned to Pythonthread 3, and the other 7 at the same time
One raxis.add(a, b) on the 8 threads of this CPU. Highlighted: what one thread runs; the other seven run the same on their own rows at the same time. PyTorch goes through its dispatcher and autograd first, and then takes the same steps from build() down.

9. The other CPUs

Everything so far was x86. The same machinery serves the rest, with other names:

CPUPyTorch buildsRaxis builds
x86-64DEFAULT, AVX2, AVX512DEFAULT, AVX2, AVX512
ARM64 (Apple M, Graviton, Raspberry Pi 4/5)DEFAULT with NEON, SVE128, SVE256DEFAULT with NEON
POWERVSXnone
IBM ZZVECTORnone

On ARM64, NEON is part of the architecture: every 64-bit ARM CPU has its 128-bit registers, so the DEFAULT build already uses them, with the vec128 specializations from section 4. SVE is the newer vector extension of some ARM servers, whose register width is decided by each CPU, from 128 to 2048 bits; PyTorch builds kernels for 128 and 256, and Raxis does not use it. And the generic Vectorized is what remains for any other CPU, a microcontroller or a RISC-V board: correct everywhere, and as fast as the compiler can make its loops.

Key takeaways

Further reading

If something is not clear, or you think I got something wrong, leave a comment.