Why I rebuilt PyTorch's core from scratch

pgmesa · · 10 min read

On this page

  1. SynapGrad: autograd on real tensors
  2. SynapX: closer to the real thing
  3. A tensor of my own
  4. Into the source
  5. Raxis
  6. How this series works
  7. What this is not
  8. A note on time, and on the tools
Rebuilding PyTorch · 1 of 8
Part of Rebuilding PyTorch (1 of 8)
  1. Why I rebuilt PyTorch's core from scratch
  2. What a PyTorch tensor really is
  3. Interlude: What the CPU does when your code reads memory
  4. How PyTorch walks a tensor of any shape
  5. How PyTorch prepares the loop of every operation
  6. How a PyTorch kernel uses the vector registers of every CPU
  7. How PyTorch sums a tensor, and splits the work between threads
  8. How PyTorch computes gradients
  9. How PyTorch runs the same operations on a GPU

Hi everyone! This is the start of a series about how PyTorch works inside. Not how to use it, and not the theory of backpropagation, of which there is plenty, but what happens in the machine when you write a + b and a tensor with a million numbers comes back before you can blink.

I have spent the last few years chasing that question, in my free time, in bursts, and through three projects, and the last of them ended up being a rewrite of the core of PyTorch. I learned a lot on the way, much of it by being wrong, and I would like to share it. So before getting into the details, let me tell you how I got here, because the question only makes sense with the story around it.

Who this is for. I am going to assume that you have written some PyTorch, and that words like tensor, shape, broadcasting or backward() do not scare you. You do not need to have read its source, and that is the point. Some C/C++ will help, because that is where the interesting code lives, but I will explain what matters as we go.

SynapGrad: autograd on real tensors

It started in 2023 with micrograd, Andrej Karpathy’s small autograd engine. It works on scalars, and I wanted the same thing for n-dimensional tensors, like PyTorch. So I wrote SynapGrad: autograd and a small nn library in pure Python, with NumPy doing the tensor operations.

It showed me how the pieces fit together: a graph that records each operation, a backward formula for each one, and layers and optimizers on top, with an API like PyTorch’s. NumPy gave me most of the operations almost for free. Convolutions were the exception: I had to write them myself and work out their backward by hand, and when the gradients finally matched PyTorch’s I was very proud of it.

But it was still a Python library on top of someone else’s arrays. Training a small CNN on MNIST took 13 minutes on my laptop, against 2 minutes for PyTorch, and PyTorch is not built that way: its core is C++, and Python is a layer on top. That was what I wanted to try next.

SynapX: closer to the real thing

SynapX gave me that structure: a library in C++, Python bindings, an nn module in Python on top, and wheels for several platforms built by GitHub Actions. I used libtorch, PyTorch’s C++ library, as the tensor backend with its own autograd turned off, which also brought the GPU almost for free. It was much closer to how PyTorch actually works, and I learned a lot from it.

The autograd was mine, but libtorch did the operations for me. So the part I was most curious about, how an operation over a million numbers can be so fast, was exactly the part I was not writing. Between that, and not having the time to keep supporting the project, I left it there.

A tensor of my own

The next step was the tensor itself. I had actually tried it before, in 2024, with a small tensor of my own in C and in C++, my first attempt at this. Adding two large tensors of the same shape, my code was close to NumPy, which was encouraging. With broadcasting it was many times slower, and I did not know how to fix it. How do you walk a tensor with any number of dimensions, when you cannot even write the nested loops because you do not know how many there are?

It looks like a simple question, and I could not answer it on my own. To go further I had to do what I had been avoiding: read PyTorch.

Into the source

Good material exists on how PyTorch is organized. Edward Yang’s “PyTorch internals” is excellent, and so is his post on the dispatcher. What I could not find, at least when I was looking for it, was the level below: the algorithms and the optimizations, explained in detail, with the code in front of you and with measurements. It may exist somewhere, but it is a niche topic, it does not come up in the first pages of a search, and not many people work at that level.

So I went to the code, and the first barrier appears very quickly. Try to follow a + b from Python to C++ in a fresh clone of PyTorch 2.14.1. The method behind it is called THPVariable_add, so you search for it:

$ git grep THPVariable_add
tools/autograd/templates/python_variable_methods.cpp:  {"__add__", castPyCFunctionWithKeywords(TypeError_to_NotImplemented_<THPVariable_add>), ...
tools/autograd/templates/python_variable_methods.cpp:  {"__radd__", castPyCFunctionWithKeywords(TypeError_to_NotImplemented_<THPVariable_add>), ...

Two uses, inside a template, and no definition. The function does not exist until you build PyTorch: it is generated from pytorchnative_functions.yaml, more than 16,000 lines of operator declarations, and written into a directory that git ignores.

This is not a flaw. Generating that code is the price of everything PyTorch supports: every operator, across every backend, with autograd, tracing and more. But it means that the path from Python to the code that does the work is not written anywhere you can read on a first visit, and it is no wonder that it took me a while to find my way around.

Once past that, though, it is wonderful. I found how a tensor is really defined, the answer to my question in DimCounter, an odometer that walks n dimensions, TensorIterator turning almost every operator into a single loop, the 2-D loops, the vectorized kernels. That is what I want to tell you about in this series.

Raxis

By then I had written an autograd engine twice, and I felt I had it under control. What I kept reusing, and never understood, was the tensor, and that is where the heavy lifting was. I also wanted to stop linking all of libtorch to my library and wrapping its tensor class in my own. So I decided to write one more library, this time from zero.

That is Raxis: the core of PyTorch rebuilt from its source, the tensor library and the autograd engine, in C++20 with Python bindings. My first attempt was in Rust, and it lasted until the first add: I moved to C++ because I wanted to compare my code side by side with PyTorch’s, and because of CUDA.

From zero needs two honest footnotes. The first is that it does not mean alone: Raxis links BLAS for matrix multiplications, SLEEF for vectorized math and mimalloc as its allocator, each for one specific job, and PyTorch is there only for the tests to compare against. The second is that it does not mean invented by me. Almost all of the interesting code is PyTorch’s: I found how it does each thing, extracted the logic that makes it fast, and adapted and simplified it to fit. I rewrote something only when it was too tied to PyTorch’s machinery to come out, and a few pieces are my own. Many places carry a // From: comment pointing to the original file, though not every line of inspiration has one.

What Raxis adds is not the ideas but the shape they are in. Readability comes first, next to performance, even where the subject is complex: fewer layers and no legacy to carry. No operator is generated: a call goes through a wrapper, an operator and a kernel, and you can follow it with your editor from end to end. And it has to keep up, so its operations are tested and timed against PyTorch.

And what matters most to me: taking only those techniques is enough. This is one training step of the same models in the four libraries, on a Ryzen 7 9700X with 8 threads:

Training stepRaxisPyTorchSynapXSynapGrad
Tiny MLP (batch 4)0.06 ms0.12 ms0.14 ms0.10 ms
MNIST MLP (batch 64)0.29 ms0.40 ms0.49 ms1.12 ms
MNIST CNN (batch 128)5.5 ms20 ms81 ms150 ms

Most of PyTorch’s time in the CNN is Dropout. On the CPU it draws its random numbers from mt19937, a serial generator. It had a parallel path through MKL, but that path is disabled in pytorchDistributionKernels.cpp because of a bug in the distributions it produced, still open. On the GPU PyTorch already uses Philox, a generator that can run in parallel, and Raxis uses Philox on the CPU too, so this win comes from doing something different from PyTorch. This could be an article too. It is not all wins. PyTorch’s convolutions are faster because they use libraries that Raxis does not link: large ones are about 2.5x faster on the CPU with oneDNN, and on an RTX 5070 Ti a training step of that CNN takes 0.83 ms with cuDNN against 1.25 ms with Raxis. Without convolutions the GPU numbers are closer: a training step of nanoGPT takes 45.7 ms with PyTorch and 47.3 ms with Raxis.

The numbers of that table can be reproduced with raxistraining_split.py. PyTorch is version 2.14.1 everywhere, also as SynapX’s backend (SynapX 0.4.0).

In 2023 I wanted to understand how a framework like PyTorch trains a model. Three projects later I can say that I understand its core: not only the theory, but the algorithms and the optimizations that make it fast. This series is my attempt to save the next curious person the time it took me.

How this series works

Every article starts from one question and follows the same path: how PyTorch does it, with links to its source; how Raxis does it, with fewer layers; and numbers, measured with both. All the links point to fixed commits, so they will keep matching the text.

I do not have a closed list of articles, and I would rather not promise one, because it will change as I write them. Roughly, the order is the one in which I learned it: tensors and the machinery that walks them, how a Python call reaches the right kernel, autograd, heavy operations like matmul and convolutions, and finally the GPU. The table of contents at the top of each article shows what is published so far.

What this is not

Raxis does not replace PyTorch, and it does not try to. It has no distributed training, no torch.compile and no autocast, no float16 or bfloat16, and only NVIDIA GPUs. All of that is real value that PyTorch gives you, and most of what makes its source hard to read exists because of it. If what Raxis has is enough for what you are training, it is a working library, fast and light, and you are welcome to use it and to contribute to it: there are still a lot of interesting things to implement, or to port from PyTorch, like the ones above.

A note on time, and on the tools

One last thing, which has nothing to do with tensors.

All of this has been done in my free time, and not steadily: it has gone in bursts, with months where I did not touch any of it. More than three and a half years have passed since the first commit of SynapGrad. I started it as a young and inexperienced engineer with a lot of curiosity, and I am writing this a bit older, with a bit more judgment about these systems. Some of the decisions you will read about seem a little silly to me now, and I have left them in, because that is how you actually learn this.

And there is a curious coincidence. SynapGrad started a few months after ChatGPT came out, so this whole path has run alongside the rise of LLMs, and it would not be honest to leave them out. For SynapGrad they could help me very little. Year after year that changed, and in the second half of 2026 AI agents have let me move Raxis and this blog forward much faster than I could have alone: searching through the PyTorch source, porting code, testing, measuring, and writing with me.

What I have tried not to lose is the understanding, which was the whole point. A tool can write a kernel for me, but it cannot understand it for me. I can explain the core, the part that makes a tensor library what it is: tensors and strides, TensorIterator, the kernel machinery, reductions and threads, autograd, and how an op reaches the GPU, and with it the most important kernels. Many of the secondary kernels I have not studied as closely. Some were added because another part needed them, ported from PyTorch and checked against it, and for others I have not yet had the time to see why PyTorch does them that way; as I write this, Raxis 0.5.0 has almost 270 kernels between the CPU and the GPU. They are there, tested against PyTorch and about as fast, for anyone who wants to open the ones that interest them, as I will keep doing. Every article here starts from a question I needed to answer, and every number can be reproduced. I am convinced that it is still worth understanding how these things work, perhaps now more than ever, and I hope this series convinces you too.

If something is not clear, or you think I got something wrong, leave a comment: I am still learning this too.