Rebuilding PyTorch
Rebuilding the core of PyTorch from scratch in Raxis to understand how it works inside, one piece at a time, in the order I learned it.
- 01Why I rebuilt PyTorch's core from scratch
Three projects, one question I could not answer, and how this series is organized.
18 November 2026 · 10 min read
- 02What a PyTorch tensor really is
What is really in memory when you hold a tensor, why x[1:3] costs the same at any size, and what contiguous means.
18 November 2026 · 30 min read
- ~
Interlude
What the CPU does when your code reads memoryCaches, cache lines, the prefetcher, pages and the TLB, cache sets, SIMD registers, branch prediction, cores and threads, measured with tensors. The hardware background the rest of the series refers to.
18 November 2026 · 70 min read
- 03How PyTorch walks a tensor of any shape
An odometer for the indices, an inner loop the compiler can vectorize, and a single division per thread.
18 November 2026 · 35 min read
- 04How PyTorch prepares the loop of every operation
Broadcasting, type promotion, an output that already exists, the order of the dimensions, merging them, and the memory a tensor shares with itself.
18 November 2026 · 40 min read
- 05How a PyTorch kernel uses the vector registers of every CPU
The two lambdas of a kernel, Vectorized<T> as one register, how exp is computed eight numbers at a time, and one binary that picks SSE, AVX2 or AVX-512 when it starts.
18 November 2026 · 25 min read
- 06How PyTorch sums a tensor, and splits the work between threads
A reduction as a TensorIterator whose output does not move, why the obvious sum loop is slow and wrong, the cascade sum, how parallel_for splits a range, and why sum() can change with the number of threads while sum(dim) cannot.
18 November 2026 · 30 min read
- 07How PyTorch computes gradients
From micrograd to PyTorch's engine, with nodes and edges, saved tensors, a dependency count instead of a recursive sort, gradients freed as the backward advances, and the reference cycle that leaks a whole graph.
18 November 2026 · 35 min read
- 08How PyTorch runs the same operations on a GPU
How an operation reaches its CUDA kernel, why launches return before the work is done, a caching allocator instead of cudaMalloc, a thread per element with a division made a multiplication, and why pinned memory and the DataLoader keep the GPU busy.
18 November 2026 · 40 min read