Rebuilding PyTorch

Rebuilding the core of PyTorch from scratch in Raxis to understand how it works inside, one piece at a time, in the order I learned it.

  1. 01
    Why I rebuilt PyTorch's core from scratch

    Three projects, one question I could not answer, and how this series is organized.

    18 November 2026 · 10 min read

  2. 02
    What a PyTorch tensor really is

    What is really in memory when you hold a tensor, why x[1:3] costs the same at any size, and what contiguous means.

    18 November 2026 · 30 min read

  3. ~

    Interlude

    What the CPU does when your code reads memory

    Caches, cache lines, the prefetcher, pages and the TLB, cache sets, SIMD registers, branch prediction, cores and threads, measured with tensors. The hardware background the rest of the series refers to.

    18 November 2026 · 70 min read

  4. 03
    How PyTorch walks a tensor of any shape

    An odometer for the indices, an inner loop the compiler can vectorize, and a single division per thread.

    18 November 2026 · 35 min read

  5. 04
    How PyTorch prepares the loop of every operation

    Broadcasting, type promotion, an output that already exists, the order of the dimensions, merging them, and the memory a tensor shares with itself.

    18 November 2026 · 40 min read

  6. 05
    How a PyTorch kernel uses the vector registers of every CPU

    The two lambdas of a kernel, Vectorized<T> as one register, how exp is computed eight numbers at a time, and one binary that picks SSE, AVX2 or AVX-512 when it starts.

    18 November 2026 · 25 min read

  7. 06
    How PyTorch sums a tensor, and splits the work between threads

    A reduction as a TensorIterator whose output does not move, why the obvious sum loop is slow and wrong, the cascade sum, how parallel_for splits a range, and why sum() can change with the number of threads while sum(dim) cannot.

    18 November 2026 · 30 min read

  8. 07
    How PyTorch computes gradients

    From micrograd to PyTorch's engine, with nodes and edges, saved tensors, a dependency count instead of a recursive sort, gradients freed as the backward advances, and the reference cycle that leaks a whole graph.

    18 November 2026 · 35 min read

  9. 08
    How PyTorch runs the same operations on a GPU

    How an operation reaches its CUDA kernel, why launches return before the work is done, a caching allocator instead of cudaMalloc, a thread per element with a division made a multiplication, and why pinned memory and the DataLoader keep the GPU busy.

    18 November 2026 · 40 min read