How PyTorch computes gradients

pgmesa · · 35 min read

Part of Rebuilding PyTorch (7 of 8)
  1. Why I rebuilt PyTorch's core from scratch
  2. What a PyTorch tensor really is
  3. Interlude: What the CPU does when your code reads memory
  4. How PyTorch walks a tensor of any shape
  5. How PyTorch prepares the loop of every operation
  6. How a PyTorch kernel uses the vector registers of every CPU
  7. How PyTorch sums a tensor, and splits the work between threads
  8. How PyTorch computes gradients
  9. How PyTorch runs the same operations on a GPU

Everything so far was about making one operation fast. Training a model needs something else as well: after the forward, the gradient of the loss with respect to every parameter. PyTorch computes it with loss.backward(), and the idea behind it fits in a few dozen lines of Python, as Andrej Karpathy’s micrograd shows. This article starts from those lines and follows what has to change when the values are tensors, the graphs have thousands of operations, and the memory they hold is the GPU’s. Most of the answers are about memory, not about math: what a node may keep, when it can let it go, and how a single wrong pointer keeps a whole graph alive.

Before you start. This article needs much less of the previous ones than they needed of each other: what a tensor is (What a tensor is, for views and shared storage) and what broadcasting does (How PyTorch prepares the loop of every operation). The math is the chain rule for derivatives, and each step is worked out with numbers. The code is C++ with std::shared_ptr, the reference-counted pointer of the standard library, explained where it matters.

This is the path the article follows:

  1. What a backward computes.
  2. micrograd: the whole idea in thirty lines.
  3. From numbers to tensors.
  4. The graph: nodes and edges.
  5. Recording an operation.
  6. Saved tensors, and the cycle that leaks a graph.
  7. The engine: a node runs when all its gradients have arrived.
  8. Which ready node runs first.
  9. Adding gradients together.
  10. Freeing memory during the backward.
  11. Deep graphs.
  12. What this article leaves out.

And at the end, the key takeaways.

The numbers come from raxisautograd.py, run in November 2026 with Python 3.12.16, PyTorch 2.14.1 and Raxis 0.5.0, on a Ryzen 7 9700X with an RTX 5070 Ti, on one thread. The CPU tables are from Windows 11 25H2 and the text gives the Linux numbers (Ubuntu 24.04 under WSL2, same machine) where they differ; the GPU tables are from Windows only.

1. What a backward computes

Take two small tensors and a loss built from them:

a = torch.tensor([2., 3.], requires_grad=True)
b = torch.tensor([4., 5.], requires_grad=True)
c = a * b          # [8, 15]
d = c + a          # [10, 18]
loss = d.sum()     # 28
loss.backward()
a.grad, b.grad     # [5, 6], [2, 3]

requires_grad=True marks a and b as the tensors whose gradients we want, the leaves of the computation. The gradient of loss with respect to a says how much loss changes when each element of a changes a little. Written element by element, the loss is

loss=∑i(aibi+ai)\text{loss} = \sum_i (a_i b_i + a_i)

so changing a0a_0 by a small amount ε\varepsilon changes the loss by (b0+1) ε(b_0 + 1)\,\varepsilon, and changing b0b_0 changes it by a0 εa_0\,\varepsilon:

∂ loss∂ai=bi+1=[5,6],∂ loss∂bi=ai=[2,3]\frac{\partial\, \text{loss}}{\partial a_i} = b_i + 1 = [5, 6], \qquad \frac{\partial\, \text{loss}}{\partial b_i} = a_i = [2, 3]

That was easy because the whole expression fits in one line. A model is a long chain of operations, and nobody writes its derivative by hand. The chain rule is what makes it mechanical: if loss depends on d, and d depends on c, the gradient of loss with respect to c is the gradient with respect to d times the derivative of d with respect to c. So each operation only has to know its own local derivative, and the gradients can be passed backwards from the loss, one operation at a time:

StepOperationReceivesLocal ruleSends
1loss = d.sum()1each did_i counts onceto d: [1, 1]
2d = c + a[1, 1]pass it onto c: [1, 1]
to a: [1, 1]
3c = a * b[1, 1]times the other factorto a: [1, 1] * b = [4, 5]
to b: [1, 1] * a = [2, 3]
4leaf a[1, 1], [4, 5]add thema.grad = [5, 6]
5leaf b[2, 3]add themb.grad = [2, 3]

This is reverse-mode automatic differentiation, or backpropagation. Three things in the table are the whole problem in small:

The rest of the article is how PyTorch does each of the three, and why.

2. micrograd: the whole idea in thirty lines

micrograd, which Karpathy wrote in 2020 to teach backpropagation, does all of this for single numbers. A Value holds a number, its gradient, the values it was computed from, and a function that knows how to pass its gradient back to them (micrograd/engine.py):

class Value:
    def __init__(self, data, _children=(), _op=''):
        self.data = data
        self.grad = 0
        self._backward = lambda: None
        self._prev = set(_children)

    def __mul__(self, other):
        out = Value(self.data * other.data, (self, other), '*')
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

Every operation creates its result and attaches to it a small function, a closure: a function that remembers the variables around it, here self, other and out. That closure is step 3 of the table: multiply the incoming gradient by the other factor, and add, with +=, into the gradients of the inputs. The += is the third point of the list above, done in the simplest way: every value has a grad that starts at 0 and collects everything that arrives.

The second point, the order, is the backward method:

    def backward(self):
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build_topo(child)
                topo.append(v)
        build_topo(self)
        self.grad = 1
        for v in reversed(topo):
            v._backward()

build_topo visits the values recursively, depth first, and appends each one only after all the values it was computed from. The result is a topological order: a list in which every value comes after its inputs. Walking it backwards, every value runs its closure only after everything computed from it has added its gradient. My first project, SynapGrad, did exactly this with NumPy arrays instead of numbers, and so do most small autograd engines.

It is correct, and it is short. What does PyTorch do differently, then, and why does it need to?

3. From numbers to tensors

Four things change when the values are tensors and the programs are real models.

Shapes. With numbers, the gradient of an input has the shape of the input, always: one number. With tensors, broadcasting breaks that. If x has shape [2, 3] and w has shape [3], x * w reads every element of w twice, once per row, so the gradient of w is the sum, over the rows, of the gradient of each use:

∂ loss∂wj=∑igij xij\frac{\partial\, \text{loss}}{\partial w_j} = \sum_i g_{ij}\, x_{ij}

where gg is the gradient that reaches x * w. The local rule “multiply by the other factor” gives a [2, 3] tensor, and it has to be summed down to the [3] of w. Every operation that broadcasts has to do this.

Several inputs and outputs. torch.max(x, dim=1) returns two tensors, the values and their positions; x.unbind(0) returns each row of x as a tensor of its own. The gradient machinery has to know which output a gradient belongs to, and which input of the next operation it goes to.

Memory. A micrograd value holds one number. A tensor saved for the backward can hold hundreds of megabytes, and a model saves many of them. When each one is freed matters, and a graph that is not freed at all runs the GPU out of memory within a few training steps.

Size. A training step of a large model records thousands of operations, and some programs record far more: a recurrent network unrolled over a long sequence, or a loop of small operations. Section 11 shows where recursion stops.

And one thing changes in favour of the tensors: speed per operation stops being the main concern. These are the costs per recorded operation, measured on a chain of 500 multiplications, for micrograd’s numbers (the class is written out in the script) and for tensors of 16 floats:

Per node, microsecondsmicrogradPyTorchRaxis
forward (compute and record)0.281.391.07
backward0.252.100.45

micrograd is faster per node. A Python object with a closure is cheap, and the tensor libraries pay to go from Python into C++, create a tensor, and check its shape and type. But a micrograd node does one multiplication, and a tensor node does one multiplication for every element: at 16 floats the tensor libraries are already ahead per number, and at the sizes of a model the bookkeeping per node disappears next to the work. That PyTorch’s backward takes 2.1 microseconds per node and Raxis’s 0.45 is section 11’s question. On Linux every number of the tensor libraries is lower (0.71 and 0.71 for the forward, 1.47 and 0.36 for the backward) and micrograd’s are the same.

4. The graph: nodes and edges

In micrograd, the graph is made of the values themselves: each value points to its inputs (_prev) and carries its closure. PyTorch separates the two. The tensors stay as they are, and the record of the forward is a second graph made of nodes, one per operation, each of which knows how to compute the gradients of that operation. A tensor that was computed by an operation points to its node through grad_fn:

>>> loss.grad_fn
<SumBackward0 object at 0x...>
>>> loss.grad_fn.next_functions
((<AddBackward0 object at 0x...>, 0),)
>>> loss.grad_fn.next_functions[0][0].next_functions
((<MulBackward0 object at 0x...>, 0), (<AccumulateGrad object at 0x...>, 0))

The names say which forward operation each node belongs to: MulBackward0 is the backward of mul, and the 0 tells it apart from other variants of the same operation. next_functions lists the node’s edges: where each of the gradients it computes has to go. An edge is a pair: a node, and which of its inputs the gradient is for (pytorchedge.h, raxisgraph.h):

struct Edge {
    NodePtr node;
    uint32_t input_nr;
};

input_nr is the answer to the second point of section 3. A node with several outputs, such as the backward of unbind, receives several gradients, one per output, in numbered slots. An edge says “this gradient goes into slot input_nr of that node”, and every tensor remembers which output of its operation it was, its output_nr, so that the edge that leads to its gradient can be built.

The leaves, a and b, have no grad_fn: nothing computed them. The gradients that reach them still need a node to arrive at, and that node is AccumulateGrad: it takes the gradient and adds it into the leaf’s .grad. Each leaf has exactly one, shared by all the operations that use it (pytorchaccumulate_grad.cpp, raxisvariable.cpp). Here is the graph of the example, as the script prints it from PyTorch and from Raxis:

SumBackward0     seq 2   edges [('AddBackward0', 0)]
  AddBackward0     seq 1   edges [('MulBackward0', 0), ('AccumulateGrad', 0)]
    MulBackward0     seq 0   edges [('AccumulateGrad', 0), ('AccumulateGrad', 0)]

The seq column is each node’s sequence number, a counter that grows by one with every node created; section 8 uses it. In a figure, with the gradients of the table of section 1 written on the edges that carry them:

The graph recorded for loss = (a * b + a).sum()Each result tensor points to its backward node through grad_fn: loss to SumBackward0, d to AddBackward0, c to MulBackward0. The leaves a and b each have one AccumulateGrad node. SumBackward0 has one edge, to AddBackward0. AddBackward0 has two edges, to MulBackward0 and to the AccumulateGrad of a. MulBackward0 has two edges, to the AccumulateGrad of a and of b. During the backward the gradient [1, 1] goes from SumBackward0 to AddBackward0, and from AddBackward0 to both MulBackward0 and a's AccumulateGrad. MulBackward0 sends [4, 5] to a's AccumulateGrad and [2, 3] to b's. a's AccumulateGrad receives two gradients and adds them: [5, 6].tensorsbackward nodesloss = d.sum()grad_fnSumBackward0seq 2d = c + agrad_fnAddBackward0seq 1c = a * bgrad_fnMulBackward0seq 0, saves a and b[1, 1][1, 1][1, 1][4, 5][2, 3]AccumulateGradof aAccumulateGradof ba.grad = [1, 1] + [4, 5] = [5, 6]b.grad = [2, 3]Arrows between nodes are edges; the numbers are the gradients that travel along them.
The forward builds the nodes and edges; the backward walks them from the top, and a's AccumulateGrad waits for both of its gradients before it writes a.grad.

A node itself is little more than its edges, its sequence number and a virtual apply that takes the incoming gradients and returns one gradient per edge (pytorchnode.h, raxisgraph.h):

class Node {
public:
    virtual TensorList apply(const TensorList& grads) = 0;
    const EdgeList& next_edges() const { return next_edges_; }
    uint64_t sequence_nr() const { return sequence_nr_; }
    ...
private:
    uint64_t sequence_nr_;
    EdgeList next_edges_;
    ...
};

Who owns whom matters later, so it is worth stating now. A tensor owns its grad_fn: as long as loss exists, SumBackward0 exists. A node owns the nodes its edges point to: SumBackward0 keeps AddBackward0 alive, which keeps MulBackward0 alive. In Raxis these are std::shared_ptr<Node>, the standard reference-counted pointer: the object is freed when the last shared_ptr that points to it goes away. PyTorch uses its own c10::intrusive_ptr, which keeps the count inside the object, but the ownership is the same. So the whole graph hangs from the tensors at its top, and dropping loss frees it, from the top down. The leaves are the exception: a holds its AccumulateGrad only through a weak pointer, one that does not keep the node alive, because the node holds a.

5. Recording an operation

Who builds a node? The operation, right after it computes its result. In Raxis, mul with autograd is the plain mul plus a few lines, written once in a macro for every binary operation (raxisautograd_macros.h):

Tensor mul(const Tensor& a, const Tensor& b) {
    Tensor result = ops::mul(a, b);
    if (autograd::requires_autograd(a, b)) {
        auto grad_fn = std::make_shared<autograd::MulBackward0>(a, b);
        grad_fn->set_num_inputs(1);
        autograd::set_next_edges(grad_fn, a, b);
        autograd::set_autograd_meta(result, grad_fn);
    }
    return result;
}

Step by step:

  1. ops::mul computes the result, with the kernels of the earlier articles. It knows nothing about gradients.
  2. requires_autograd checks whether anything has to be recorded: grad mode is on (it is off inside torch.no_grad()) and at least one input requires grad.
  3. The node is created with the inputs, so that it can save what its formula will need (section 6).
  4. set_next_edges builds one edge per input. For an input computed by another operation, the edge goes to that input’s grad_fn, slot output_nr. For a leaf, it goes to its AccumulateGrad. For a tensor that does not require grad, it is an empty edge (raxisvariable.cpp).
  5. set_autograd_meta sets result.grad_fn and requires_grad.

The node’s apply is the local rule of the table, with the summing of section 3 for broadcast inputs (raxisbinary_backward.cpp):

TensorList MulBackward0::apply(const TensorList& grads) {
    const Tensor& grad = grads[0];
    Tensor grad_a, grad_b;
    if (a_requires_grad_)
        grad_a = unbroadcast(grad * b_.unpack(), a_shape_);
    if (b_requires_grad_)
        grad_b = unbroadcast(grad * a_.unpack(), b_shape_);
    return {grad_a, grad_b};
}

unbroadcast sums the gradient over the dimensions that broadcasting added or stretched, down to the input’s shape. PyTorch does that step in another place: its formulas return the gradient in the result’s shape, and the engine sums it down to the shape the edge expects, after every node, in validate_outputs (pytorchengine.cpp).

PyTorch has no hand-written MulBackward0. Its derivatives are listed in a file, one formula per input (pytorchderivatives.yaml):

- name: mul.Tensor(Tensor self, Tensor other) -> Tensor
  self: mul_tensor_backward(grad, other, self.scalar_type())
  other: mul_tensor_backward(grad, self, other.scalar_type())

and a code generator writes the node classes and the recording code from it when PyTorch is built. The file has 688 entries, and the generator turns them into the node classes of hundreds of operators without anyone writing the same twenty lines each time. It is also why, the first time you look for MulBackward0 in PyTorch’s source, you do not find it: it only exists in the build directory, in torch/csrc/autograd/generated/Functions.h. Raxis writes the nodes by hand, with macros for the common shapes, so that each one can be read and followed in an editor.

How much does recording cost? The script times a * b on 16 floats, without and with an input that requires grad:

Microseconds per callPyTorchRaxis
a * b, nothing requires grad1.110.90
w * b, w requires grad1.861.22
w * b inside no_grad()1.120.90

Recording adds 0.3 to 0.75 microseconds per operation: a node allocated, two edges, two tensors saved. Under no_grad() there is no extra cost at all, which is why evaluation and inference code wraps the forward in it. On Linux both libraries are about twice as fast (0.53, 0.91, 0.53 for PyTorch and 0.57, 0.77, 0.56 for Raxis): calling into C++ from Python costs more on Windows, in both, and I did not look into why.

6. Saved tensors, and the cycle that leaks a graph

MulBackward0 needs a and b in its backward, so it keeps them. Other nodes keep other things: SinBackward0 needs the input, because the derivative of sin⁡x\sin x is cos⁡x\cos x; ExpBackward0 needs the output instead, because the derivative of exe^x is exe^x itself and it is already computed; AddBackward0 needs nothing. Each saved tensor is a SavedVariable, a member of the node (raxissaved_variable.h):

class MulBackward0 : public Node {
    ...
    SavedVariable a_{this, "self"};
    SavedVariable b_{this, "other"};
};

Why not a plain Tensor member? Look at what saving the output would do. The output owns its node through grad_fn. If the node also owned the output, the two would own each other: output, then grad_fn, then node, then output again. A reference cycle. With reference counting, neither count can reach zero, so neither is ever freed, and since the node owns the rest of the graph through its edges, nothing below it is freed either: every tensor saved anywhere in that graph stays in memory for good.

Python has the same pattern and survives it. micrograd’s closure for out refers to out itself (out.grad), so a value owns its closure and the closure owns the value: a cycle. Python counts references too, but it also runs a cycle collector from time to time, which finds groups of objects that only refer to each other and frees them. C++ has no such thing: a std::shared_ptr cycle stays until the program ends.

This is not a hypothetical. SynapX, my second project, was an autograd engine in C++ over libtorch, and its nodes for ReLU, sigmoid, softmax, exp, sqrt, pow and max saved their output as a plain tensor. Every training step left its whole graph on the GPU, memory grew step after step until it ran out, and I could not find why at the time; it is one of the reasons I left the project in 2025. The fix, in October 2026, was a few lines (synapxfunctional.cpp): give the nodes a detached alias of each output, a new tensor that shares the output’s memory but has no grad_fn.

That is what SavedVariable does in PyTorch and Raxis. An input is saved as it is: its grad_fn is an earlier node, which the current node already owns through its edges, so no cycle can form. An output is saved as a detached alias (pytorchsaved_variable.cpp, raxissaved_variable.cpp):

void SavedVariable::save(const Tensor& tensor, bool is_output) {
    data_ = is_output && tensor.defined() ? tensor.detach() : tensor;
    ...
    saved_version_ = tensor.version();
}
Saving an output as it is makes a cycle; saving a detached alias does notArrows are owning pointers. Left: the output y owns its node ExpBackward0 through grad_fn, and the node owns y as its saved output, so y and the node own each other; the node also owns the rest of the graph through its edges, and none of it is ever freed. Right: the node saves a detached alias of y, a second tensor that shares y's memory and has no grad_fn. The node owns the alias, the alias owns only the memory, and nothing points back to y: when y is dropped, the node, the alias and the rest of the graph go with it.Output saved as it isyy's memoryExpBackward0grad_fnsavedthe rest ofthe graphedgesy and its node own each other: neither countreaches zero, and the whole graph stays.Output saved as a detached aliasyy's memoryExpBackward0grad_fnalias of ygrad_fn: Nonesavedthe rest ofthe graphedgesNothing points back to y. Drop y, and the node,the alias and the graph go with it.
Each arrow is an owning pointer, a std::shared_ptr. A detached alias shares the output's memory, so nothing is copied, but it has no grad_fn, so it cannot close the loop.

The memory is the same, so nothing is copied; only the link to the node is gone. The saved version is the other half of SavedVariable’s job: if an in-place operation writes into a saved tensor before the backward reads it, unpack() notices that the version has changed and raises “one of the variables needed for gradient computation has been modified by an inplace operation”. In-place operations are a topic of their own, which this article leaves out (section 12).

The same trap is open to anyone who writes a custom torch.autograd.Function in Python. Its forward receives a ctx object, and the documented way to keep a tensor for the backward is ctx.save_for_backward(y). If, instead, the function stores its output as an attribute, ctx.y = y, the cycle is back: y owns its node, the node owns ctx, ctx owns y. The script runs ten steps of such a function on a 4 MiB tensor on the GPU, with Python’s cycle collector paused, and then runs the collector:

GPU memory left after 10 steps, MBbefore the collector runsafter
PyTorch, ctx.y = y400
PyTorch, ctx.save_for_backward(y)00
Raxis, ctx.y = y4040
Raxis, ctx.save_for_backward(y)00

With ctx.y = y, every step leaves its 4 MiB behind. PyTorch tells Python’s collector what its C++ objects refer to, so the collector finds the cycle and frees it, whenever it happens to run. Raxis’s Python bindings do not, so the collector cannot see the cycle and the memory is never freed. That is a limitation of Raxis, listed in its known issues; in both libraries save_for_backward is the way that frees the memory at once.

7. The engine: a node runs when all its gradients have arrived

The graph is recorded; now loss.backward() has to walk it. micrograd builds the full topological order first, recursively, and then walks it. PyTorch does it differently, in two steps (pytorchengine.cpp, raxisengine.cpp):

  1. Count the dependencies. Starting from the root, loss.grad_fn, visit every node once, and count for each one how many edges point to it. That is how many gradients it will receive.
  2. Run every node whose count has reached zero. The root starts ready. When a node runs, it sends one gradient along each edge; the gradient goes into the receiving node’s input buffer, one slot per input, and the receiving node’s count goes down by one. When the count reaches zero, all its gradients are in its buffer, and it moves to the ready queue.

The second step is Kahn’s algorithm for a topological order, done while the backward runs instead of before. In Raxis the loop is (raxisengine.cpp, PyTorch’s in pytorchengine.cpp):

while (!ready.empty()) {
    NodeTask task = ready.pop();
    Node& node = *task.node;
    TensorList outputs = call_function(node, task.inputs);
    if (!keep_graph) {
        node.release_variables();
    }
    const EdgeList& edges = node.next_edges();
    for (size_t i = 0; i < edges.size(); ++i) {
        Node* next = edges[i].node.get();
        TensorList& buffer = not_ready[next];      // created on first use
        accumulate(buffer, edges[i].input_nr, std::move(outputs[i]));
        if (--dependencies.at(next) == 0) {
            ready.push({edges[i].node, std::move(buffer)});
            not_ready.erase(next);
        }
    }
}

On the example, the counts are 1 for AddBackward0, 1 for MulBackward0, 2 for a’s AccumulateGrad (it receives from AddBackward0 and from MulBackward0) and 1 for b’s. In the table, the nodes are shortened to Sum, Add, Mul, acc a and acc b, and the number after a waiting node is how many gradients it still expects:

StepRunsSendsStill waitingReady next
0Add 1, Mul 1
acc a 2, acc b 1
Sum
1Sum[1, 1] to AddMul 1
acc a 2, acc b 1
Add
2Add[1, 1] to Mul
[1, 1] to acc a
acc a 1, holding [1, 1]
acc b 1
Mul
3Mul[4, 5] to acc a
[2, 3] to acc b
acc a with [5, 6]
acc b with [2, 3]
4acc aa.grad = [5, 6]acc b
5acc bb.grad = [2, 3]

In step 3, the gradient for a from MulBackward0 lands in a buffer that already holds [1, 1], and the two are added there: [5, 6] reaches a’s AccumulateGrad already summed, and .grad is written once.

Why is this better than micrograd’s way? Three reasons, and the first is not the order itself: both are valid topological orders.

8. Which ready node runs first

In step 3 above, two nodes became ready at once. When there is a choice, which one runs? The ready queue is a priority queue, a structure that always hands out its largest element first, and the largest is the node with the highest sequence number (pytorchengine.h, raxisengine.cpp). Sequence numbers grow as the forward creates nodes, so the highest one is the operation that ran last in the forward: when the dependencies leave a choice, the backward follows the forward in reverse.

That matters for memory. When a graph splits into branches, running the most recent node first tends to carry one branch to its end before starting another, so fewer gradients sit in the buffers of half-finished branches at the same time. Any order allowed by the counts gives the same gradients; the sequence number only chooses among them.

AccumulateGrad breaks the rule on purpose: its sequence number is UINT64_MAX, the largest possible (pytorchaccumulate_grad.cpp, raxisvariable.cpp). A leaf’s gradient is final once it is ready, so as soon as it is, it is written into .grad and leaves the engine’s buffers, before any other node runs. In the table of section 7 that is steps 4 and 5: both accumulators became ready in step 3 and ran next. They have the same sequence number, so either may run first.

9. Adding gradients together

When two gradients reach the same slot, the buffer adds them. A parameter used by kk operations receives kk gradients, and adding them as old + new would allocate k−1k - 1 new tensors of the parameter’s size. So the engine adds in place, into the tensor already in the buffer, when that is safe (pytorchinput_buffer.cpp, raxisengine.cpp):

bool can_accumulate_inplace(const Tensor& t) {
    return t.use_count() == 1
        && t.unsafe_get_tensor_impl()->storage().use_count() == 1
        && t.is_non_overlapping_and_dense();
}

Safe means that nobody else can see the tensor being changed: the buffer holds the only reference to it and to its memory, and every element has its own address. The example meets both conditions. In step 2, AddBackward0 sends the same [1, 1] to both of its edges, because the gradient of an addition is just the incoming gradient: one tensor, in two buffers at once, MulBackward0’s and a’s accumulator’s. Had MulBackward0 still been waiting for another gradient, adding into that tensor in place would have changed the gradient it was going to read; the reference count catches that. And that [1, 1] came from SumBackward0, which returns the number 1 broadcast to the shape of d with a stride of 0, as a view: one float in memory, read twice. Adding [4, 5] into it in place would write both values into the same float; the density check catches that. So in step 3 the engine allocates a new tensor for [5, 6]. In a model, where most gradients are fresh tensors owned only by the buffer, it succeeds almost every time.

The last stop is AccumulateGrad writing .grad (raxisvariable.cpp). If .grad already exists, from an earlier backward, it adds into it in place, which is why gradients accumulate across backward() calls until the optimizer’s zero_grad() clears them. If not, it can steal the incoming gradient, use that tensor as .grad without a copy, when it is not shared with anything and has the same strides as the parameter; otherwise it copies it into a new tensor with the parameter’s strides. Why the strides? So that the optimizer, which updates the parameter and its gradient element by element, walks both in the same order through memory.

10. Freeing memory during the backward

A node’s saved tensors are only needed by its own apply. Once it has run, PyTorch frees them, unless you ask it to keep the graph: that is the release_variables() call in the loop of section 7 (pytorchengine.cpp, raxisgraph.cpp). So memory goes down as the backward advances, and the graph that is left at the end holds no tensors.

The script measures it on the GPU, where the allocator counts every byte: a 64 MiB tensor x goes through sin eight times, and the sum of the result is the loss. Each SinBackward0 saves its input, so the forward keeps x and the first seven results, eight tensors of 64 MiB:

MiB on the GPUafter the forwardpeak in the backwardafter it
backward()512640128
backward(retain_graph=True)512704576

The two libraries give the same numbers, to the megabyte. The peak is at the first node: it multiplies the incoming gradient by cos⁡\cos of its saved input, which needs a 64 MiB temporary for the cosine and 64 MiB for the result, 512 + 128 = 640. Then it frees its saved input, and so does every node after it, so each step needs no more than the first. At the end, only x and x.grad are left, 128 MiB. With retain_graph=True the saved tensors stay, so the last node runs with all of them still in memory (704), and they remain after the backward (576), ready for a second one.

Without retain_graph, a second backward() through the same graph finds the saved tensors gone and raises “Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed)” (raxissaved_variable.cpp). That error, which most people meet early on, is this optimization showing through.

11. Deep graphs

What happens with a very long chain of operations? micrograd’s build_topo calls itself once per value along the chain, and Python limits how deep calls can nest, 1000 by default:

Chain of micrograd multiplicationsbackward()
500ok
990ok
1000RecursionError
2000RecursionError

The limit can be raised, but not by much: each nested Python call also uses the stack of the C program that runs Python, and past some depth the process crashes instead of raising.

PyTorch’s engine has no recursion: counting the dependencies uses an explicit stack, and the main loop takes nodes from the ready queue. There is a second, less obvious place where recursion hides, though: freeing the graph. When the last reference to the top node goes away, its destructor drops its edges, which frees the next node, whose destructor frees the next one, and so on. Destructors calling destructors, one level per node: a chain of a hundred thousand nodes would overflow the stack of the program and crash it. PyTorch’s code points to the issue that reported it, pytorch#5534, from 2018. Both libraries now free the graph iteratively: a dying node moves the children that only it owns onto a list, and a loop frees them one at a time, so each destructor finds no children left to free recursively (pytorchfunction.cpp, raxisgraph.cpp). On a chain of 100,000 multiplications of 16 floats:

Millisecondsbackwardfreeing the graph
PyTorch28032
Raxis7714

On Linux, 151 and 6.8 for PyTorch, 39 and 4.4 for Raxis. Neither crashes, and freeing 100,000 nodes costs a fraction of a microsecond each.

The backward column shows the gap of section 3 again: PyTorch spends about four times longer per node. Its engine does more work around each node, which Raxis leaves out: validate_outputs checks the shape, type and device of every gradient a node returns and fixes them when they are reducible (the broadcast summing of section 5 is one of these fixes); each node runs under a guard that restores the thread’s settings, and with a hook for the profiler; the ready queue is protected by a lock, because PyTorch runs the backward of each GPU on a thread of its own; and gradients move between CUDA streams with the right synchronization. I did not measure how the 2 microseconds divide between these. For a model with large tensors they are invisible; for a graph of many tiny operations, as here, they are most of the time.

12. What this article leaves out

Raxis follows PyTorch in all of the above, and in more that this article leaves out:

In-place operations and views. x.add_(1) changes a tensor that a node may have saved, and changing a view changes its base. PyTorch handles both with version counters, rebase_history and the CopySlices node, and Raxis ports them. Edward Yang’s Autograd and Mutation explains them well; following them through the code is a possible topic for a later article.

Hooks, custom Functions and second derivatives. Tensor and node hooks run at fixed points of the loop of section 7; torch.autograd.Function lets Python define a node; create_graph=True records the backward itself, so it can be differentiated again. All three are in Raxis, and they are another possible topic.

What Raxis leaves out on purpose: the code generator (the nodes are written by hand, so they can be read), the thread per device (with one GPU it would gain little, and it is most of the engine’s complexity), forward-mode differentiation, and anomaly mode.

Key takeaways

Further reading

If something is not clear, or you think I got something wrong, leave a comment.