How PyTorch computes gradients
pgmesa · · 35 min read
Part of Rebuilding PyTorch (7 of 8)
- Why I rebuilt PyTorch's core from scratch
- What a PyTorch tensor really is
- Interlude: What the CPU does when your code reads memory
- How PyTorch walks a tensor of any shape
- How PyTorch prepares the loop of every operation
- How a PyTorch kernel uses the vector registers of every CPU
- How PyTorch sums a tensor, and splits the work between threads
- How PyTorch computes gradients
- How PyTorch runs the same operations on a GPU
Everything so far was about making one operation fast. Training a model needs something else as well: after the forward, the gradient of the loss with respect to every parameter. PyTorch computes it with loss.backward(), and the idea behind it fits in a few dozen lines of Python, as Andrej Karpathy’s micrograd shows. This article starts from those lines and follows what has to change when the values are tensors, the graphs have thousands of operations, and the memory they hold is the GPU’s. Most of the answers are about memory, not about math: what a node may keep, when it can let it go, and how a single wrong pointer keeps a whole graph alive.
Before you start. This article needs much less of the previous ones than they needed of each other: what a tensor is (What a tensor is, for views and shared storage) and what broadcasting does (How PyTorch prepares the loop of every operation). The math is the chain rule for derivatives, and each step is worked out with numbers. The code is C++ with
std::shared_ptr, the reference-counted pointer of the standard library, explained where it matters.
This is the path the article follows:
- What a backward computes.
- micrograd: the whole idea in thirty lines.
- From numbers to tensors.
- The graph: nodes and edges.
- Recording an operation.
- Saved tensors, and the cycle that leaks a graph.
- The engine: a node runs when all its gradients have arrived.
- Which ready node runs first.
- Adding gradients together.
- Freeing memory during the backward.
- Deep graphs.
- What this article leaves out.
And at the end, the key takeaways.
The numbers come from raxisautograd.py, run in November 2026 with Python 3.12.16, PyTorch 2.14.1 and Raxis 0.5.0, on a Ryzen 7 9700X with an RTX 5070 Ti, on one thread. The CPU tables are from Windows 11 25H2 and the text gives the Linux numbers (Ubuntu 24.04 under WSL2, same machine) where they differ; the GPU tables are from Windows only.
1. What a backward computes
Take two small tensors and a loss built from them:
a = torch.tensor([2., 3.], requires_grad=True)
b = torch.tensor([4., 5.], requires_grad=True)
c = a * b # [8, 15]
d = c + a # [10, 18]
loss = d.sum() # 28
loss.backward()
a.grad, b.grad # [5, 6], [2, 3]
requires_grad=True marks a and b as the tensors whose gradients we want, the leaves of the computation. The gradient of loss with respect to a says how much loss changes when each element of a changes a little. Written element by element, the loss is
so changing by a small amount changes the loss by , and changing changes it by :
That was easy because the whole expression fits in one line. A model is a long chain of operations, and nobody writes its derivative by hand. The chain rule is what makes it mechanical: if loss depends on d, and d depends on c, the gradient of loss with respect to c is the gradient with respect to d times the derivative of d with respect to c. So each operation only has to know its own local derivative, and the gradients can be passed backwards from the loss, one operation at a time:
| Step | Operation | Receives | Local rule | Sends |
|---|---|---|---|---|
| 1 | loss = d.sum() | 1 | each counts once | to d: [1, 1] |
| 2 | d = c + a | [1, 1] | pass it on | to c: [1, 1]to a: [1, 1] |
| 3 | c = a * b | [1, 1] | times the other factor | to a: [1, 1] * b = [4, 5]to b: [1, 1] * a = [2, 3] |
| 4 | leaf a | [1, 1], [4, 5] | add them | a.grad = [5, 6] |
| 5 | leaf b | [2, 3] | add them | b.grad = [2, 3] |
This is reverse-mode automatic differentiation, or backpropagation. Three things in the table are the whole problem in small:
- The forward has to be recorded. Step 3 needs the values of
aandb, so the multiplication has to keep them from the forward until the backward. - The order matters. Step 2 cannot run before step 1 has given it its gradient, and step 4 must wait for two gradients, one from step 2 and one from step 3.
- Gradients that reach the same tensor add up.
ais used twice, so it receives two gradients, and its gradient is their sum.
The rest of the article is how PyTorch does each of the three, and why.
2. micrograd: the whole idea in thirty lines
micrograd, which Karpathy wrote in 2020 to teach backpropagation, does all of this for single numbers. A Value holds a number, its gradient, the values it was computed from, and a function that knows how to pass its gradient back to them (micrograd/engine.py):
class Value:
def __init__(self, data, _children=(), _op=''):
self.data = data
self.grad = 0
self._backward = lambda: None
self._prev = set(_children)
def __mul__(self, other):
out = Value(self.data * other.data, (self, other), '*')
def _backward():
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = _backward
return out
Every operation creates its result and attaches to it a small function, a closure: a function that remembers the variables around it, here self, other and out. That closure is step 3 of the table: multiply the incoming gradient by the other factor, and add, with +=, into the gradients of the inputs. The += is the third point of the list above, done in the simplest way: every value has a grad that starts at 0 and collects everything that arrives.
The second point, the order, is the backward method:
def backward(self):
topo = []
visited = set()
def build_topo(v):
if v not in visited:
visited.add(v)
for child in v._prev:
build_topo(child)
topo.append(v)
build_topo(self)
self.grad = 1
for v in reversed(topo):
v._backward()
build_topo visits the values recursively, depth first, and appends each one only after all the values it was computed from. The result is a topological order: a list in which every value comes after its inputs. Walking it backwards, every value runs its closure only after everything computed from it has added its gradient. My first project, SynapGrad, did exactly this with NumPy arrays instead of numbers, and so do most small autograd engines.
It is correct, and it is short. What does PyTorch do differently, then, and why does it need to?
3. From numbers to tensors
Four things change when the values are tensors and the programs are real models.
Shapes. With numbers, the gradient of an input has the shape of the input, always: one number. With tensors, broadcasting breaks that. If x has shape [2, 3] and w has shape [3], x * w reads every element of w twice, once per row, so the gradient of w is the sum, over the rows, of the gradient of each use:
where is the gradient that reaches x * w. The local rule “multiply by the other factor” gives a [2, 3] tensor, and it has to be summed down to the [3] of w. Every operation that broadcasts has to do this.
Several inputs and outputs. torch.max(x, dim=1) returns two tensors, the values and their positions; x.unbind(0) returns each row of x as a tensor of its own. The gradient machinery has to know which output a gradient belongs to, and which input of the next operation it goes to.
Memory. A micrograd value holds one number. A tensor saved for the backward can hold hundreds of megabytes, and a model saves many of them. When each one is freed matters, and a graph that is not freed at all runs the GPU out of memory within a few training steps.
Size. A training step of a large model records thousands of operations, and some programs record far more: a recurrent network unrolled over a long sequence, or a loop of small operations. Section 11 shows where recursion stops.
And one thing changes in favour of the tensors: speed per operation stops being the main concern. These are the costs per recorded operation, measured on a chain of 500 multiplications, for micrograd’s numbers (the class is written out in the script) and for tensors of 16 floats:
| Per node, microseconds | micrograd | PyTorch | Raxis |
|---|---|---|---|
| forward (compute and record) | 0.28 | 1.39 | 1.07 |
| backward | 0.25 | 2.10 | 0.45 |
micrograd is faster per node. A Python object with a closure is cheap, and the tensor libraries pay to go from Python into C++, create a tensor, and check its shape and type. But a micrograd node does one multiplication, and a tensor node does one multiplication for every element: at 16 floats the tensor libraries are already ahead per number, and at the sizes of a model the bookkeeping per node disappears next to the work. That PyTorch’s backward takes 2.1 microseconds per node and Raxis’s 0.45 is section 11’s question. On Linux every number of the tensor libraries is lower (0.71 and 0.71 for the forward, 1.47 and 0.36 for the backward) and micrograd’s are the same.
4. The graph: nodes and edges
In micrograd, the graph is made of the values themselves: each value points to its inputs (_prev) and carries its closure. PyTorch separates the two. The tensors stay as they are, and the record of the forward is a second graph made of nodes, one per operation, each of which knows how to compute the gradients of that operation. A tensor that was computed by an operation points to its node through grad_fn:
>>> loss.grad_fn
<SumBackward0 object at 0x...>
>>> loss.grad_fn.next_functions
((<AddBackward0 object at 0x...>, 0),)
>>> loss.grad_fn.next_functions[0][0].next_functions
((<MulBackward0 object at 0x...>, 0), (<AccumulateGrad object at 0x...>, 0))
The names say which forward operation each node belongs to: MulBackward0 is the backward of mul, and the 0 tells it apart from other variants of the same operation. next_functions lists the node’s edges: where each of the gradients it computes has to go. An edge is a pair: a node, and which of its inputs the gradient is for (pytorchedge.h, raxisgraph.h):
struct Edge {
NodePtr node;
uint32_t input_nr;
};
input_nr is the answer to the second point of section 3. A node with several outputs, such as the backward of unbind, receives several gradients, one per output, in numbered slots. An edge says “this gradient goes into slot input_nr of that node”, and every tensor remembers which output of its operation it was, its output_nr, so that the edge that leads to its gradient can be built.
The leaves, a and b, have no grad_fn: nothing computed them. The gradients that reach them still need a node to arrive at, and that node is AccumulateGrad: it takes the gradient and adds it into the leaf’s .grad. Each leaf has exactly one, shared by all the operations that use it (pytorchaccumulate_grad.cpp, raxisvariable.cpp). Here is the graph of the example, as the script prints it from PyTorch and from Raxis:
SumBackward0 seq 2 edges [('AddBackward0', 0)]
AddBackward0 seq 1 edges [('MulBackward0', 0), ('AccumulateGrad', 0)]
MulBackward0 seq 0 edges [('AccumulateGrad', 0), ('AccumulateGrad', 0)]
The seq column is each node’s sequence number, a counter that grows by one with every node created; section 8 uses it. In a figure, with the gradients of the table of section 1 written on the edges that carry them:
A node itself is little more than its edges, its sequence number and a virtual apply that takes the incoming gradients and returns one gradient per edge (pytorchnode.h, raxisgraph.h):
class Node {
public:
virtual TensorList apply(const TensorList& grads) = 0;
const EdgeList& next_edges() const { return next_edges_; }
uint64_t sequence_nr() const { return sequence_nr_; }
...
private:
uint64_t sequence_nr_;
EdgeList next_edges_;
...
};
Who owns whom matters later, so it is worth stating now. A tensor owns its grad_fn: as long as loss exists, SumBackward0 exists. A node owns the nodes its edges point to: SumBackward0 keeps AddBackward0 alive, which keeps MulBackward0 alive. In Raxis these are std::shared_ptr<Node>, the standard reference-counted pointer: the object is freed when the last shared_ptr that points to it goes away. PyTorch uses its own c10::intrusive_ptr, which keeps the count inside the object, but the ownership is the same. So the whole graph hangs from the tensors at its top, and dropping loss frees it, from the top down. The leaves are the exception: a holds its AccumulateGrad only through a weak pointer, one that does not keep the node alive, because the node holds a.
5. Recording an operation
Who builds a node? The operation, right after it computes its result. In Raxis, mul with autograd is the plain mul plus a few lines, written once in a macro for every binary operation (raxisautograd_macros.h):
Tensor mul(const Tensor& a, const Tensor& b) {
Tensor result = ops::mul(a, b);
if (autograd::requires_autograd(a, b)) {
auto grad_fn = std::make_shared<autograd::MulBackward0>(a, b);
grad_fn->set_num_inputs(1);
autograd::set_next_edges(grad_fn, a, b);
autograd::set_autograd_meta(result, grad_fn);
}
return result;
}
Step by step:
ops::mulcomputes the result, with the kernels of the earlier articles. It knows nothing about gradients.requires_autogradchecks whether anything has to be recorded: grad mode is on (it is off insidetorch.no_grad()) and at least one input requires grad.- The node is created with the inputs, so that it can save what its formula will need (section 6).
set_next_edgesbuilds one edge per input. For an input computed by another operation, the edge goes to that input’sgrad_fn, slotoutput_nr. For a leaf, it goes to itsAccumulateGrad. For a tensor that does not require grad, it is an empty edge (raxisvariable.cpp).set_autograd_metasetsresult.grad_fnandrequires_grad.
The node’s apply is the local rule of the table, with the summing of section 3 for broadcast inputs (raxisbinary_backward.cpp):
TensorList MulBackward0::apply(const TensorList& grads) {
const Tensor& grad = grads[0];
Tensor grad_a, grad_b;
if (a_requires_grad_)
grad_a = unbroadcast(grad * b_.unpack(), a_shape_);
if (b_requires_grad_)
grad_b = unbroadcast(grad * a_.unpack(), b_shape_);
return {grad_a, grad_b};
}
unbroadcast sums the gradient over the dimensions that broadcasting added or stretched, down to the input’s shape. PyTorch does that step in another place: its formulas return the gradient in the result’s shape, and the engine sums it down to the shape the edge expects, after every node, in validate_outputs (pytorchengine.cpp).
PyTorch has no hand-written MulBackward0. Its derivatives are listed in a file, one formula per input (pytorchderivatives.yaml):
- name: mul.Tensor(Tensor self, Tensor other) -> Tensor
self: mul_tensor_backward(grad, other, self.scalar_type())
other: mul_tensor_backward(grad, self, other.scalar_type())
and a code generator writes the node classes and the recording code from it when PyTorch is built. The file has 688 entries, and the generator turns them into the node classes of hundreds of operators without anyone writing the same twenty lines each time. It is also why, the first time you look for MulBackward0 in PyTorch’s source, you do not find it: it only exists in the build directory, in torch/csrc/autograd/generated/Functions.h. Raxis writes the nodes by hand, with macros for the common shapes, so that each one can be read and followed in an editor.
How much does recording cost? The script times a * b on 16 floats, without and with an input that requires grad:
| Microseconds per call | PyTorch | Raxis |
|---|---|---|
a * b, nothing requires grad | 1.11 | 0.90 |
w * b, w requires grad | 1.86 | 1.22 |
w * b inside no_grad() | 1.12 | 0.90 |
Recording adds 0.3 to 0.75 microseconds per operation: a node allocated, two edges, two tensors saved. Under no_grad() there is no extra cost at all, which is why evaluation and inference code wraps the forward in it. On Linux both libraries are about twice as fast (0.53, 0.91, 0.53 for PyTorch and 0.57, 0.77, 0.56 for Raxis): calling into C++ from Python costs more on Windows, in both, and I did not look into why.
6. Saved tensors, and the cycle that leaks a graph
MulBackward0 needs a and b in its backward, so it keeps them. Other nodes keep other things: SinBackward0 needs the input, because the derivative of is ; ExpBackward0 needs the output instead, because the derivative of is itself and it is already computed; AddBackward0 needs nothing. Each saved tensor is a SavedVariable, a member of the node (raxissaved_variable.h):
class MulBackward0 : public Node {
...
SavedVariable a_{this, "self"};
SavedVariable b_{this, "other"};
};
Why not a plain Tensor member? Look at what saving the output would do. The output owns its node through grad_fn. If the node also owned the output, the two would own each other: output, then grad_fn, then node, then output again. A reference cycle. With reference counting, neither count can reach zero, so neither is ever freed, and since the node owns the rest of the graph through its edges, nothing below it is freed either: every tensor saved anywhere in that graph stays in memory for good.
Python has the same pattern and survives it. micrograd’s closure for out refers to out itself (out.grad), so a value owns its closure and the closure owns the value: a cycle. Python counts references too, but it also runs a cycle collector from time to time, which finds groups of objects that only refer to each other and frees them. C++ has no such thing: a std::shared_ptr cycle stays until the program ends.
This is not a hypothetical. SynapX, my second project, was an autograd engine in C++ over libtorch, and its nodes for ReLU, sigmoid, softmax, exp, sqrt, pow and max saved their output as a plain tensor. Every training step left its whole graph on the GPU, memory grew step after step until it ran out, and I could not find why at the time; it is one of the reasons I left the project in 2025. The fix, in October 2026, was a few lines (synapxfunctional.cpp): give the nodes a detached alias of each output, a new tensor that shares the output’s memory but has no grad_fn.
That is what SavedVariable does in PyTorch and Raxis. An input is saved as it is: its grad_fn is an earlier node, which the current node already owns through its edges, so no cycle can form. An output is saved as a detached alias (pytorchsaved_variable.cpp, raxissaved_variable.cpp):
void SavedVariable::save(const Tensor& tensor, bool is_output) {
data_ = is_output && tensor.defined() ? tensor.detach() : tensor;
...
saved_version_ = tensor.version();
}
std::shared_ptr. A detached alias shares the output's memory, so nothing is copied, but it has no grad_fn, so it cannot close the loop.The memory is the same, so nothing is copied; only the link to the node is gone. The saved version is the other half of SavedVariable’s job: if an in-place operation writes into a saved tensor before the backward reads it, unpack() notices that the version has changed and raises “one of the variables needed for gradient computation has been modified by an inplace operation”. In-place operations are a topic of their own, which this article leaves out (section 12).
The same trap is open to anyone who writes a custom torch.autograd.Function in Python. Its forward receives a ctx object, and the documented way to keep a tensor for the backward is ctx.save_for_backward(y). If, instead, the function stores its output as an attribute, ctx.y = y, the cycle is back: y owns its node, the node owns ctx, ctx owns y. The script runs ten steps of such a function on a 4 MiB tensor on the GPU, with Python’s cycle collector paused, and then runs the collector:
| GPU memory left after 10 steps, MB | before the collector runs | after |
|---|---|---|
PyTorch, ctx.y = y | 40 | 0 |
PyTorch, ctx.save_for_backward(y) | 0 | 0 |
Raxis, ctx.y = y | 40 | 40 |
Raxis, ctx.save_for_backward(y) | 0 | 0 |
With ctx.y = y, every step leaves its 4 MiB behind. PyTorch tells Python’s collector what its C++ objects refer to, so the collector finds the cycle and frees it, whenever it happens to run. Raxis’s Python bindings do not, so the collector cannot see the cycle and the memory is never freed. That is a limitation of Raxis, listed in its known issues; in both libraries save_for_backward is the way that frees the memory at once.
7. The engine: a node runs when all its gradients have arrived
The graph is recorded; now loss.backward() has to walk it. micrograd builds the full topological order first, recursively, and then walks it. PyTorch does it differently, in two steps (pytorchengine.cpp, raxisengine.cpp):
- Count the dependencies. Starting from the root,
loss.grad_fn, visit every node once, and count for each one how many edges point to it. That is how many gradients it will receive. - Run every node whose count has reached zero. The root starts ready. When a node runs, it sends one gradient along each edge; the gradient goes into the receiving node’s input buffer, one slot per input, and the receiving node’s count goes down by one. When the count reaches zero, all its gradients are in its buffer, and it moves to the ready queue.
The second step is Kahn’s algorithm for a topological order, done while the backward runs instead of before. In Raxis the loop is (raxisengine.cpp, PyTorch’s in pytorchengine.cpp):
while (!ready.empty()) {
NodeTask task = ready.pop();
Node& node = *task.node;
TensorList outputs = call_function(node, task.inputs);
if (!keep_graph) {
node.release_variables();
}
const EdgeList& edges = node.next_edges();
for (size_t i = 0; i < edges.size(); ++i) {
Node* next = edges[i].node.get();
TensorList& buffer = not_ready[next]; // created on first use
accumulate(buffer, edges[i].input_nr, std::move(outputs[i]));
if (--dependencies.at(next) == 0) {
ready.push({edges[i].node, std::move(buffer)});
not_ready.erase(next);
}
}
}
On the example, the counts are 1 for AddBackward0, 1 for MulBackward0, 2 for a’s AccumulateGrad (it receives from AddBackward0 and from MulBackward0) and 1 for b’s. In the table, the nodes are shortened to Sum, Add, Mul, acc a and acc b, and the number after a waiting node is how many gradients it still expects:
| Step | Runs | Sends | Still waiting | Ready next |
|---|---|---|---|---|
| 0 | Add 1, Mul 1 acc a 2, acc b 1 | Sum | ||
| 1 | Sum | [1, 1] to Add | Mul 1 acc a 2, acc b 1 | Add |
| 2 | Add | [1, 1] to Mul [1, 1] to acc a | acc a 1, holding [1, 1] acc b 1 | Mul |
| 3 | Mul | [4, 5] to acc a [2, 3] to acc b | acc a with [5, 6] acc b with [2, 3] | |
| 4 | acc a | a.grad = [5, 6] | acc b | |
| 5 | acc b | b.grad = [2, 3] |
In step 3, the gradient for a from MulBackward0 lands in a buffer that already holds [1, 1], and the two are added there: [5, 6] reaches a’s AccumulateGrad already summed, and .grad is written once.
Why is this better than micrograd’s way? Three reasons, and the first is not the order itself: both are valid topological orders.
- Gradients live only while they are needed. micrograd gives every value a
.gradand keeps all of them until the graph dies. Here, a gradient exists only in the buffer of a node that has not run yet, and the buffer is handed to the node and dropped when it runs. Only the leaves keep theirs, in.grad. For an intermediate tensor of 100 MB, that is the difference between keeping its gradient for the whole backward and keeping it for one step. - No recursion. Counting the dependencies uses an explicit stack (a
std::vectorof nodes still to visit), and the loop above has no recursion either. Section 11 measures what recursion costs. - Part of the graph can be skipped.
torch.autograd.grad(loss, [x])asks for the gradient of one tensor only. The engine first marks the nodes that lie on a path tox(raxisengine.cpp) and runs only those; the rest of the graph is never touched.
8. Which ready node runs first
In step 3 above, two nodes became ready at once. When there is a choice, which one runs? The ready queue is a priority queue, a structure that always hands out its largest element first, and the largest is the node with the highest sequence number (pytorchengine.h, raxisengine.cpp). Sequence numbers grow as the forward creates nodes, so the highest one is the operation that ran last in the forward: when the dependencies leave a choice, the backward follows the forward in reverse.
That matters for memory. When a graph splits into branches, running the most recent node first tends to carry one branch to its end before starting another, so fewer gradients sit in the buffers of half-finished branches at the same time. Any order allowed by the counts gives the same gradients; the sequence number only chooses among them.
AccumulateGrad breaks the rule on purpose: its sequence number is UINT64_MAX, the largest possible (pytorchaccumulate_grad.cpp, raxisvariable.cpp). A leaf’s gradient is final once it is ready, so as soon as it is, it is written into .grad and leaves the engine’s buffers, before any other node runs. In the table of section 7 that is steps 4 and 5: both accumulators became ready in step 3 and ran next. They have the same sequence number, so either may run first.
9. Adding gradients together
When two gradients reach the same slot, the buffer adds them. A parameter used by operations receives gradients, and adding them as old + new would allocate new tensors of the parameter’s size. So the engine adds in place, into the tensor already in the buffer, when that is safe (pytorchinput_buffer.cpp, raxisengine.cpp):
bool can_accumulate_inplace(const Tensor& t) {
return t.use_count() == 1
&& t.unsafe_get_tensor_impl()->storage().use_count() == 1
&& t.is_non_overlapping_and_dense();
}
Safe means that nobody else can see the tensor being changed: the buffer holds the only reference to it and to its memory, and every element has its own address. The example meets both conditions. In step 2, AddBackward0 sends the same [1, 1] to both of its edges, because the gradient of an addition is just the incoming gradient: one tensor, in two buffers at once, MulBackward0’s and a’s accumulator’s. Had MulBackward0 still been waiting for another gradient, adding into that tensor in place would have changed the gradient it was going to read; the reference count catches that. And that [1, 1] came from SumBackward0, which returns the number 1 broadcast to the shape of d with a stride of 0, as a view: one float in memory, read twice. Adding [4, 5] into it in place would write both values into the same float; the density check catches that. So in step 3 the engine allocates a new tensor for [5, 6]. In a model, where most gradients are fresh tensors owned only by the buffer, it succeeds almost every time.
The last stop is AccumulateGrad writing .grad (raxisvariable.cpp). If .grad already exists, from an earlier backward, it adds into it in place, which is why gradients accumulate across backward() calls until the optimizer’s zero_grad() clears them. If not, it can steal the incoming gradient, use that tensor as .grad without a copy, when it is not shared with anything and has the same strides as the parameter; otherwise it copies it into a new tensor with the parameter’s strides. Why the strides? So that the optimizer, which updates the parameter and its gradient element by element, walks both in the same order through memory.
10. Freeing memory during the backward
A node’s saved tensors are only needed by its own apply. Once it has run, PyTorch frees them, unless you ask it to keep the graph: that is the release_variables() call in the loop of section 7 (pytorchengine.cpp, raxisgraph.cpp). So memory goes down as the backward advances, and the graph that is left at the end holds no tensors.
The script measures it on the GPU, where the allocator counts every byte: a 64 MiB tensor x goes through sin eight times, and the sum of the result is the loss. Each SinBackward0 saves its input, so the forward keeps x and the first seven results, eight tensors of 64 MiB:
| MiB on the GPU | after the forward | peak in the backward | after it |
|---|---|---|---|
backward() | 512 | 640 | 128 |
backward(retain_graph=True) | 512 | 704 | 576 |
The two libraries give the same numbers, to the megabyte. The peak is at the first node: it multiplies the incoming gradient by of its saved input, which needs a 64 MiB temporary for the cosine and 64 MiB for the result, 512 + 128 = 640. Then it frees its saved input, and so does every node after it, so each step needs no more than the first. At the end, only x and x.grad are left, 128 MiB. With retain_graph=True the saved tensors stay, so the last node runs with all of them still in memory (704), and they remain after the backward (576), ready for a second one.
Without retain_graph, a second backward() through the same graph finds the saved tensors gone and raises “Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed)” (raxissaved_variable.cpp). That error, which most people meet early on, is this optimization showing through.
11. Deep graphs
What happens with a very long chain of operations? micrograd’s build_topo calls itself once per value along the chain, and Python limits how deep calls can nest, 1000 by default:
| Chain of micrograd multiplications | backward() |
|---|---|
| 500 | ok |
| 990 | ok |
| 1000 | RecursionError |
| 2000 | RecursionError |
The limit can be raised, but not by much: each nested Python call also uses the stack of the C program that runs Python, and past some depth the process crashes instead of raising.
PyTorch’s engine has no recursion: counting the dependencies uses an explicit stack, and the main loop takes nodes from the ready queue. There is a second, less obvious place where recursion hides, though: freeing the graph. When the last reference to the top node goes away, its destructor drops its edges, which frees the next node, whose destructor frees the next one, and so on. Destructors calling destructors, one level per node: a chain of a hundred thousand nodes would overflow the stack of the program and crash it. PyTorch’s code points to the issue that reported it, pytorch#5534, from 2018. Both libraries now free the graph iteratively: a dying node moves the children that only it owns onto a list, and a loop frees them one at a time, so each destructor finds no children left to free recursively (pytorchfunction.cpp, raxisgraph.cpp). On a chain of 100,000 multiplications of 16 floats:
| Milliseconds | backward | freeing the graph |
|---|---|---|
| PyTorch | 280 | 32 |
| Raxis | 77 | 14 |
On Linux, 151 and 6.8 for PyTorch, 39 and 4.4 for Raxis. Neither crashes, and freeing 100,000 nodes costs a fraction of a microsecond each.
The backward column shows the gap of section 3 again: PyTorch spends about four times longer per node. Its engine does more work around each node, which Raxis leaves out: validate_outputs checks the shape, type and device of every gradient a node returns and fixes them when they are reducible (the broadcast summing of section 5 is one of these fixes); each node runs under a guard that restores the thread’s settings, and with a hook for the profiler; the ready queue is protected by a lock, because PyTorch runs the backward of each GPU on a thread of its own; and gradients move between CUDA streams with the right synchronization. I did not measure how the 2 microseconds divide between these. For a model with large tensors they are invisible; for a graph of many tiny operations, as here, they are most of the time.
12. What this article leaves out
Raxis follows PyTorch in all of the above, and in more that this article leaves out:
In-place operations and views.
x.add_(1)changes a tensor that a node may have saved, and changing a view changes its base. PyTorch handles both with version counters,rebase_historyand theCopySlicesnode, and Raxis ports them. Edward Yang’s Autograd and Mutation explains them well; following them through the code is a possible topic for a later article.
Hooks, custom Functions and second derivatives. Tensor and node hooks run at fixed points of the loop of section 7;
torch.autograd.Functionlets Python define a node;create_graph=Truerecords the backward itself, so it can be differentiated again. All three are in Raxis, and they are another possible topic.
What Raxis leaves out on purpose: the code generator (the nodes are written by hand, so they can be read), the thread per device (with one GPU it would gain little, and it is most of the engine’s complexity), forward-mode differentiation, and anomaly mode.
Key takeaways
- The idea fits in micrograd’s thirty lines (section 2). What PyTorch adds is mostly about memory, not math.
- A node never owns its own output (section 6). Outputs are saved as detached aliases; otherwise the whole graph leaks, as it did in SynapX.
- Nothing recurses (sections 7 and 11). A node runs when all the gradients it waits for have arrived, and even freeing the graph is a loop, so a chain of 100,000 operations is fine.
- Memory goes down during the backward (section 10). Each node frees its saved tensors as soon as it has run, unless
retain_graph=True.
Further reading
- Andrej Karpathy’s micrograd, and his video The spelled-out intro to neural networks and backpropagation, which builds it line by line.
- The PyTorch documentation, Autograd mechanics: what is saved,
retain_graph, in-place operations, multithreading. - Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul and Jeffrey Mark Siskind, “Automatic differentiation in machine learning: a survey”, JMLR, 2018: forward and reverse mode, and where each one wins.
- Edward Yang, PyTorch internals (2019), for where autograd sits among the rest of PyTorch, and Autograd and Mutation (2026) for what comes next.
If something is not clear, or you think I got something wrong, leave a comment.