From the chain rule to loss.backward()

A program is a chain of small operations whose derivatives we know. For y = tanh(w·x + b), write u = w·x and v = u + b. The chain rule says the derivative of the output with respect to anything is the product of the local derivatives along the way, summed over every path that connects them.

We can evaluate these products in two directions. Forward mode starts at one input and carries its derivative forward; it needs one pass per input. Reverse mode starts at the output with ∂y/∂y = 1 and carries the gradient backwards; one pass gives the derivative with respect to every input. A loss is one number and a model has millions of weights, so training uses reverse mode. That is what "backpropagation" is.

This is neither symbolic differentiation (no formula for the whole derivative is ever built) nor numerical differentiation (no small steps h). Each local derivative is exact, and the engine only multiplies and adds numbers. A variable used twice gets the sum of the gradients from both uses: that is the "summed over every path" part.

PyTorch does not compile our model. It runs our Python line by line ("define by run"), and each operation on a tensor with requires_grad=True does two things: computes its result, and creates a backward Node. The node is stored in the result's grad_fn. It keeps whatever its formula will need (for mul, both inputs; for tanh, its own result) and a list of next edges, one per input, pointing at the node that produced that input.

Every node also gets a sequence number from a counter that goes up by one per node created on this thread. It is how the engine will later decide which node to run first, and how the profiler links a forward operation to its backward node.

A tensor we created ourselves with requires_grad=True, such as a weight, has no grad_fn; it is a leaf. The edge that points at it leads to an AccumulateGrad node, which has no outputs. When it runs, it adds the gradient it received into the leaf's .grad.

Two lines of its source explain two everyday facts. Its constructor sets its sequence number to UINT64_MAX, so it runs as soon as it is ready and its gradient is written early. And the existing gradient is updated with variable_grad += new_grad: gradients accumulate across backward() calls. That is why every training loop calls optimizer.zero_grad().

Engine::execute starts with compute_dependencies: a walk over the graph from the root that counts, for each node, how many edges point at it. A node may only run when all of its incoming gradients have arrived, so this count is how many arrivals it waits for.

Then the root goes on a ready queue, a priority queue ordered by CompareNodeTaskTime: the task whose node has the higher sequence number runs first. The worker pops a task and calls evaluate_function, which runs the node's formula and, for each output, decrements the next node's dependency count and adds the gradient into that node's input buffer. When a count reaches zero, the node is pushed on the queue with its finished buffer. When the queue is empty, backward is done.

Why the highest sequence number? A node made later in the forward pass depends on nodes made earlier, so running later nodes first tends to finish gradients before they are needed. The dependency counts make it correct either way; the priority decides which ready node goes first. Karpathy's micrograd does the same job with an explicit topological sort.

Write Python as we would with PyTorch. The first lines that assign a number are leaves (torch.tensor(…, requires_grad=True)); after that, any expression with + - * / ** and tanh exp log sin cos sigmoid relu. The last line is the output, and we call backward() on it. Each step pops one node from the ready queue.

A node keeps the values its formula needs: MulBackward0 keeps both inputs, TanhBackward0 keeps its result. Those are saved tensors, and for a real model they are most of the memory a training step uses. So unless we ask for retain_graph=True, the engine frees them as soon as each node has run: evaluate_function calls release_variables() right after the formula.

The graph's structure survives, but its numbers are gone. Calling backward() on the same output a second time reaches a node whose saved values were freed, and PyTorch stops with the error below. (The lab's "backward() again" button does what a training loop does instead: a new forward pass, which records a new graph.)

Tensor.backward calls torch.autograd.backward. Because a loss is one number, it may skip the gradient argument: _make_grads creates torch.ones_like(out), which is the ∂loss/∂loss = 1 the chain starts from. For any other shape it raises "grad can be implicitly created only for scalar outputs". Then _engine_run_backward calls Variable._execution_engine.run_backward, the C++ Engine::execute above, with accumulate_grad=True (that is what makes it write .grad; torch.autograd.grad passes False and returns the gradients instead).

Two views from real PyTorch 2.14.1. On the left, a pre-hook on every node of our mixed example recorded the order the engine ran them, with each node's sequence number and the gradient it received; the lab's preset "Our mixed example" gives the same order. On the right, the profiler recorded the neuron: forward operations carry the same sequence numbers as the backward nodes they created, and inside MulBackward0 we can see the two multiplications of its formula.

Five questions in the style of a machine learning or numerical methods exam. Write our answer first, then open the model answer and mark ourselves honestly. Numbers and orders in the answers were produced by the lab's engine and match PyTorch.