backward

→None

backward(tensors: Tensor | Sequence[Tensor], grad_tensors: Tensor | Sequence[Tensor | None] | None = None, retain_graph: bool | None = None, create_graph: bool = False, inputs: Tensor | Sequence[Tensor] | None = None)

source edit

Compute gradients of tensors w.r.t. the leaf variables in their graph.

Top-level entry point that triggers reverse-mode automatic differentiation across the computation graph rooted at tensors. For every leaf tensor x reachable from tensors whose requires_grad is True, this function accumulates $\partial \mathcal{L} / \partial x$ into x.grad, where $\mathcal{L}$ is the (possibly weighted) sum of the root tensors.

The chain rule is applied edge-by-edge during a topological walk of the graph in reverse order, so each intermediate Jacobian-vector product fires exactly once.

Parameters

tensorsTensor or list of Tensor

Root tensors at which the backward pass starts. When more than one root is supplied each receives its own seed and the contributions are summed at every shared leaf.

grad_tensorsTensor, sequence of (Tensor or None), or None= None

Seed cotangent vectors, one per root tensor, each of its root's shape and device (another real dtype is cast to the root's). A single tensor stands for a one-element sequence. None — for the whole argument or one entry — is an implicit 1, allowed only for a real root of one element.

retain_graphbool= None

If True the intermediate saved tensors are not freed after the backward pass, so the same graph can be traversed again. Defaults to create_graph.

create_graphbool= False

If True the operations performed during backward are themselves recorded in the graph, enabling higher-order differentiation (e.g. Hessian-vector products, meta-learning). Implies stronger memory usage. Defaults to False.

inputsTensor, sequence of Tensor, or None= None

Restrict accumulation to these tensors: only their .grad changes, every other leaf in the graph keeps what it had. An input that is not a leaf has its .grad filled too (it is marked Tensor.retain_grad, as the reference does). An input the roots do not reach is left alone. None (default) accumulates into every leaf.

Returns

None

Gradients are accumulated in-place onto each leaf tensor's .grad attribute. Existing .grad values are added to, not overwritten — call Tensor.zero_grad (or the optimizer's zero_grad) between successive backward passes if accumulation is undesired.

Raises

RuntimeError

A root does not require grad, the number of gradients differs from the number of roots, a non-scalar root has no gradient, inputs is empty, or one of inputs does not require grad.

TypeError

A gradient is neither a tensor nor None, or an entry of inputs is not a tensor.

(ShapeMismatch, DtypeMismatch, DeviceMismatch)

A gradient's shape or device differs from its root's, or only one of the two is complex.

Notes

Reverse-mode AD computes

\frac{\partial \mathcal{L}}{\partial x} = \sum_{i} \left( \frac{\partial \mathcal{L}}{\partial t_i} \right)^{\!\top} \frac{\partial t_i}{\partial x},

propagating cotangents $\bar t_i = \partial \mathcal{L} / \partial t_i$ from the roots through each saved op contract until every reachable leaf has received its contribution.

Memory/compute trade-off:

retain_graph=False (default) is the cheapest mode — once the walk finishes, every saved tensor is freed.
retain_graph=True, create_graph=False keeps activations so the same graph can be traversed again.
create_graph=True additionally records the backward ops in a new graph, doubling memory in the worst case but enabling $\nabla^2 \mathcal{L}$ and beyond.

Examples

>>> import lucid
>>> from lucid.autograd import backward
>>> x = lucid.tensor([1.0, 2.0, 3.0], requires_grad=True)
>>> y = (x * x).sum()
>>> backward(y)
>>> x.grad
tensor([2., 4., 6.])
Only the tensors in inputs accumulate:
>>> a = lucid.ones(2, requires_grad=True)
>>> b = lucid.ones(2, requires_grad=True)
>>> backward((a * b * 3).sum(), inputs=[a])
>>> a.grad, b.grad
(tensor([3., 3.]), None)

Used by 3

>>> import lucid >>> from lucid.autograd import backward >>> x = lucid.tensor([1.0, 2.0, 3.0], requires_grad=True) >>> y = (x * x).sum() >>> backward(y) >>> x.grad tensor([2., 4., 6.]) Only the tensors in inputs accumulate: >>> a = lucid.ones(2, requires_grad=True) >>> b = lucid.ones(2, requires_grad=True) >>> backward((a * b * 3).sum(), inputs=[a]) >>> a.grad, b.grad (tensor([3., 3.]), None)