Tensor

The chain rule and reverse-mode AD

Given a scalar loss $\mathcal{L}$ and a sequence of operations $\mathbf{z} = f_n(\cdots f_2(f_1(\mathbf{x})) \cdots)$, the chain rule gives:

where $J_{f_i}$ is the Jacobian of the $i$-th operation. Reverse-mode AD (backpropagation) evaluates this product right-to-left, starting from the scalar output with seed gradient $\frac{\partial \mathcal{L}}{\partial \mathcal{L}} = 1$, so it computes gradients for all inputs in a single backward pass — $\mathcal{O}(1)$ passes regardless of the number of parameters.

Gradient accumulation

Gradients are added to tensor.grad rather than overwritten. This is intentional: it supports patterns like accumulated gradient steps. Zero gradients explicitly before each optimisation step:

for param in model.parameters():
    param.grad = None   # or param.grad.zero_()

Returns self unchanged when the source dtype is already bool. Otherwise each element is reduced to True if nonzero and False if zero.

No-op when the tensor is already on CPU. On Apple Silicon both CPU and Metal share the same physical DRAM (unified memory); this call relabels the dispatch target so subsequent ops route through Apple Accelerate (vDSP / vForce / BLAS / LAPACK).

The address says where the tensor's values are now, not which tensor it is: an in-place op that computes its result into a new buffer moves it. Compare addresses to ask whether two tensors read the same memory at this moment.

Equivalent to len(tensor.shape); matches the conventional rank $d$ such that $\text{shape} \in \mathbb{N}^d$.

Returns self unchanged when the source dtype is already float64. Otherwise allocates a new tensor with widened precision; device is preserved.

The total memory footprint of the tensor is:

Broadcasting rules follow the standard right-aligned semantics: each dimension of self.shape must either equal the corresponding entry of other.shape or be 1. Size-1 axes are stretched by repeating their single entry. Formally, for each axis

Returns self unchanged when the source dtype is already float32. Otherwise performs a copy-cast via the engine's astype op; the device is preserved.

Indicator-style encoding: $\text{idx}(t) = 0 \cdot \mathbf{1}\{\text{is\_metal}\} + (-1) \cdot \mathbf{1}\{\text{is\_cpu}\}$. Lucid only supports a single Metal device on Apple Silicon, so a positive index is always 0.

Returns self unchanged when the source dtype is already float16. Lossy narrowing from float32/float64 may overflow to inf for magnitudes above ~65504 and underflow to 0 for magnitudes below ~6e-5.

Returns self unchanged when the source dtype is already int32. Casts from floating point truncate toward zero; values outside [-2**31, 2**31 - 1] produce undefined results.

A tensor of shape $(s_0, \ldots, s_{d-1})$ is contiguous iff its strides satisfy the C-order recurrence

(in element units). Many Accelerate/MLX kernels require contiguous inputs; if not, call contiguous first.

Identically false: $\text{is\_pinned}(t) \equiv \text{False}$. Unified memory makes the concept moot — all host buffers are already DMA-accessible to the GPU without page-locking.

Triggers a device-to-host synchronisation when called on a Metal tensor — the value cannot be inspected until any pending MLX computation has completed. Avoid calling in tight loops on GPU tensors; prefer tensor.cpu().tolist() for batch extraction.

Defined only when $\text{numel} = \prod_i s_i = 1$. item is one of the sanctioned engine-to-host bridge points (rule H4) and detaches from autograd: the returned Python scalar carries no gradient information.

Returns self unchanged when the source dtype is already int64. Casts from floating point truncate toward zero. Typically used for index tensors (e.g. gather / scatter).

No-op when the tensor is already on Metal. Under Apple Silicon's unified memory architecture the physical DRAM is shared with the CPU; this call only updates the dispatch target so subsequent ops run through the MLX/Metal backend.

Allocates $\prod_i s_i \cdot e$ bytes of uninitialised memory where $e$ is dtype.itemsize. Faster than new_zeros because the engine skips the zero-fill kernel. Must be followed by a write before any read.

Constant tensor $f \cdot \mathbf{1}$ where $f$ is fill_value and $\mathbf{1}$ is the all-ones tensor of the given shape. fill_value is promoted to dtype before the broadcast fill.

Initialises every element to the multiplicative identity $1$. For arbitrary fill values use new_full.

Always copies — never aliases data's storage even when data is already a Tensor. Routes through the same _to_impl bridge boundary as the public constructor (rule H4).

Initialises every element to the additive identity $0$. The zero-fill is delegated to a fused engine kernel — Accelerate catlas_*set on CPU and MLX broadcast-fill on Metal.

This is one of the sanctioned bridge points between Lucid and the outside world (see project rule H4). The returned ndarray does not participate in autograd; downstream NumPy operations will not produce gradients.

Layout is C-contiguous; for a tensor of shape $(s_0, \ldots, s_{d-1})$ and itemsize $e$, the NumPy strides equal

See also is_pinned (always False) and share_memory_ (also a no-op). The function is the identity: $\text{pin\_memory}(t) \equiv t$.

• On a non-leaf the hook runs once its producer's output slot has received every consumer's gradient — consumers created before or after the hook alike — and before the producer's backward runs. No retain_grad is needed. lucid.autograd.grad runs it too when the tensor lies on a path to the requested inputs.
• On a leaf the hook runs just before the pass's gradient is added into .grad, so it sees that pass's gradient alone, not the accumulated one, and an unrelated backward never runs it.
• Several hooks run in registration order, each on what the one before it returned. retain_grad keeps the gradient left after all of them.
• The hook runs with grad mode off, or on under backward(create_graph=True), where its result stays part of the graph.
• After an in-place op on self, a hook registered earlier still sees the gradient of the value before the write.
• The hook is kept by the graph node that produced self, and is released with that node's saved state after a backward without retain_graph. A hook on the output of an op that saves nothing for backward lives as long as the graph does.

Chain-rule effect: if the hook returns a tensor $\tilde g$, the engine substitutes $\frac{\partial \mathcal{L}}{\partial \mathbf{x}} \leftarrow \tilde g$ before continuing backward propagation.

Reshape preserves element order under row-major (C) traversal; the element at flat index $k = \sum_i i_k \cdot \prod_{j>k} s_j$ in self becomes the element at the same flat index in the result. A view is returned when the source is contiguous; otherwise a contiguous copy is materialised first.

Call it before the backward whose gradient you want to inspect; it may come after the ops that consume self. Calling it after backward has no effect on that pass. Conceptually retains $\frac{\partial \mathcal{L}}{\partial \mathbf{y}}$ for the intermediate node $\mathbf{y}$ instead of discarding it after its parents have consumed it. The retained gradient is the one left after the hooks registered with register_hook have run, whichever was registered first, and it accumulates across backward passes as a leaf's does. A leaf keeps its gradient anyway.

The full shape and the element count are related by

For a C-contiguous tensor of shape $(s_0, \ldots, s_{d-1})$ the element strides satisfy the row-major recurrence

Non-contiguous tensors (e.g. transposed or sliced views) may have arbitrary strides; the address of element [i_0, \ldots, i_{d-1}] is base + \sum_k i_k \cdot \text{stride}[k].

No-op when the source already matches the target device and dtype, unless copy=True; in that case the same Tensor instance is returned without allocation. Apple Silicon's unified memory means device transfers between CPU and Metal copy the dispatch label but share physical DRAM when the underlying buffer is a SharedStorage.

Numpy-free for every supported dtype (F16 / F32 / F64 / I8 / I16 / I32 / I64 / Bool / C64). Delegates to the engine-side TensorImpl.tolist(), which mirrors item()'s dtype dispatch but walks the full shape recursively and yields Python complex for C64 leaves. Forces a device-to-host synchronisation for Metal tensors via the underlying to_bytes() snapshot. Autograd information is dropped — the returned Python objects are pure value copies.

Values are cast element-wise: $x_i \leftarrow \text{cast}_{\tau}(x_i)$ where $\tau = \text{other.dtype}$. Casts between floating-point types preserve gradients; integer→float→integer round-trips lose information in the integer truncation step.

Unfold is the fundamental primitive behind 1-D convolution and sliding-window aggregations. On the CPU the result is a view of self's buffer, so a write through a window reaches self; the windows overlap when step < size, and a write through overlapping windows is refused, since an element then repeats. On metal the result is a copy.