vhp

→tuple of (Tensor, Tensor or tuple of Tensor)

vhp(func: Callable[..., Tensor], inputs: Tensor | tuple[Tensor, ...], v: Tensor | tuple[Tensor, ...] | None = None, create_graph: bool = False, strict: bool = False)

source edit

Vector-Hessian product $v^\top H$ of a scalar-valued func.

For $f : \mathbb{R}^n \to \mathbb{R}$ with Hessian $H_{ij} = \partial^2 f / \partial x_i \partial x_j$ and a vector $v \in \mathbb{R}^n$ , returns

v^\top H = \nabla_x \big( v^\top \nabla_x f(x) \big)

along with $f(x)$ , without forming $H$ : one backward pass that records its own graph, and one backward pass through it.

Parameters

funccallable

Function of one or more Tensor inputs returning a tensor of one element.

inputsTensor or tuple of Tensor

Point

x

. Left as it is: the product is taken at copies that require grad.

vTensor or tuple of Tensor= None

One vector per input, each of its input's shape and device. May be left out when inputs is one single-element tensor, which is then seeded with 1.

create_graphbool= False

If True the output and the product are differentiable with respect to the inputs that require grad (and v). Defaults to False.

strictbool= False

If True, refuse an output that does not require grad, a gradient that does not depend on the inputs (a linear func, say) and a product that would be zeros for want of a path — and, under create_graph, a product that is a constant. Defaults to False.

Returns

tuple of (Tensor, Tensor or tuple of Tensor)

(output, vhp) where output = func(*inputs) — detached unless create_graph — and vhp is shaped and packed like inputs: one tensor for one tensor, a tuple for a tuple.

Raises

RuntimeError

func does not return a single tensor of one element, v is left out and inputs is not one single-element tensor, or strict refuses a result.

(ShapeMismatch, DeviceMismatch)

An entry of v does not have its input's shape or device.

Notes

For a twice continuously differentiable func the Hessian is symmetric and $v^\top H = (H v)^\top$ , so this and hvp agree up to rounding; this one costs a pass less.

Examples

>>> import lucid
>>> from lucid.autograd import vhp
>>> x = lucid.tensor([1.0, 2.0])
>>> def f(x):
...     return (x ** 3).sum()
>>> out, product = vhp(f, x, lucid.tensor([1.0, 1.0]))
>>> product
tensor([6., 12.])

Used by 1

lucid.autograd

>>> import lucid >>> from lucid.autograd import vhp >>> x = lucid.tensor([1.0, 2.0]) >>> def f(x): ... return (x ** 3).sum() >>> out, product = vhp(f, x, lucid.tensor([1.0, 1.0])) >>> product tensor([6., 12.])