get_total_norm

→Tensor

get_total_norm(tensors: Iterable[Tensor] | Tensor, norm_type: float = 2.0, error_if_nonfinite: bool = False, foreach: bool | None = None)

source edit

Compute the $\ell_p$ norm of tensors, taken jointly.

The measurement step of clip_grad_norm_, which hands it the parameters' gradients. It measures the tensors it is given, as the reference framework's does — it used to read .grad from them, so the usual call, get_total_norm([p.grad for p in model.parameters()]), read the gradients' own (absent) gradients and answered 0.

Parameters

tensorsiterable of Tensor, or Tensor

The tensors whose elements make up the norm — typically the gradients. None entries are skipped; the empty case returns a zero tensor.

norm_typefloat= 2.0

Order

p

of the norm. Default 2.0; use math.inf for the max-norm.

error_if_nonfinitebool= False

Raise on inf / nan results instead of returning them.

foreachbool= None

Accepted for API compatibility; ignored — Lucid processes the tensors sequentially through the C++ engine.

Returns

Tensor

Scalar tensor with the combined norm.

Notes

The returned scalar is

\|g\|_p \;=\; \left(\sum_i |g_i|^p\right)^{1/p}

taken over every element of every tensor — the quantity clip_grad_norm_ thresholds against max_norm.

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> from lucid.nn.utils.clip_grad import get_total_norm
>>> get_total_norm([lucid.tensor([3.0, 4.0])]).item()
5.0
>>> model = nn.Linear(4, 2)
>>> model(lucid.randn(8, 4)).sum().backward()
>>> g_norm = get_total_norm([p.grad for p in model.parameters()])
>>> g_norm.shape
()
>>> float(g_norm) > 0.0
True

Used by 1

lucid.nn.utils

>>> import lucid >>> import lucid.nn as nn >>> from lucid.nn.utils.clip_grad import get_total_norm >>> get_total_norm([lucid.tensor([3.0, 4.0])]).item() 5.0 >>> model = nn.Linear(4, 2) >>> model(lucid.randn(8, 4)).sum().backward() >>> g_norm = get_total_norm([p.grad for p in model.parameters()]) >>> g_norm.shape () >>> float(g_norm) > 0.0 True