class

LBFGS

extendsOptimizer

LBFGS(params: Iterable[Parameter] | Iterable[dict[str, object]], lr: float = 1.0, max_iter: int = 20, max_eval: int | None = None, tolerance_grad: float = 1e-07, tolerance_change: float = 1e-09, history_size: int = 100, line_search_fn: str | None = None)

source edit

Limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) optimizer.

L-BFGS is a quasi-Newton method that approximates the inverse Hessian using a limited history of gradient and parameter difference vectors. One call to step runs up to max_iter iterations. Each iteration computes a search direction $d_k$ via the two-loop recursion:

d_k = -H_k \nabla L(\theta_k)

where $H_k$ is the L-BFGS inverse-Hessian approximation built from the last history_size curvature pairs $\{(s_i, y_i)\}$ :

\begin{aligned} s_i &= \theta_{i+1} - \theta_i \\ y_i &= \nabla L(\theta_{i+1}) - \nabla L(\theta_i) \end{aligned}

with the initial diagonal scaling

H_{\text{diag}} = \frac{s_{k-1}^\top y_{k-1}}{y_{k-1}^\top y_{k-1}}.

A pair is stored only when $y^\top s > 10^{-10}$ . The parameters then move to $\theta_{k+1} = \theta_k + t_k d_k$ . With line_search_fn=None the step is fixed, $t_k = \text{lr}$ (on the very first iteration $\text{lr} \cdot \min(1, 1/\lVert\nabla L\rVert_1)$ ). With "strong_wolfe", $t_k$ is found by cubic interpolation so that

\begin{aligned} L(\theta_k + t d_k) &\le L(\theta_k) + c_1 t \, \nabla L(\theta_k)^\top d_k \\ \lvert \nabla L(\theta_k + t d_k)^\top d_k \rvert &\le c_2 \lvert \nabla L(\theta_k)^\top d_k \rvert \end{aligned}

with $c_1 = 10^{-4}$ and $c_2 = 0.9$ .

Parameters

paramsiterable of Parameter or iterable of dict

Parameters to optimise. Only one parameter group is supported.

lrfloat= 1.0

Step size (default: 1.0). The fixed step without a line search, and the first trial step of the strong-Wolfe search.

max_iterint= 20

Maximum number of L-BFGS iterations per step call (default: 20).

max_evalint or None= None

Maximum number of closure evaluations per step call (default: None, meaning max_iter * 5 // 4).

tolerance_gradfloat= 1e-07

step stops once

\lVert\nabla L\rVert_\infty \le

tolerance_grad (default: 1e-7).

tolerance_changefloat= 1e-09

step stops once the directional derivative, the parameter change

\lVert t_k d_k\rVert_\infty

or the change of the loss falls below this (default: 1e-9).

history_sizeint= 100

Number of

(s, y)

curvature pairs retained in memory (default: 100).

line_search_fnstr or None= None

"strong_wolfe" or None for a fixed step (default: None).

Attributes

param_groupslist of dict

The single parameter group.

defaultsdict

Default hyperparameter values.

Raises

ValueError

If params holds more than one parameter group, or an option is out of range (lr < 0, history_size < 1, an unknown line_search_fn).

Notes

Unlike first-order optimizers, L-BFGS requires a closure argument in step that clears gradients, computes the loss, and calls loss.backward(). Without a closure the method raises ValueError.

The state (the last direction d and step t, the curvature pairs old_dirs / old_stps with their ro = 1 / yᵀs, H_diag, prev_flat_grad, prev_loss, n_iter and func_evals) carries over from one step to the next and round-trips through state_dict.

L-BFGS is best suited for full-batch or large-batch training where the curvature information is reliable. It is not recommended for stochastic mini-batch training because noisy gradients corrupt the Hessian approximation.

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> import lucid.optim as optim
>>> model = nn.Linear(4, 2)
>>> x = lucid.randn(1, 4)
>>> y = lucid.zeros((1, 2))
>>> criterion = nn.MSELoss()
>>> optimizer = optim.LBFGS(model.parameters(), lr=1.0, max_iter=20)
>>> def closure():
...     optimizer.zero_grad()
...     loss = criterion(model(x), y)
...     loss.backward()
...     return loss
>>> loss = optimizer.step(closure)   # step returns the closure's first loss

Used by 2

Constructors

dunder

init

→None

__init__(params: Iterable[Parameter] | Iterable[dict[str, object]], lr: float = 1.0, max_iter: int = 20, max_eval: int | None = None, tolerance_grad: float = 1e-07, tolerance_change: float = 1e-09, history_size: int = 100, line_search_fn: str | None = None)

source edit

Initialise the LBFGS. See the class docstring for parameter semantics.

Instance methods

add_param_group

→None

add_param_group(group: dict[str, object])

source edit

Refuse a second group: L-BFGS optimises one flat vector of parameters.

Raises

ValueError

When the optimizer already has its parameter group.

step

→Tensor

step(closure: _OptimizerClosure = None)

source edit

Run up to max_iter L-BFGS iterations.

Each iteration takes the two-loop-recursion direction, moves along it by the fixed step lr or by a strong-Wolfe line search, and stores the new curvature pair. The loop ends after max_iter iterations, after max_eval closure evaluations, or when a tolerance_grad / tolerance_change rule is met.

Parameters

closurecallable= None

A zero-argument callable that:

Calls optimizer.zero_grad() to clear stale gradients.
Runs the forward pass and computes the scalar loss.
Calls loss.backward() to populate gradients.
Returns the loss tensor.

This argument is required — passing None raises ValueError.

Returns

Tensor

The closure's loss at the parameters step started from (its first evaluation), as the reference framework returns it.

Raises

ValueError

If closure is None.

Notes

The closure is called up to max_eval times per step call, always with gradients enabled. Ensure that any side effects (e.g. batch norm running stats) are handled appropriately if this matters for your use-case.

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> import lucid.optim as optim
>>> model = nn.Linear(4, 2)
>>> x = lucid.randn(1, 4)
>>> y = lucid.zeros((1, 2))
>>> criterion = nn.MSELoss()
>>> optimizer = optim.LBFGS(model.parameters(), lr=1.0, max_iter=20)
>>> def closure():
...     optimizer.zero_grad()
...     output = model(x)
...     loss = criterion(output, y)
...     loss.backward()
...     return loss
>>> loss = optimizer.step(closure)   # step returns the closure's first loss

zero_grad

→None

zero_grad(set_to_none: bool = True)

source edit

Set gradients of all parameters to None.

L-BFGS always sets gradients to None regardless of the set_to_none argument, because the closure passed to step is responsible for zeroing and recomputing gradients on each function evaluation.

Parameters

set_to_nonebool= True

Ignored; kept for API compatibility with Optimizer (default: True).

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> import lucid.optim as optim
>>> model = nn.Linear(4, 2)
>>> x = lucid.randn(1, 4)
>>> y = lucid.zeros((1, 2))
>>> criterion = nn.MSELoss()
>>> optimizer = optim.LBFGS(model.parameters(), lr=1.0, max_iter=20)
>>> def closure():
...     optimizer.zero_grad()
...     loss = criterion(model(x), y)
...     loss.backward()
...     return loss
>>> loss = optimizer.step(closure)   # step returns the closure's first loss

LBFGS(params: Iterable[Parameter] | Iterable[dict[str, object]], lr: float = 1.0, max_iter: int = 20, max_eval: int | None = None, tolerance_grad: float = 1e-07, tolerance_change: float = 1e-09, history_size: int = 100, line_search_fn: str | None = None)

>>> import lucid >>> import lucid.nn as nn >>> import lucid.optim as optim >>> model = nn.Linear(4, 2) >>> x = lucid.randn(1, 4) >>> y = lucid.zeros((1, 2)) >>> criterion = nn.MSELoss() >>> optimizer = optim.LBFGS(model.parameters(), lr=1.0, max_iter=20) >>> def closure(): ... optimizer.zero_grad() ... loss = criterion(model(x), y) ... loss.backward() ... return loss >>> loss = optimizer.step(closure) # step returns the closure's first loss

__init__(params: Iterable[Parameter] | Iterable[dict[str, object]], lr: float = 1.0, max_iter: int = 20, max_eval: int | None = None, tolerance_grad: float = 1e-07, tolerance_change: float = 1e-09, history_size: int = 100, line_search_fn: str | None = None)

>>> import lucid >>> import lucid.nn as nn >>> import lucid.optim as optim >>> model = nn.Linear(4, 2) >>> x = lucid.randn(1, 4) >>> y = lucid.zeros((1, 2)) >>> criterion = nn.MSELoss() >>> optimizer = optim.LBFGS(model.parameters(), lr=1.0, max_iter=20) >>> def closure(): ... optimizer.zero_grad() ... output = model(x) ... loss = criterion(output, y) ... loss.backward() ... return loss >>> loss = optimizer.step(closure) # step returns the closure's first loss