Optimizer
Optimizer(params: Iterable[Parameter] | Iterable[dict[str, object]], defaults: dict[str, object])Base class for all optimizers.
Every concrete optimizer must subclass Optimizer and implement
step. The base class manages parameter groups, provides
zero_grad, and handles state_dict / load_state_dict
checkpointing.
Parameters
paramsiterable of Parameter or iterable of dictlucid.nn.Parameter objects, or
a list of parameter-group dicts. Each group dict must contain at
least a "params" key; any other keys override the corresponding
entry in defaults for that group.defaultsdict{"lr": 1e-3, "weight_decay": 0}).
These are copied into each param group that does not override them.Attributes
param_groupslist of dict"params": list[Parameter] plus all hyperparameter keys from
defaults (possibly overridden per group).defaultsdictstatedictstate_dict and consumed by
load_state_dict; an optimizer that keeps its state in
Python (lucid.optim.SparseAdam) holds its live state here.Notes
Every concrete step() override is automatically wrapped at class
definition time (via __init_subclass__) to flush all parameter
tensors to the Metal GPU after each update. This ensures that
parameter values are materialised before the next forward pass.
Each step reads the hyper-parameters from param_groups, so a value
assigned there takes effect at the next step, the way a
hand-written warm-up or fine-tuning schedule expects:
for group in optimizer.param_groups:
group["lr"] = 1e-4
Each group is checked against the optimizer's rules when it is added and
again when an edited group is next used. A value the constructor would
reject (a negative lr, a beta outside [0, 1)) raises
InvalidArgument, a ValueError, and nothing of that group is
applied.
Parameter groups allow different hyperparameters per group:
optimizer = optim.SGD(
[
{"params": model.backbone.parameters(), "lr": 1e-3},
{"params": model.head.parameters(), "lr": 1e-2},
],
lr=1e-4, # default, overridden per group above
)
Examples
Optimizer is not used directly. Use a concrete subclass:
>>> import lucid
>>> import lucid.nn as nn
>>> model = nn.Linear(4, 2)
>>> loss = model(lucid.randn(1, 4)).sum()
>>> import lucid.optim as optim
>>> optimizer = optim.Adam(model.parameters(), lr=1e-3)
>>> optimizer.zero_grad()
>>> loss.backward()
>>> optimizer.step()Used by 9
Constructors
1Instance methods
5Add a parameter group, creating one new engine optimizer for it.
The group's hyper-parameters are checked first, against the rules
the optimizer's constructor applies; a group that breaks one raises
InvalidArgument and is not added.
Restore from a state_dict produced by state_dict.
Hyperparameters in param_groups are restored, and each
parameter's state (returned from _save_state) is restored
via _load_state — for the engine optimizers, every buffer
and per-parameter step goes back into the engine.
Return a checkpointable snapshot of the optimizer.
Mirrors reference framework's optimizer state_dict layout:
state:{param_index: {key: value, ...}}— each parameter's optimizer state under the reference framework's keys: the engine's buffers and per-parameter scalars (exp_avg,momentum_buffer,step...) as numpy arrays, or the Python-side state of an optimizer that keeps its own (LBFGS).param_groups: list of group dicts; each group'sparamsis a list of integer indices into the flat parameter list.
Perform a single optimization step.
Subclasses must override this to update parameters from their current gradients. Optionally accepts a closure that re-evaluates the model and returns the loss (required by some optimizers, e.g. LBFGS; ignored by most first-order methods).
Zero gradients of all parameters.