optim
30 memberslucid.optimlucid.optim — parameter update rules and learning-rate schedules.
Thirteen optimizers over the shared Optimizer base — SGD, Adam,
AdamW, LBFGS, RMSprop, Adagrad, Adadelta, Adamax,
RAdam, NAdam, ASGD, Rprop, SparseAdam — plus sixteen
schedulers in lucid.optim.lr_scheduler: fourteen schedules and the
SequentialLR / ChainedScheduler composites that wrap them.
Updates run under lucid.no_grad() and write into the parameter in place.
That is load-bearing rather than incidental: assigning the result of a
differentiable op to a parameter makes it a non-leaf, after which backward
flows through it and accumulates nothing, and training silently becomes a
no-op that no test or loss curve reports as an error.
Checkpointing: Optimizer and every scheduler expose state_dict /
load_state_dict, so a resumed run continues the same schedule instead of
restarting it. Schedulers deliberately do not serialise the callables
LambdaLR and MultiplicativeLR hold — reconstruct those with the same
function, then load the rest of the state onto them.
Classes
Optimizer6 methodsBase class for all optimizers.
SGD2 methodsStochastic Gradient Descent optimizer with optional momentum and weight decay.
Adam2 methodsAdaptive Moment Estimation optimizer (Kingma & Ba, 2015).
AdamW2 methodsAdam optimizer with decoupled weight decay regularisation.
LBFGS4 methodsLimited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) optimizer.
RMSprop2 methodsRoot Mean Square Propagation optimizer.
Adagrad2 methodsAdaptive Gradient optimizer.
Adadelta2 methodsAdadelta optimizer — an adaptive learning rate method with no global LR.
Adamax2 methodsAdamax optimizer — a variant of Adam based on the infinity norm.
RAdam2 methodsRectified Adam optimizer with variance-adapted step size.
NAdam2 methodsNesterov-accelerated Adaptive Moment Estimation optimizer.
ASGD2 methodsAveraged Stochastic Gradient Descent optimizer.
Rprop2 methodsResilient Backpropagation optimizer.
SparseAdam2 methodsLazy Adam for gradients that touch a few rows of a parameter.
StepLR2 methodsDecay the learning rate by a fixed multiplicative factor every fixed number of epochs.
ExponentialLR2 methodsDecay the learning rate by a constant factor every epoch.
MultiStepLR2 methodsDecay the learning rate by a fixed factor at a list of epoch milestones.
CosineAnnealingLR2 methodsAnneal the learning rate following a cosine curve over T_max epochs.
LambdaLR2 methodsSet the learning rate using a user-supplied multiplicative factor function.
CyclicLR2 methodsCycle the learning rate between base_lr and max_lr.
ReduceLROnPlateau4 methodsReduce the learning rate when a monitored metric stops improving.
NoamScheduler2 methodsNoam learning rate schedule from the original Transformer paper.
MultiplicativeLR2 methodsMultiply the learning rate by a factor returned by a function each epoch.
LinearLR2 methodsLinearly interpolate the learning rate from a start factor to an end factor.
ConstantLR2 methodsHold the learning rate at a constant scaled value, then restore the base LR.
PolynomialLR2 methodsDecay the learning rate using a polynomial function over a fixed number of steps.
CosineAnnealingWarmRestarts3 methodsCosine annealing with periodic warm restarts (SGDR).
OneCycleLR2 methods1Cycle learning rate policy.
SequentialLR5 methodsChain multiple schedulers and switch between them at epoch milestones.
ChainedScheduler5 methodsApply multiple schedulers simultaneously, compounding their LR changes.