class

ASGD

extendsOptimizer

ASGD(params: Iterable[Parameter] | Iterable[dict[str, object]], lr: float = 0.01, lambd: float = 0.0001, alpha: float = 0.75, t0: float = 1000000.0, weight_decay: float = 0)

source edit

Averaged Stochastic Gradient Descent optimizer.

ASGD performs standard SGD updates but maintains a running average of the iterate sequence, which serves as the final parameter estimate. The averaging improves convergence in the presence of noise and is particularly effective near the end of training.

Each step decays the parameter and takes an SGD step with the current step size $\eta_t$ ( $g_t$ includes any weight_decay term):

\theta_t = \theta_{t-1} \bigl(1 - \lambda \, \eta_t\bigr) - \eta_t \, g_t

then folds the result into the running average ax with weight $\mu_t$ (ax simply copies $\theta_t$ while $\mu_t = 1$ ):

\bar{\theta}_t = \bar{\theta}_{t-1} + \mu_t \bigl(\theta_t - \bar{\theta}_{t-1}\bigr)

and sets the step size and weight for the next step, starting from $\eta_1 = \eta_0$ and $\mu_1 = 1$ :

\eta_{t+1} = \frac{\eta_0}{(1 + \lambda \, \eta_0 \, t)^\alpha}, \qquad \mu_{t+1} = \frac{1}{\max(1, \; t - t_0)}

Until $t_0$ the average therefore tracks the parameter; after it, the average is the mean of the iterates since $t_0$ . The state of each parameter is step, eta, mu and ax.

Parameters

paramsiterable of Parameter or iterable of dict

Parameters to optimise, or a list of parameter-group dicts.

lrfloat= 0.01

Initial learning rate

\eta_0

(default: 1e-2).

lambdfloat= 0.0001

Decay term

\lambda

(default: 1e-4).

alphafloat= 0.75

Power for LR decay

\alpha

(default: 0.75).

t0float= 1000000.0

Step at which averaging begins (default: 1e6).

weight_decayfloat= 0

L2 regularisation coefficient (default: 0).

Attributes

param_groupslist of dict

Parameter groups with keys "params", "lr", "lambd", "alpha", "t0", and "weight_decay".

defaultsdict

Default hyperparameter values.

Notes

ASGD can match or exceed the convergence rate of SGD with careful learning-rate tuning, and the averaging step provides additional regularisation. The default t0=1e6 delays averaging until very late in training; reduce it to start averaging earlier.

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> import lucid.optim as optim
>>> model = nn.Linear(4, 2)
>>> loss = model(lucid.randn(1, 4)).sum()
>>> import lucid.optim as optim
>>> optimizer = optim.ASGD(model.parameters(), lr=1e-2, t0=1e5)
>>> optimizer.zero_grad()
>>> loss.backward()
>>> optimizer.step()

Used by 2

Constructors

dunder

init

→None

__init__(params: Iterable[Parameter] | Iterable[dict[str, object]], lr: float = 0.01, lambd: float = 0.0001, alpha: float = 0.75, t0: float = 1000000.0, weight_decay: float = 0)

source edit

Initialise the ASGD. See the class docstring for parameter semantics.

Instance methods

step

→Tensor | None

step(closure: _OptimizerClosure = None)

source edit

Perform a single ASGD step.

>>> import lucid >>> import lucid.nn as nn >>> import lucid.optim as optim >>> model = nn.Linear(4, 2) >>> loss = model(lucid.randn(1, 4)).sum() >>> import lucid.optim as optim >>> optimizer = optim.ASGD(model.parameters(), lr=1e-2, t0=1e5) >>> optimizer.zero_grad() >>> loss.backward() >>> optimizer.step()