class

Adagrad

extendsOptimizer

Adagrad(params: Iterable[Parameter] | Iterable[dict[str, object]], lr: float = 0.01, lr_decay: float = 0, weight_decay: float = 0, initial_accumulator_value: float = 0, eps: float = 1e-10)

source edit

Adaptive Gradient optimizer.

Adagrad adapts the learning rate for each parameter based on the sum of all past squared gradients. Parameters that receive large or frequent gradient updates get smaller effective learning rates:

\begin{aligned} G_t &= G_{t-1} + g_t^2 \\ \theta_t &= \theta_{t-1} - \frac{\eta_t}{\sqrt{G_t} + \epsilon} \, g_t \end{aligned}

where the effective learning rate decays with time as:

\eta_t = \frac{\eta_0}{1 + (t - 1) \cdot \lambda}

and $\lambda$ is the lr_decay parameter.

Parameters

paramsiterable of Parameter or iterable of dict

Parameters to optimise, or a list of parameter-group dicts.

lrfloat= 0.01

Initial learning rate

\eta_0

(default: 1e-2).

lr_decayfloat= 0

Learning-rate decay applied to the effective LR at each step

\lambda

(default: 0).

weight_decayfloat= 0

L2 regularisation coefficient (default: 0).

initial_accumulator_valuefloat= 0

Value every entry of

G

starts from (default: 0).

epsfloat= 1e-10

Term

\epsilon

added to the denominator for numerical stability (default: 1e-10).

Attributes

param_groupslist of dict

Parameter groups with keys "params", "lr", "lr_decay", "weight_decay", "initial_accumulator_value", and "eps".

defaultsdict

Default hyperparameter values.

Notes

Adagrad is particularly well-suited for sparse data (e.g. NLP with word embeddings) because infrequently updated parameters retain a larger effective learning rate. The main drawback is that the accumulated squared-gradient sum $G_t$ only grows, so the effective learning rate can become vanishingly small over long training runs. initial_accumulator_value precedes eps in the constructor signature, matching the reference framework.

Examples

The sparse-feature case it suits — an embedding whose rows are hit
unevenly:
>>> import lucid
>>> import lucid.nn as nn
>>> import lucid.optim as optim
>>> table = nn.Embedding(10, 3)
>>> optimizer = optim.Adagrad(table.parameters(), lr=1e-2)
>>> optimizer.zero_grad()
>>> (table(lucid.tensor([1, 4, 4, 7])) ** 2).sum().backward()
>>> optimizer.step()

Used by 2

Constructors

dunder

init

→None

__init__(params: Iterable[Parameter] | Iterable[dict[str, object]], lr: float = 0.01, lr_decay: float = 0, weight_decay: float = 0, initial_accumulator_value: float = 0, eps: float = 1e-10)

source edit

Initialise the Adagrad. See the class docstring for parameter semantics.

Instance methods

step

→Tensor | None

step(closure: _OptimizerClosure = None)

source edit

Perform a single Adagrad step.

The sparse-feature case it suits — an embedding whose rows are hit unevenly: >>> import lucid >>> import lucid.nn as nn >>> import lucid.optim as optim >>> table = nn.Embedding(10, 3) >>> optimizer = optim.Adagrad(table.parameters(), lr=1e-2) >>> optimizer.zero_grad() >>> (table(lucid.tensor([1, 4, 4, 7])) ** 2).sum().backward() >>> optimizer.step()