class

LocalResponseNorm

extendsModule

LocalResponseNorm(size: int, alpha: float = 0.0001, beta: float = 0.75, k: float = 1.0)

source edit

Local Response Normalization (LRN) across adjacent channels.

Normalises each activation by the mean of the squared activations in a window of $n = \text{size}$ channels around it, at the same spatial position:

y_c = \frac{x_c}{\left(k + \frac{\alpha}{n} \sum_{j=c-\lfloor n/2\rfloor}^{c+\lfloor (n-1)/2\rfloor} x_j^2\right)^{\beta}}

Channels outside [0, C) count as zero and still count in the denominator $n$ ; for an even size the window leans towards the lower channels. The module is a thin wrapper over lucid.nn.functional.local_response_norm, which owns the rule.

LRN was introduced in the AlexNet paper as a biologically-inspired lateral inhibition mechanism: strongly-activated neurons suppress their neighbors, encouraging competition across feature detectors at the same spatial location. It has largely been superseded by Batch Normalization in modern architectures, but remains available for reproducing historical results.

Parameters

sizeint

Number of adjacent channels included in the normalization window (the

\text{size}

term in the formula above). Must be a positive integer.

alphafloat= 0.0001

Scaling factor

\alpha

for the squared sum. Default: 1e-4.

betafloat= 0.75

Exponent

\beta

applied to the normalization term. Default: 0.75.

kfloat= 1.0

Bias constant

k

that prevents division by zero. Default: 1.0.

Notes

Input: $(N, C, *)$ where $*$ denotes zero or more spatial dimensions. The channel axis must be axis 1.
Output: same shape as the input.

Inputs with fewer than 2 dimensions are returned unchanged.
The original AlexNet used size=5, alpha=1e-4, beta=0.75, k=2.0 — adjusting k is important to control how strongly unsuppressed channels are penalized.

Examples

Reproduce the AlexNet LRN configuration:
>>> import lucid
>>> import lucid.nn as nn
>>> lrn = nn.LocalResponseNorm(size=5, alpha=1e-4, beta=0.75, k=2.0)
>>> x = lucid.randn(2, 96, 27, 27)   # e.g. after first conv in AlexNet
>>> out = lrn(x)
>>> out.shape
(2, 96, 27, 27)
Minimal example with a 1-D channel tensor:
>>> lrn_small = nn.LocalResponseNorm(size=3)
>>> x2 = lucid.randn(4, 32, 64)   # (N, C, L)
>>> out2 = lrn_small(x2)
>>> out2.shape
(4, 32, 64)

Used by 1

lucid.nn.modules

Constructors

dunder

init

→None

__init__(size: int, alpha: float = 0.0001, beta: float = 0.75, k: float = 1.0)

source edit

Initialise the LocalResponseNorm module. See the class docstring for parameter semantics.

Instance methods

extra_repr

→str

extra_repr()

source edit

Return a string representation of the layer's configuration.

forward

→Tensor

forward(x: Tensor)

source edit

Apply normalisation to the input tensor.

Parameters

xTensor

Input tensor whose shape is documented in the class docstring.

Returns

Tensor

Normalised tensor of the same shape as x.

Reproduce the AlexNet LRN configuration: >>> import lucid >>> import lucid.nn as nn >>> lrn = nn.LocalResponseNorm(size=5, alpha=1e-4, beta=0.75, k=2.0) >>> x = lucid.randn(2, 96, 27, 27) # e.g. after first conv in AlexNet >>> out = lrn(x) >>> out.shape (2, 96, 27, 27) Minimal example with a 1-D channel tensor: >>> lrn_small = nn.LocalResponseNorm(size=3) >>> x2 = lucid.randn(4, 32, 64) # (N, C, L) >>> out2 = lrn_small(x2) >>> out2.shape (4, 32, 64)