local_response_norm(x: Tensor, size: int, alpha: float = 0.0001, beta: float = 0.75, k: float = 1.0)Local response normalization (Krizhevsky, Sutskever & Hinton, 2012).
Implements lateral inhibition: each activation is divided by a function of the squared activations of its channel neighbours. Introduced with AlexNet and largely superseded by batch / layer normalisation, but still used in some classic architectures.
Parameters
(N, C, *spatial) (any number of spatial
dims). Inputs with fewer than 2 dimensions are returned
unchanged.sizeintalphafloat= 0.00011e-4.betafloat= 0.750.75.kfloat= 1.01.0.Returns
TensorSame shape as x.
Notes
The scale is the mean of the squared activations over a window of channels, so is divided by :
Channels outside [0, C) count as zero and stay in the
denominator . The window reaches size // 2 channels
below and (size - 1) // 2 above it, so for an even
size it leans towards the lower channels. This is the
reference framework's definition (an average pool of
along the channel axis); the AlexNet paper writes the same rule
with the folded into . LRN predates
BatchNorm and provides only lateral inhibition (no learnt affine),
which is why modern architectures usually do not include it.
Examples
>>> import lucid
>>> from lucid.nn.functional import local_response_norm
>>> x = lucid.randn(1, 96, 27, 27) # AlexNet conv1 output shape
>>> y = local_response_norm(x, size=5, alpha=1e-4, beta=0.75, k=2.0)
>>> y.shape
(1, 96, 27, 27)