sparse

→Tensor

sparse_(tensor: Tensor, sparsity: float, std: float = 0.01, generator: _C_engine.Generator | None = None)

source edit

Initialise a 2-D tensor in-place with a sparse random matrix.

For each column, a random subset of ceil(sparsity * rows) rows is set to zero; the remaining entries are drawn from $\mathcal{N}(0, \text{std}^2)$ . Sparse initialisation, due to Martens (2010), encourages each output unit to depend on only a small number of inputs at the start of training and helps in ill-conditioned regimes — particularly for very wide layers where a dense Gaussian would have an unfavourable condition number.

Parameters

tensorTensor

Two-dimensional tensor to fill in place.

sparsityfloat

Fraction of entries in each column that are zeroed. Must lie in [0, 1].

stdfloat= 0.01

Standard deviation of the Gaussian used for the non-zero entries. Default 0.01.

generatorGenerator= None

PRNG used for both the Gaussian fill and the column zero-mask. Default None.

Returns

Tensor

tensor populated with a column-sparse Gaussian matrix.

Raises

ValueError

If tensor.ndim != 2 or sparsity is outside [0, 1].

Notes

The expected per-entry variance is

\mathrm{Var}(W_{ij}) = (1 - \text{sparsity}) \cdot \text{std}^2.

Examples

>>> import lucid
>>> from lucid.nn.init import sparse_
>>> w = lucid.empty(256, 256)
>>> _ = sparse_(w, sparsity=0.9, std=0.01)
>>> int((w == 0).sum().item())   # ceil(0.9 * 256) = 231 zeros in each column
59136

>>> import lucid >>> from lucid.nn.init import sparse_ >>> w = lucid.empty(256, 256) >>> _ = sparse_(w, sparsity=0.9, std=0.01) >>> int((w == 0).sum().item()) # ceil(0.9 * 256) = 231 zeros in each column 59136