RNN
_FlatRecurrentRNN(input_size: int, hidden_size: int, num_layers: int = 1, nonlinearity: str = 'tanh', bias: bool = True, batch_first: bool = False, dropout: float = 0.0, bidirectional: bool = False, device: DeviceLike = None, dtype: DTypeLike = None)Multi-layer Elman recurrent neural network (RNN).
Applies a stack of Elman RNN cells over an input sequence. At each time step the hidden state is updated by:
where is either (default) or
, controlled by nonlinearity.
The output of layer is fed as the input of layer
. When bidirectional=True, two RNNs process the
sequence in opposite directions and their outputs are concatenated
along the feature axis at each time step.
Inter-layer dropout (probability dropout) is applied between
adjacent layers during training.
Warning:
Vanilla RNNs are prone to the vanishing gradient problem: gradients are multiplied by at every time step, causing them to shrink exponentially for sequences longer than ~10–20 steps. For longer sequences, prefer
LSTMorGRU, which use gating mechanisms to maintain gradient flow.
Parameters
input_sizeinthidden_sizeintnum_layersint= 11.nonlinearity(tanh, relu)= 'tanh''tanh' is recommended for
most use cases; 'relu' can help when gradients vanish with
tanh. Default: 'tanh'.biasbool= TrueFalse, all bias parameters are omitted. Default: True.batch_firstbool= FalseTrue, input/output tensors are (N, L, *) instead of
the default (L, N, *). Default: False.dropoutfloat= 0.00.0 disables dropout. Default: 0.0.bidirectionalbool= FalseTrue, use a bidirectional RNN; the output feature
dimension becomes 2 * hidden_size. Default: False.deviceDeviceLike= NonedtypeDTypeLike= NoneNotes
- Input
x:(L, N, input_size)or(N, L, input_size)whenbatch_first=True. - h_0 (optional):
(D * num_layers, N, H)whereD = 2if bidirectional else1. Defaults to zeros. - output:
(L, N, D * H)or(N, L, D * H). - h_n:
(D * num_layers, N, H)— hidden state at the final time step for each layer and direction. - Unbatched: an
(L, input_size)input runs as a batch of one whateverbatch_firstsays. Its states are(D * num_layers, H), the output is(L, D * H), and a suppliedh_0of any other shape raisesValueError.
The parameters are weight_ih_l{k} (H, I), weight_hh_l{k}
(H, H), bias_ih_l{k} and bias_hh_l{k} (H,), with
_reverse for the backward direction — as for LSTM. A
checkpoint saved under the older cell_l{k}.weight_ih names loads.
flatten_parameters is a no-op kept for API compatibility.
A PackedSequence may be passed directly: padded steps are kept out
of the recurrence, h_n holds each sequence's state at its own
last step, and the output comes back packed. enforce_sorted=False
is fine — the batch is returned in the order you passed it.
Examples
Two-layer RNN, batch-first:
>>> import lucid, lucid.nn as nn
>>> rnn = nn.RNN(8, 16, num_layers=2, batch_first=True)
>>> x = lucid.randn(2, 5, 8) # (N=2, L=5, I=8)
>>> out, h_n = rnn(x)
>>> out.shape, h_n.shape
((2, 5, 16), (2, 2, 16))
Bidirectional RNN with ReLU activation:
>>> rnn_bi = nn.RNN(
... 8, 16, nonlinearity='relu',
... bidirectional=True, batch_first=True,
... )
>>> x2 = lucid.randn(3, 7, 8)
>>> out2, h_n2 = rnn_bi(x2)
>>> out2.shape # D*H = 2*16 = 32
(3, 7, 32)
>>> h_n2.shape # D*num_layers = 2*1 = 2
(2, 3, 16)See Also
RNNCell—Single time-step Elman cell.LSTM—Gated RNN with a separate cell state (better for long seqs).GRU—Gated RNN without a separate cell state.
Used by 1
Constructors
1__init__
→None__init__(input_size: int, hidden_size: int, num_layers: int = 1, nonlinearity: str = 'tanh', bias: bool = True, batch_first: bool = False, dropout: float = 0.0, bidirectional: bool = False, device: DeviceLike = None, dtype: DTypeLike = None)Initialise the RNN module. See the class docstring for parameter semantics.
Instance methods
3Return a string representation of the layer's configuration.
No-op for API compatibility (see LSTM.flatten_parameters).