class
Transformer
extends
ModuleTransformer(d_model: int = 512, nhead: int = 8, num_encoder_layers: int = 6, num_decoder_layers: int = 6, dim_feedforward: int = 2048, dropout: float = 0.1, activation: str | Callable[[Tensor], Tensor] = 'relu', custom_encoder: TransformerEncoder | None = None, custom_decoder: TransformerDecoder | None = None, layer_norm_eps: float = 1e-05, batch_first: bool = False, norm_first: bool = False, bias: bool = True, device: DeviceLike = None, dtype: DTypeLike = None, attention_dropout: float | None = None)Full encoder-decoder Transformer architecture.
Implements the complete sequence-to-sequence model introduced in "Attention Is All You Need" (Vaswani et al., 2017, arXiv:1706.03762). The model consists of:
- An
TransformerEncoderthat encodes the source sequence into a continuous memory representation. - A
TransformerDecoderthat generates the target sequence conditioned on the encoder memory.
High-level data flow:
Both the encoder and decoder are built with a final
LayerNorm applied after the last layer.
Parameters
d_modelint= 512Dimensionality of all model representations
. Default:
512.nheadint= 8Number of attention heads in every multi-head attention
sub-layer (encoder self-attention, decoder self-attention,
and decoder cross-attention all use the same
nhead).
Default: 8.num_encoder_layersint= 6Number of layers in the encoder stack. Default:
6.num_decoder_layersint= 6Number of layers in the decoder stack. Default:
6.dim_feedforwardint= 2048Inner dimension of the position-wise FFN in each layer.
Default:
2048.dropoutfloat= 0.1Dropout probability applied throughout the model.
Default:
0.1.activationstr or callable= 'relu'Activation function for all FFN layers:
"relu" (default),
"gelu", or a callable applied as-is.batch_firstbool= FalseIf
True, all input and output tensors use
(batch, seq, feature) layout. Default: False
(sequence-first).deviceDeviceLike= NoneDevice for all parameters.
dtypeDTypeLike= NoneData type for all parameters.
Attributes
encoderModuleThe encoder module (
num_encoder_layers stacked encoder
layers + final LayerNorm, or custom_encoder).decoderModuleThe decoder module (
num_decoder_layers stacked decoder
layers + final LayerNorm, or custom_decoder).d_modelintModel dimension used at construction.
nheadintNumber of attention heads used at construction.
Notes
When batch_first=False (default):
src:tgt:- Output:
When batch_first=True:
src:tgt:- Output:
where is the source length, is the target
length, is the batch size, and = d_model.
Examples
>>> import lucid
>>> import lucid.nn as nn
>>> model = nn.Transformer(d_model=512, nhead=8,
... num_encoder_layers=6,
... num_decoder_layers=6)
>>> src = lucid.randn(10, 2, 512) # (src_len, batch, d_model)
>>> tgt = lucid.randn(7, 2, 512) # (tgt_len, batch, d_model)
>>> out = model(src, tgt)
>>> out.shape
(7, 2, 512)Used by 1
Constructors
1dunder
__init__
→None__init__(d_model: int = 512, nhead: int = 8, num_encoder_layers: int = 6, num_decoder_layers: int = 6, dim_feedforward: int = 2048, dropout: float = 0.1, activation: str | Callable[[Tensor], Tensor] = 'relu', custom_encoder: TransformerEncoder | None = None, custom_decoder: TransformerDecoder | None = None, layer_norm_eps: float = 1e-05, batch_first: bool = False, norm_first: bool = False, bias: bool = True, device: DeviceLike = None, dtype: DTypeLike = None, attention_dropout: float | None = None)Initialise the Transformer module. See the class docstring for parameter semantics.
Static methods
1Instance methods
2Return a string representation of the layer's configuration.
forward(src: Tensor, tgt: Tensor, src_mask: Tensor | None = None, tgt_mask: Tensor | None = None, memory_mask: Tensor | None = None, src_key_padding_mask: Tensor | None = None, tgt_key_padding_mask: Tensor | None = None, memory_key_padding_mask: Tensor | None = None, src_is_causal: bool | None = None, tgt_is_causal: bool | None = None, memory_is_causal: bool = False)Run the forward pass of the module.