class

BERTForSequenceClassification

extendsSequenceClassificationModel

BERTForSequenceClassification(config: BERTConfig)

source edit

BERT with a pooled-CLS linear classifier for sequence-level tasks.

Wraps the bidirectional encoder with a dropout-regularised linear head operating on the [CLS] pooled embedding. This is the standard fine-tuning recipe for GLUE-style sentence/sentence-pair tasks (SST-2, MNLI, QQP, RTE, ...) introduced in Devlin et al., 2018 §4.1.

Parameters

configBERTConfig

BERT hyperparameters. config.num_labels sets the output dimension; config.classifier_dropout (falling back to hidden_dropout) sets the dropout applied before the linear.

Attributes

bertBERTModel

Underlying bidirectional encoder trunk.

dropoutnn.Dropout

Dropout layer applied to the pooled [CLS] embedding.

classifiernn.Linear

Final linear of shape (hidden_size, num_labels) producing logits.

Notes

Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805).

The pooled representation is $p = \tanh(W_{\mathrm{pool}}\, h_{[\mathrm{CLS}]} + b_{\mathrm{pool}})$ , and the final logits are $z = W_{\mathrm{cls}}\,\mathrm{Dropout}(p) + b_{\mathrm{cls}}$ . When labels is provided the loss is exposed as output.loss: cross-entropy over num_labels classes, or — when num_labels == 1, the single-output regression setting GLUE's STS-B uses — mean squared error against the float targets.

Examples

>>> import lucid
>>> from lucid.models.text.bert import BERTConfig, BERTForSequenceClassification
>>> cfg = BERTConfig(num_labels=3, num_hidden_layers=2, hidden_size=128,
...                  num_attention_heads=2, intermediate_size=512)
>>> model = BERTForSequenceClassification(cfg).eval()
>>> input_ids = lucid.tensor([[101, 7592, 102]])
>>> out = model(input_ids)
>>> out.logits.shape   # (B=1, num_labels=3)
(1, 3)

Used by 2

Constructors

dunder

init

→None

__init__(config: BERTConfig)

source edit

Instance methods

forward

→SequenceClassificationOutput

forward(input_ids: Tensor, attention_mask: Tensor | None = None, token_type_ids: Tensor | None = None, labels: Tensor | None = None)

source edit

Sum the word, position and token-type embeddings.

Exactly one of input_ids and inputs_embeds is required. inputs_embeds skips the word-embedding lookup, which is what a caller doing soft prompts or adapter tuning needs; position_ids overrides the default 0..T-1, which packed or shifted sequences need.

>>> import lucid >>> from lucid.models.text.bert import BERTConfig, BERTForSequenceClassification >>> cfg = BERTConfig(num_labels=3, num_hidden_layers=2, hidden_size=128, ... num_attention_heads=2, intermediate_size=512) >>> model = BERTForSequenceClassification(cfg).eval() >>> input_ids = lucid.tensor([[101, 7592, 102]]) >>> out = model(input_ids) >>> out.logits.shape # (B=1, num_labels=3) (1, 3)