bert_base_nsp

→BERTForNextSentencePrediction

bert_base_nsp(pretrained: bool | str = False, weights: BERTBaseWeights | None = None, overrides: object = {})

source edit

Construct BERT-Base with the next-sentence-prediction head.

The binary head from pre-training task #2: given a sentence pair packed with token_type_ids, score whether the second follows the first. Unlike the fine-tuning heads, this one was trained — it is half of the original objective — so the released encoder and this head belong together.

Model Size

Parameters

pretrainedbool or str= False

Encoder-weight selector. False → fully random init; True → loads the bert_base encoder (BERTBaseWeights.DEFAULT) into the .bert trunk. The pooler and the two-way classifier are randomly initialised — the distributed checkpoint drops the NSP head.

weights(BERTBaseWeights, optional, keyword - only)= None

Explicit encoder-weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional BERTConfig field overrides.

Returns

BERTForNextSentencePrediction

BERT-Base wrapped with the two-way sentence-pair head.

Notes

Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805) §3.1 (task #2).

Examples

>>> import lucid
>>> from lucid.models.text.bert import bert_base_nsp
>>> model = bert_base_nsp(vocab_size=99, hidden_size=32,
...                       num_hidden_layers=2, num_attention_heads=2,
...                       intermediate_size=64).eval()
>>> out = model(lucid.tensor([[1, 2, 3, 4]]))
>>> out.logits.shape   # (N=1, 2) — IsNext / NotNext
(1, 2)

Used by 1

lucid.models.text.bert