bert_base_causal_lm(pretrained: bool | str = False, weights: BERTBaseWeights | None = None, overrides: object = {})Construct a BERT-Base trunk with a left-to-right LM head.
The trunk of bert_base under a causal attention mask, with the
tied vocabulary head predicting the next token rather than a masked
one. BERT was not pre-trained this way — the checkpoints saw
bidirectional context — so the released weights are a starting point
for causal fine-tuning, not a generative model as they stand.
Model Size
Parameters
pretrainedbool or str= FalseFalse → fully random init; True
→ loads the bert_base encoder
(BERTBaseWeights.DEFAULT) into the .bert trunk.pretrained.**overridesobject= {}BERTConfig field overrides.Returns
BERTForCausalLMBERT-Base wrapped with the causal LM head.
Notes
Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of
Deep Bidirectional Transformers for Language Understanding", NAACL 2019
(arXiv:1810.04805). The paper's §3.1 objective is masked, not causal;
this wrapper exists because the architecture supports the mask and
downstream code asks for it, and bert_base_mlm is the head the
checkpoints were trained with.
Examples
>>> import lucid
>>> from lucid.models.text.bert import bert_base_causal_lm
>>> model = bert_base_causal_lm(vocab_size=99, hidden_size=32,
... num_hidden_layers=2,
... num_attention_heads=2,
... intermediate_size=64).eval()
>>> out = model(lucid.tensor([[1, 2, 3, 4]]))
>>> out.logits.shape # (1, T=4, vocab_size=99)
(1, 4, 99)