Pretrained weight tags for lucid.models.bert_base_mlm.
BERT-Base encoder + tied MLM head over the 30 522 WordPiece vocab.
Attributes
WIKIPEDIA_BOOKSCORPUSWeightEntryWikipedia + BookCorpus pre-training checkpoint (encoder + MLM
head) sourced from
transformers/google-bert/bert-base-uncased.DEFAULTWeightEntryAlias for
WIKIPEDIA_BOOKSCORPUS.Notes
Reference: Devlin, Chang, Lee, Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805).
Examples
>>> from lucid.models import bert_base_mlm
>>> model = bert_base_mlm(pretrained=True).eval()