Pretrained weight tags for lucid.models.bert_large_mlm.
BERT-Large encoder + tied MLM head over the 30 522 WordPiece vocab.
Attributes
WIKIPEDIA_BOOKSCORPUSWeightEntryWikipedia + BookCorpus pre-training checkpoint (encoder + MLM
head) sourced from
transformers/google-bert/bert-large-uncased.DEFAULTWeightEntryAlias for
WIKIPEDIA_BOOKSCORPUS.Notes
Reference: Devlin, Chang, Lee, Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805).
Examples
>>> from lucid.models import bert_large_mlm
>>> model = bert_large_mlm(pretrained=True).eval()