Pretrained weight tags for lucid.models.bert_base.
Devlin et al. 2018 L=12, H=768, A=12 uncased encoder (109.5 M params).
Attributes
WIKIPEDIA_BOOKSCORPUSWeightEntryWikipedia + BookCorpus pre-training checkpoint sourced from
transformers/google-bert/bert-base-uncased.DEFAULTWeightEntryAlias for
WIKIPEDIA_BOOKSCORPUS.Notes
Reference: Devlin, Chang, Lee, Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805).
Examples
>>> from lucid.models import bert_base
>>> model = bert_base(pretrained=True).eval()