Pretrained weight tags for lucid.models.bert_large.
Devlin et al. 2018 L=24, H=1024, A=16 uncased encoder (335.1 M params).
Attributes
WIKIPEDIA_BOOKSCORPUSWeightEntryWikipedia + BookCorpus pre-training checkpoint sourced from
transformers/google-bert/bert-large-uncased.DEFAULTWeightEntryAlias for
WIKIPEDIA_BOOKSCORPUS.Notes
Reference: Devlin, Chang, Lee, Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805).
Examples
>>> from lucid.models import bert_large
>>> model = bert_large(pretrained=True).eval()