Pretrained weight tags for lucid.models.bert_small.
Turc et al. 2019 L=4, H=512, A=8 distilled encoder (28.8 M params).
Attributes
WIKIPEDIA_BOOKSCORPUSWeightEntryWikipedia + BookCorpus pre-training checkpoint sourced from
transformers/google/bert_uncased_L-4_H-512_A-8.DEFAULTWeightEntryAlias for
WIKIPEDIA_BOOKSCORPUS.Notes
Reference: Turc, Chang, Lee, Toutanova, "Well-Read Students Learn Better", 2019 (arXiv:1908.08962).
Examples
>>> from lucid.models import bert_small
>>> model = bert_small(pretrained=True).eval()