Pretrained weight tags for lucid.models.bert_mini.
Turc et al. 2019 L=4, H=256, A=4 distilled encoder (11.2 M params).
Attributes
WIKIPEDIA_BOOKSCORPUSWeightEntryWikipedia + BookCorpus pre-training checkpoint sourced from
transformers/google/bert_uncased_L-4_H-256_A-4.DEFAULTWeightEntryAlias for
WIKIPEDIA_BOOKSCORPUS.Notes
Reference: Turc, Chang, Lee, Toutanova, "Well-Read Students Learn Better", 2019 (arXiv:1908.08962).
Examples
>>> from lucid.models import bert_mini
>>> model = bert_mini(pretrained=True).eval()