Pretrained weight tags for lucid.models.bert_tiny.
Turc et al. 2019 L=2, H=128, A=2 distilled encoder (4.4 M params).
Attributes
WIKIPEDIA_BOOKSCORPUSWeightEntryWikipedia + BookCorpus pre-training checkpoint sourced from
transformers/google/bert_uncased_L-2_H-128_A-2.DEFAULTWeightEntryAlias for
WIKIPEDIA_BOOKSCORPUS.Notes
Reference: Turc, Chang, Lee, Toutanova, "Well-Read Students Learn Better", 2019 (arXiv:1908.08962).
Examples
>>> from lucid.models import bert_tiny
>>> model = bert_tiny(pretrained=True).eval()