Pretrained weight tags for lucid.models.gpt2_small.
Radford et al. 2019 12-layer trunk (H=768, A=12, 124 M params).
Attributes
WEBTEXTWeightEntryWebText pre-training checkpoint sourced from
transformers/gpt2.DEFAULTWeightEntryAlias for
WEBTEXT.Notes
Reference: Radford, Wu, Child, Luan, Amodei, Sutskever, "Language Models are Unsupervised Multitask Learners", OpenAI 2019.
Examples
>>> from lucid.models import gpt2_small
>>> model = gpt2_small(pretrained=True).eval()