Pretrained weight tags for lucid.models.gpt.
Radford et al. 2018 12-layer decoder trunk (H=768, A=12, 116.5 M params).
Attributes
BOOKCORPUSWeightEntryBookCorpus pre-training checkpoint sourced from
transformers/openai-community/openai-gpt.DEFAULTWeightEntryAlias for
BOOKCORPUS.Notes
Reference: Radford, Narasimhan, Salimans, Sutskever, "Improving Language Understanding by Generative Pre-Training", OpenAI 2018.
Examples
>>> from lucid.models import gpt
>>> model = gpt(pretrained=True).eval()