Pretrained weight tags for lucid.models.gpt_lm.
GPT-1 decoder + tied lm_head over the 40 478-token BPE vocab.
Attributes
BOOKCORPUSWeightEntryBookCorpus pre-training checkpoint (decoder + LM head) sourced
from
transformers/openai-community/openai-gpt.DEFAULTWeightEntryAlias for
BOOKCORPUS.Notes
Reference: Radford, Narasimhan, Salimans, Sutskever, "Improving Language Understanding by Generative Pre-Training", OpenAI 2018.
Examples
>>> from lucid.models import gpt_lm
>>> model = gpt_lm(pretrained=True).eval()