class
GPT2Tokenizer
extends
ByteLevelBPETokenizerGPT2Tokenizer(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)GPT-2 tokenizer — pure-Python reference.
A thin convenience subclass of
lucid.utils.tokenizer.ByteLevelBPETokenizer with
<|endoftext|> pre-registered as the bos / eos / unk marker.
Encoding adds no special tokens, as the reference GPT-2 tokenizer
adds none.
Parameters
vocabdict[str, int]Token-string → id map (byte-mapped Unicode keys).
mergeslist[tuple[str, str]]Ordered BPE merges; index = rank.
add_prefix_spacebool= FalsePrepend a space so the first word receives the
Ġ marker.
Off in GPT-2 (matches the published vocab); on in RoBERTa.normalizerNormalizer= NoneDefault
lucid.utils.tokenizer.normalizers.NFC.Override the default GPT-2 registry (single
<|endoftext|>
shared across bos / eos / unk).Examples
>>> from lucid.models.text.gpt2._tokenizer import GPT2Tokenizer
>>> vocab = {"l": 0, "o": 1, "o</w>": 2, "lo</w>": 3}
>>> tok = GPT2Tokenizer(vocab=vocab, merges=[("l", "o</w>")])
>>> tok.encode("lo")
[0, 1]
The ids themselves, not an object wrapping them. Two here rather
than one because the merge is only applied when both halves are
present as written, and the pretokeniser split this differently.See Also
GPT2TokenizerFast—C++-backed variant with identical output.- lucid.models.text.gpt.GPTTokenizer—Predecessor without an
end-of-text marker.