class
GPTTokenizer
extends
BPETokenizerGPTTokenizer(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)GPT-1 tokenizer — lowercased word-level BPE with </w>.
The end-of-word marker is what distinguishes this from generic BPE:
a word is seeded as list(word[:-1]) + [word[-1] + "</w>"], so the
merge table can tell "in" inside inside from "in</w>" as a
whole word. Without it a subword and a complete word share an id and
the vocabulary silently means two things.
Parameters
vocabdict[str, int]Token-string → id map from GPT-1's
vocab.json. Entries ending
in </w> are whole-word forms.mergeslist[tuple[str, str]]Ordered BPE merges; index = rank, lower applied first.
normalizerNormalizer= NoneDefaults to
NFC then Lowercase — §4.1 lowercases.pre_tokenizerPreTokenizer= NoneDefaults to
WhitespacePunctuationSplit, the
BasicTokenizer role in the reference's no-spaCy fallback.GPT-1 trained without bos/eos; pass them only if downstream code
needs them. They are registered, never inserted by
encode.Examples
>>> from lucid.models.text.gpt import GPTTokenizer
>>> vocab = {"l": 0, "o": 1, "o</w>": 2, "lo</w>": 3}
>>> tok = GPTTokenizer(vocab=vocab, merges=[("l", "o</w>")])
>>> tok.encode("lo")
[3]
>>> tok.decode(tok.encode("lo"))
'lo'