class
GPTTokenizerFast
extends
GPTTokenizerGPTTokenizerFast(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)GPT-1 tokenizer — same results, same Python merge loop.
There is no C++ acceleration for this scheme. The engine's BPE
seeds per codepoint, which was measured directly: encoding
"hello</w>" against a vocabulary containing "o</w>" returns
['he', 'l', 'l', 'o', '<', '/', 'w', '>'] — the marker is torn
into four symbols. Expressing GPT-1's end-of-word symbol there needs
an engine change, so this subclass exists to keep the name working and
the output correct, not because it is faster.
Byte-level families (GPT-2, RoBERTa) do have a genuine fast path; see
lucid.utils.tokenizer.ByteLevelBPETokenizerFast.
Parameters
vocabdict[str, int]Token-string → id map from GPT-1's
vocab.json. Entries ending
in </w> are whole-word forms.mergeslist[tuple[str, str]]Ordered BPE merges; index = rank, lower applied first.
normalizerNormalizer= NoneDefaults to
NFC then Lowercase — §4.1 lowercases.pre_tokenizerPreTokenizer= NoneDefaults to
WhitespacePunctuationSplit, the
BasicTokenizer role in the reference's no-spaCy fallback.GPT-1 trained without bos/eos; pass them only if downstream code
needs them. They are registered, never inserted by
encode.Examples
>>> from lucid.models.text.gpt._tokenizer import GPTTokenizerFast
>>> vocab = {"l": 0, "o": 1, "o</w>": 2, "lo</w>": 3}
>>> tok = GPTTokenizerFast(vocab=vocab, merges=[("l", "o</w>")])
>>> tok.encode("lo")
[3]
One id: the merge applies and the pair collapses into the vocabulary
entry that covers both.