class
GPT2TokenizerFast
extends
ByteLevelBPETokenizerFastGPT2TokenizerFast(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)GPT-2 tokenizer — C++-backed.
Bit-identical to GPT2Tokenizer. The BPE merge loop runs
in C++ via the engine-side BPE binding for production
throughput.
Constructor parameters mirror GPT2Tokenizer — see that
class for the full reference.
Examples
>>> from lucid.models.text.gpt2._tokenizer import GPT2TokenizerFast
>>> vocab = {"l": 0, "o": 1, "o</w>": 2, "lo</w>": 3}
>>> tok = GPT2TokenizerFast(vocab=vocab, merges=[("l", "o</w>")])
>>> tok.encode("lo")
[0, 1]
Same ids as the plain tokenizer — the speed comes from how the merges
are looked up, not from a different segmentation, and the two must
not disagree.