Abstract base for every tokenizer in lucid.utils.tokenizer.
Subclasses implement four hooks:
_encode_one(text) -> list[int]— algorithm-specific encode for one pre-normalized + pre-tokenized chunk. The base class handles normalization, pre-tokenization, and special-token insertion around this hook._decode_one(ids) -> str— algorithm-specific decode of one id sequence (special tokens NOT yet stripped — the base class handlesskip_special_tokens).vocab_size— total number of distinct ids.algo— short algorithm name for serialisation ("bpe","wordpiece","unigram","byte_bpe").
The base class layers
encode / decode / encode_batch /
decode_batch / __call__ on top — those are
fully generic and don't need to be overridden.
Examples
Every tokenizer here is one of these, so the three calls below are
the whole surface a caller needs — build or train, encode, decode:
>>> from lucid.utils.tokenizer import WordTokenizer
>>> tok = WordTokenizer()
>>> tok.train(["the quick brown fox", "the lazy dog"], vocab_size=32)
>>> tok.decode(tok.encode("the quick dog"))
'the quick dog'
Subclassing one means supplying the algorithm and inheriting the
rest — the special-token handling, save / from_pretrained,
and the batch helpers.See Also
algorithm with identical encode outputs.
Used by 9
Constructors
2Initialise the special-token registry; subclasses must populate the vocab + algorithm state before calling this.
__call__
→dict__call__(text: str | list[str], add_special_tokens: bool = True, padding: bool | str = False, truncation: bool = False, max_length: int | None = None, return_tensors: str | None = None, return_attention_mask: bool = True)Hugging-Face-style entry point.
Parameters
textstr or list of stradd_special_tokensbool= Truepaddingbool or str= FalseFalse — no padding. True or "longest" — pad
every sequence to the longest in the batch.
"max_length" — pad every sequence to max_length.truncationbool= FalseTrue, truncate each sequence to max_length
(no-op if max_length is None). The content is cut,
never the framing: max_length counts the special tokens
and they stay, so an EOS survives truncation.max_lengthint= Nonepadding="max_length".return_tensorslucid= "lucid""lucid", wrap each output value as a
lucid.Tensor (dtype int32) and stack the batch
into a 2-D tensor. When None, return plain Python
lists.return_attention_maskbool= TrueTrue (and padding is set), include an
"attention_mask" entry: 1 for real tokens, 0 for
padding.Returns
dictAlways contains "input_ids". Optionally
"attention_mask" (when padding + return_attention_mask)
and "special_tokens_mask" (always, for downstream
BERT-style MLM consumers).
Properties
12Short algorithm name ("bpe" / "wordpiece" / ...).
Used by save to route serialisation to the right
format and by from_pretrained to pick the right
subclass when loading from a generic path.
Every special-token id (canonical + extras), deduped.
Id of the bos special token (None if undefined).
Id of the cls special token (None if undefined).
Id of the eos special token (None if undefined).
Id of the mask special token (None if undefined).
Id of the pad special token (None if undefined).
The configured post-processor, or None for the class default.
Assign a lucid.utils.tokenizer._post_processors.PostProcessor
to change how encode frames sequences — for example a
lucid.utils.tokenizer._post_processors.TemplateProcessing
reproducing a checkpoint's template — or None to go back to
the class's default layout.
Returns
PostProcessor or NoneThe processor loaded from tokenizer.json or assigned here.
Examples
>>> from lucid.utils.tokenizer import SpecialTokens, WordTokenizer
>>> from lucid.utils.tokenizer._post_processors import TemplateProcessing
>>> tok = WordTokenizer(
... {"<unk>": 0, "</s>": 1, "hi": 2},
... special_tokens=SpecialTokens(unk="<unk>", eos="</s>"),
... )
>>> tok.encode("hi")
[2, 1]
>>> tok.post_processor = TemplateProcessing("$A", special_tokens={})
>>> tok.encode("hi")
[2]Id of the sep special token (None if undefined).
special_tokens: SpecialTokensThe registry of canonical + extra special tokens.
Id of the unk special token (None if undefined).
Total number of distinct token ids (including specials).
Instance methods
10Inverse of convert_tokens_to_ids.
Map one or many token strings to their ids.
Unknown tokens raise KeyError (callers wanting an
UNK fallback should do their own .get(token, unk_id)
lookup).
Decode a sequence of ids back to text.
When skip_special_tokens=True every id in
all_special_ids is dropped before the
algorithm-specific _decode_one runs.
Parameters
idsiterable of intskip_special_tokens(bool, optional, keyword - only)= TrueTrue, every id in all_special_ids
(BOS/EOS/PAD/CLS/SEP/MASK + any extras) is filtered out
before _decode_one reassembles the surface string.Returns
strReconstructed text — exact reverse of encode for
lossless algorithms (Unigram / byte-BPE); lossy for
normalising flavours (NFC + Lowercase).
decode_batch
→list[str]decode_batch(batch_ids: Iterable[Iterable[int]] | list[list[int]], skip_special_tokens: bool = True)Vectorised decode over a list of id sequences.
Encode a single string to a list of token ids.
When add_special_tokens=True the sequence is framed by the
tokenizer's post_processor if one is configured (a
tokenizer.json carries it — T5's appends only </s>),
and otherwise by the class's default layout: BOS/EOS, BERT's
[CLS] … [SEP], or nothing for GPT-2.
Parameters
textstr_encode_one hook before the BPE / WordPiece /
Unigram merge loop runs.add_special_tokens(bool, optional, keyword - only)= TrueTrue, frame the ids with special tokens as
described above, via
_build_inputs_with_special_tokens. Pass False
to return the raw merge-loop output untouched (useful for
offline analysis, alignment, or batching pipelines that
inject specials elsewhere).Returns
list of intToken ids for text in left-to-right order.
encode_batch
→list of list of intencode_batch(texts: Iterable[str] | list[str], add_special_tokens: bool = True)Vectorised encode over a list of strings.
Default loops through encode; subclasses with a true
batched fast path (the C++ Fast tokenizers) override.
Parameters
textsiterable of stradd_special_tokens(bool, optional, keyword - only)= Trueencode for each item — when True,
every sequence is wrapped with its canonical BOS/EOS or
CLS/SEP markers.Returns
list of list of intOne id sequence per input string, same order as texts.
Return the token-string → id map.
Default returns an empty dict; subclasses with a real vocab
override this. Used by save and by special-token-id
resolution in _refresh_special_ids.
Return the surface form for token_id or None if
out of range.
Default does an O(N) reverse scan of get_vocab;
subclasses with an O(1) reverse table should override.
Count the special tokens encode(add_special_tokens=True) adds.
Parameters
pairbool= FalseReturns
intHow many ids framing adds around the content — the amount
__call__ reserves out of max_length when it
truncates.
Examples
>>> from lucid.utils.tokenizer import SpecialTokens, WordTokenizer
>>> tok = WordTokenizer(
... {"<unk>": 0, "<s>": 1, "</s>": 2},
... special_tokens=SpecialTokens(unk="<unk>", bos="<s>", eos="</s>"),
... )
>>> tok.num_special_tokens_to_add()
2Persist the tokenizer to directory.
Default writes tokenizer.json (unified format) +
special_tokens_map.json. Subclasses with algorithm-
specific legacy files (vocab.json + merges.txt for
BPE, vocab.txt for WordPiece) override this hook to
emit those alongside.