class
BERTTokenizer
extends
WordPieceTokenizerBERTTokenizer(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)BERT tokenizer — pure-Python reference.
A thin convenience subclass of
lucid.utils.tokenizer.WordPieceTokenizer with BERT's
canonical [UNK]/[CLS]/[SEP]/[PAD]/[MASK] registry, the
lucid.utils.tokenizer.normalizers.BERTNormalizer
(lowercase + accent-strip + Chinese-char split + clean-text), and
lucid.utils.tokenizer.pre_tokenizers.WhitespacePunctuationSplit
pre-tokenizer baked in as defaults.
Loads any published HF BERT-family vocab.txt checkpoint
unchanged.
Parameters
vocabdict[str, int]BERT-style vocab. Continuation subwords prefixed with
##.
Pass {} and call WordPieceTokenizer.train to
train a fresh vocab from a corpus.unk_tokenstr= "[UNK]"Token emitted when no valid longest-match prefix is found.
Must be present in
vocab for the id to materialise.continuing_prefixstr= "##"Marker for non-initial subwords (BERT convention).
max_chars_per_wordint= 100Words longer than this skip the longest-match loop and emit
the
unk_token directly (mirrors BERT BasicTokenizer).do_lower_casebool= TrueForwarded to
BERTNormalizer as both lowercase and
strip_accents. Set to False for cased checkpoints
(bert-base-cased, multilingual models), which keep their
accents as well as their case. Ignored if normalizer is
given explicitly.normalizerNormalizer= NoneOverride the default
BERTNormalizer.pre_tokenizerPreTokenizer= NoneOverride the default
WhitespacePunctuationSplit.Override the default BERT registry.
Examples
Train a small BERT-style tokenizer from a corpus, then round-trip:
>>> from lucid.models.text.bert import BERTTokenizer
>>> tok = BERTTokenizer(vocab={})
>>> tok.train(["hello world", "hello bert"] * 8, vocab_size=60)
>>> ids = tok.encode("Hello World", add_special_tokens=False)
>>> # default do_lower_case=True → "Hello" and "hello" share ids.
>>> tok.decode(ids, skip_special_tokens=False)
'hello world'
Encoding with add_special_tokens=True (the default) frames the
sequence as [CLS] … [SEP], the input every BERT head was trained
on; [CLS] is the position the pooler reads.
>>> vocab = {"[UNK]": 0, "[CLS]": 1, "[SEP]": 2, "hello": 3}
>>> BERTTokenizer(vocab=vocab).encode("hello")
[1, 3, 2]See Also
BERTTokenizerFast—C++-backed variant with identical output.
Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)