Lavender-2

Appearance

Tokenizer ​

What a tokenizer does ​

A neural network cannot consume raw text — it consumes integer IDs. A tokenizer is the bridge: it maps a string to a sequence of IDs that index into a vocabulary of size V. The model's embedding layer is then a V × d_model matrix of learned vectors, one per token ID.

Two extremes on the vocabulary-size axis:

  • Character-level. V is tiny (~256 for ASCII + some Unicode), but sequences become very long. The model wastes compute on short patterns it has to rediscover from scratch (t, h, e → "the").
  • Word-level. Sequences are short, but V explodes and any rare or unseen word collapses to <unk>, destroying information.

Byte-Pair Encoding (BPE) is the pragmatic middle. Start from bytes (256 symbols), then repeatedly merge the most frequent adjacent pair into a new token. Frequent sub-words collapse to single tokens; rare words decompose into a few sub-tokens. No <unk> is ever needed because bytes cover everything.

What changed in v0.2.0 ​

The v0.1.x line trained its own byte-level BPE from scratch on the upsampled training corpus (32 K dev vocab, 128 K target). That worked but forced the downstream cache chain to anchor on a tokenizer that could drift between training runs, and it took tens of minutes to an hour on a real corpus.

v0.2.0 replaces that with tiktoken's o200k_base encoding — the encoding tiktoken.encoding_for_model("gpt-4o") resolves to. Vocabulary is fixed (n_vocab = 200_019), ships bundled in the tiktoken wheel, loads in milliseconds, and is identical across every machine.

The extended encoding we build at runtime (o200k_base_lavender_chat) adds four chat-format specials on top:

TokenPurpose
<|im_start|>turn start
<|im_sep|>role / content separator
<|im_end|>turn end
<|pad|>batch padding

Final vocab size: 200,023.

The TiktokenAdapter ​

The rest of the notebook expects the HuggingFace tokenizers-style API: tok.encode(text).ids, tok.decode(ids), tok.token_to_id(...), tok.get_vocab_size(), tok.save(path), tok.to_str(). TiktokenAdapter is a thin wrapper that exposes those method names on top of a tiktoken.Encoding:

  • encode(text) passes allowed_special=ALLOWED_SPECIAL so the four chat specials are recognised as tokens rather than raising.
  • token_to_id(s) looks up specials in the adapter's local table and falls back to encode_single_token(s) for regular tokens.
  • save(path) writes a small JSON sidecar ({adapter, base, specials, n_vocab}) — not loadable as a HuggingFace tokenizer.json, but TiktokenAdapter.from_file(path) rebuilds the live adapter from it.
  • to_str() returns a content-derived JSON blob used by _tokenizer_sha; any change to the base encoding or specials invalidates every downstream dataset cache automatically.

Chat template ​

The rendered template at training time is GPT-4o's production ChatML:

text
<|im_start|>user<|im_sep|>What is the weather?<|im_end|><|im_start|>assistant<|im_sep|>It's sunny.<|im_end|>

No whitespace between turns. The role string (user, assistant, system) is regular text, encoded with the base BPE — only the structural markers are specials. That matches OpenAI's serving behavior and keeps the representation portable to any model that uses o200k_base.

Why this is usually the right default now ​

  • No training run. tiktoken.get_encoding("o200k_base") loads in milliseconds; the old BPE trainer took tens of minutes on the real corpus.
  • Stable across runs. Every machine and every session gets the exact same vocabulary. No merge-order drift when the corpus order changes.
  • Portability. Checkpoints tokenised with o200k_base are directly comparable to any GPT-4o / GPT-4.1 / ChatGPT-4o-latest representation. Migrating to or from pretrained weights is tokenisation-compatible.
  • Coverage. o200k_base is optimised for modern multilingual text (including code, emoji, CJK); it dominates hand-rolled small BPEs on those segments.

What hasn't changed ​

  • Still byte-safe. o200k_base is a byte-level BPE just like the v0.1.x tokenizer. Any UTF-8 input encodes losslessly.
  • Still CPU-only at encode time. tiktoken is Rust-backed; there is no CUDA encode path, and there doesn't need to be — encoding a batch of conversations at __getitem__ time is microseconds.
  • Still content-keyed downstream. The tok_sha in the PretrainDataset / SFTDataset / DPODataset cache keys comes from adapter.to_str() now instead of the HF to_str(), but the invalidation semantics are identical: change the encoding, the specials, or the chat template → the caches rebuild.

Reading the code ​

The tokenizer cell lives in full_llm_pipeline.ipynb; search for TiktokenAdapter. The chat template is rendered by conversation_to_text(...) (for pretrain) and by explicit [im_start] + role_ids + [im_sep] + content_ids + [im_end] construction inside SFTDataset.__init__ and DPODataset.__init__.

Papers and references ​

Next stage: Architecture — the Transformer block that consumes these token IDs.