Lavender-2

Appearance

v0.2.0 — Tokenizer swap to tiktoken (GPT-4o) ​

Status: active architecture line, supersedes v0.1.4. Current package release: v0.2.1.

Version source: historical pyproject.toml state (version = "0.2.0").

What changed vs v0.1.x ​

v0.1.x trained a byte-level BPE tokenizer from scratch on the upsampled training corpus (32 K dev vocab, 128 K target). v0.2.0 replaces that with tiktoken's o200k_base encoding — the encoding tiktoken.encoding_for_model("gpt-4o") resolves to — extended at runtime with four chat-format special tokens so the model can be trained on the same ChatML template OpenAI uses for GPT-4o.

No BPE training runs in this line. tiktoken.get_encoding(...) loads the bundled vocabulary in milliseconds, the extended encoding is built in-process, and the TiktokenAdapter wraps it to expose the same API (.encode(text).ids, .decode(ids), .token_to_id(...), .get_vocab_size(), .save(path), .to_str()) the rest of the notebook already calls.

Per docs/AGENTS.md §2.2, "swap the tokenizer algorithm" is one of the listed MAJOR-bump triggers, so this is cut as v0.2.0 rather than a MINOR. Per §2.4 the outgoing v0.1.x notebook is archived at docs/versions/snapshots/0.1.x/full_llm_pipeline.ipynb (outputs stripped); the live notebook no longer matches the snapshot.

Tokenizer ​

  • Base encoding: tiktoken.get_encoding("o200k_base") — n_vocab = 200,019. The vocabulary + merge ranks ship with the tiktoken wheel; no training data is consumed.

  • Chat specials added (extended encoding name o200k_base_lavender_chat):

    TokenRole
    <|im_start|>turn start
    <|im_sep|>role / content separator
    <|im_end|>turn end
    <|pad|>batch padding

  • Vocab size (extended): 200,023. All three model configs (DEMO_CFG, PRODUCTION_32B_CFG) now pull vocab_size=VOCAB_SIZE so the embedding matrix tracks the live tokenizer automatically.

  • Inherited specials (from o200k_base): <|endoftext|> (id 199999), <|endofprompt|> (id 200018).

  • Special-token safety: TiktokenAdapter.encode(text) passes an ALLOWED_SPECIAL frozen set that includes all six special-token strings above. tiktoken's default raises on special-token text to prevent prompt injection from leaking; our ChatML template deliberately emits them, so we opt in.

Chat template (data format) ​

Input to the pipeline is still [{"role": ..., "content": ...}] from the normalizers — no dataset code changed. The template rendering did.

v0.1.x (removed) ​

text
<|user|>
What is the weather?
<|end|>
<|assistant|>
It's sunny.
<|end|>

Custom special tokens (<|user|>, <|assistant|>, <|end|>) that only this repo's tokenizer recognised.

v0.2.0 ​

text
<|im_start|>user<|im_sep|>What is the weather?<|im_end|><|im_start|>assistant<|im_sep|>It's sunny.<|im_end|>

GPT-4o's production template. Turns are concatenated with no whitespace between them. The role string (user, assistant, system) is encoded with normal BPE, not as a special token — only the structural markers are. This matches OpenAI's serving behavior and keeps the representation portable to any model that uses o200k_base.

Code map ​

  • full_llm_pipeline.ipynb:

    • Imports cell: import tiktoken.
    • Tokenizer cell (f5f2b0dc): completely rewritten — loads o200k_base, extends specials, constructs TiktokenAdapter. Removed the UTF-8 byte pre-flight and the BPE training loop; neither is needed without training.
    • conversation_to_text (46c6f365): emits ChatML per turn.
    • SFTDataset.__init__ (76e4909a): turn layout is [im_start] + role_ids + [im_sep] + content_ids + [im_end].
    • DPODataset.__init__ (75ef3a6b): same template for both chosen and rejected sides.
    • generate() (26549d01): stops on either IM_END_ID or EOS_ID; test prompt uses ChatML with an open assistant header.
    • chat() (c3289c00): builds ChatML history; reply extraction uses rfind("<|im_start|>assistant<|im_sep|>") + the next <|im_end|>.
    • Model config cell (a64c93a3): PRODUCTION_32B_CFG.vocab_size now tracks VOCAB_SIZE (was hardcoded 128,256 for LLaMA-3).

  • pyproject.toml: tiktoken>=0.12.0 added to dependencies.

  • uv.lock: 106 packages resolved (105 → 106, adds tiktoken).

  • requirements.txt: regenerated from uv.lock via uv export per docs/AGENTS.md §1.4.

Downstream cache invalidation ​

Every data/cache/*.bin from v0.1.4 becomes stale on first v0.2.0 run — automatically. The cache keys embed tok_sha, pad_id, and the content fingerprint of train_texts / train_data. tok_sha differs (different tokenizer.to_str()), pad_id differs (new <|pad|> is at 200,022 instead of BPE vocab position 0), and texts_fp differs (the ChatML rendering changes the character content). All three force a clean rebuild via PretrainDataset.__init__ / SFTDataset.__init__ / DPODataset.__init__. No manual intervention needed; nothing silently uses a stale cache.

To reclaim disk from the old shards: rm data/cache/*.bin data/cache/*.meta.json data/cache/tokenizer.json. The new shards will be written in their place.

What's not in this release ​

  • Not validated end-to-end on real data. The code compiles, the token-ID arithmetic is correct on paper, and each of the 8 edited cells passes ast.parse. A full pretrain → SFT → DPO → chat run against the 12-dataset corpus has not been executed in this repo.
  • chat.py not updated. The standalone CLI inference script still loads a v0.1.x tokenizer.json and emits the old ChatML variant. It will need a similar refresh before inference on a v0.2.0 checkpoint works end-to-end.
  • DPO rejected-side determinism unchanged. random.shuffle on the rejected turn is still not seeded per conversation; a partial rebuild uses a fresh RNG. Acceptable for the same reasons as v0.1.4 — the cache design skips rebuilds whenever possible.

Incremental revisions ​

  • v0.2.1 — long-running process visibility sweep.

Upstream references ​