Lavender-2

Appearance

Lavender-2 TODO ​

Owner-visible, multi-round work items. Per-round agent scratch does not live here — that goes in the agent's live task tool.

Format: one line per item, link to the relevant file/commit where it helps. Keep it terse; expand context in the linked file, not here.

Next (actively targeted) ​

  • [ ] Re-run full_llm_pipeline.ipynb top-to-bottom against the uv kernel (Lavender (uv .venv)) and confirm the pre-flight dataset size-estimate and tokenizer RAM-estimate outputs are sane on this host.
  • [ ] Add an evaluation harness entry (MMLU / lm-evaluation-harness wiring) — called out as a known gap in docs/versions/0.1.0.md.
  • [ ] Add a smoke-test script that imports open_mythos.main + builds MythosConfig(**mythos_1b) inside the uv venv, so environment drift fails loudly in CI rather than silently in a notebook cell.

Later (not blocking; revisit when relevant) ​

  • [ ] Wikipedia cap at 32 GiB — reconsider once the full English dump exceeds that or the training budget grows. See docs/versions/project-history.md.
  • [ ] Consider pinning requirements.txt alongside uv.lock as a lockfile export in CI rather than a hand-regenerated file.
  • [ ] Add a uv-based pre-commit hook that blocks commits when uv.lock and requirements.txt disagree, closing LLM_CHECK.md §1.3 #8 mechanically.

Done (keep the three most recent MAJOR items; archive the rest) ​

  • [x] 2026-04-22 — v0.2.0 cut: swapped tokenizer from a trained byte-level BPE to tiktoken o200k_base (the GPT-4o encoding) extended with four chat specials (<|im_start|>, <|im_sep|>, <|im_end|>, <|pad|>); updated the chat template to GPT-4o ChatML across conversation_to_text, SFTDataset, DPODataset, generate(), and chat(); snapshotted the outgoing v0.1.x notebook to docs/versions/snapshots/0.1.x/ per docs/AGENTS.md §2.4. Release notes: docs/versions/0.2.0.md.
  • [x] 2026-04-22 — Swapped local dev environment from Anaconda to uv as the single source of truth; disabled auto_activate_base; registered the lavender-uv Jupyter kernel. Rule codified in docs/AGENTS.md §1.
  • [x] 2026-04-22 — Added project-specific docs/AGENTS.md and this docs/TODO.md per root AGENTS.md §2.