Lavender-2

Appearance

v0.1.0 — Baseline (superseded by v0.2.0) ​

Status: superseded. The v0.1.x line is preserved here for reference and at docs/versions/snapshots/0.1.x/full_llm_pipeline.ipynb as an executable archive. The current baseline is v0.2.0 which swaps the tokenizer to tiktoken (o200k_base / GPT-4o) and updates the chat template to GPT-4o ChatML. The pre-v0.1.0 history in git log is exploratory.

Version source: pyproject.toml (version = "0.1.0").

What's shipped ​

  • Data ingest: download_data.py with a 12-dataset registry covering dialogue (WildChat, SODA, DailyDialog, Topical-Chat, MultiWOZ, Taskmaster), encyclopedic text (Wikipedia 20231101.en), gated chat logs (LMSYS-Chat-1M, Chatbot Arena), and distilled reasoning traces (Claude Opus 4.6, KIMI K2.5, filtered Opus-4.6). HF cache is pinned to data/.hf_cache/.
  • Normalization: one _norm_<name> per dataset in full_llm_pipeline.ipynb, dispatched via NORMALIZERS. Produces a uniform [{role, content}, …] turn list.
  • Tokenizer: byte-level BPE trained from scratch (32K vocab dev / 128256 target).
  • Model: plain-PyTorch decoder-only Transformer with RMSNorm, RoPE, GQA, SwiGLU MoE (top-k routing + Switch-style load-balancing aux loss). Two configs — a 4-layer dev config and a 40-layer / 70B-param reference config.
  • Training: AdamW + CosineAnnealingLR, bf16 AMP on GPU, gradient clipping at 1.0. Distributed training path assumes FSDP or DeepSpeed ZeRO-3 on 8× A100-80GB; the dev config runs single-GPU or CPU.
  • Inference: chat.py loads a checkpoint folder and runs an interactive chat loop using the same KV-cache code the training loop builds.
  • Docs: this site, built from docs/ as a VitePress project and deployed by Cloudflare Pages. All docs tooling is scoped to docs/; see docs/deployment/cloudflare.md.

What's fixed in this release ​

  • Wikipedia download no longer OOMs on unlimited runs: wikimedia/wikipedia now uses streaming=False, and the download loop saves non-streaming datasets via memory-mapped Arrow rather than accumulating all rows into a Python list. See download_data.py:102 and download_data.py:173.

What's not in this release (called out so the next agent doesn't rediscover) ​

  • No automated test suite, no lint/typecheck CI. Docs workflow is the only CI.
  • The 70B reference config is a design target, not a measured result; no full pretraining run has been executed in this repo.
  • requirements.txt is hand-mirrored from pyproject.toml rather than uv pip compiled — the two can drift.
  • No evaluation harness (e.g. MMLU, lm-evaluation-harness wiring) is shipped.

Upstream references ​

Full technique-by-technique provenance with papers and reference code is in docs/techniques.md.

The cross-project agent rules are vendored as a git submodule at AGENTS.md/, tracking https://github.com/Trance-0/AGENTS.md.

Incremental revisions ​

Each incremental release has its own file:

  • v0.1.1 — pre-flight observability in the pipeline notebook.
  • v0.1.2 — progress visibility and watchdog-safe heartbeats.
  • v0.1.3 — resumable on-disk cache for tokenised datasets.
  • v0.1.4 — memory-mapped binary shards for tokenised datasets.