v0.1.1 — Pre-flight observability

Status: superseded by v0.1.2.

Version source: historical pyproject.toml state (version = "0.1.1").

---

What changed

Non-architectural ergonomics added to full_llm_pipeline.ipynb. The model family, tokenizer algorithm, training objective, and runtime target all stay at the v0.1 architecture baseline.

• Dataset loader pre-flight. Before the 12-source ingest loop, the notebook probes each dataset with datasets.load_dataset_builder(...).info and prints per-source download / on-disk sizes, summed known totals, and estimated peak RAM for train_data.
• Host capacity check. When psutil is available, the notebook reports available RAM and free disk under DATA_DIR, with warnings when projected download or RAM use exceeds local capacity.
• Tokenizer RAM estimate. The BPE tokenizer cell computes corpus UTF-8 bytes, prints an empirical 5-8x peak-RAM band for the HuggingFace trainer, and explains why HF tokenizers is CPU-only.
• Progress visibility. The tokenizer trainer is fed through tqdm(train_texts, desc="Feeding corpus", unit="doc") with length=len(train_texts) so Rust-side progress bars have accurate ETAs.

Documentation

• docs/training/v0/tokenizer.md and docs/training/v0/data.md gained matching explanatory prose.
• docs/AGENTS.md recorded the v0 baseline and the notebook snapshot policy for future MAJOR bumps.