v0.1.3 — Resumable tokenisation cache

Status: superseded by v0.1.4.

Version source: historical pyproject.toml state (version = "0.1.3").

---

What changed

Non-architectural data-pipeline durability. A prior training attempt died mid-tokenisation and lost all tokenisation work on restart; this release persists each tokenisation stage to data/cache/ every 30 minutes of wall time.

• Cache identity helpers. _tokenizer_sha(tokenizer) invalidates downstream caches whenever tokenizer JSON changes. _texts_fingerprint(texts) and _raw_ds_fingerprint(raw_ds) catch input-content changes.
• Atomic cache IO. _load_cache / _save_cache use tmp-rename writes around torch.save / torch.load, with Heartbeat coverage for multi-GB IO.
• Tokenizer cache. data/cache/tokenizer.json is reused when its vocab size matches VOCAB_SIZE; retraining changes tok_sha and invalidates downstream caches.
• Pretrain cache. pretrain_all_ids.pt stores the concatenated token stream incrementally, and pretrain_samples.pt stores the final chunked samples.
• SFT cache. sft_samples.pt stores resumable (ids, mask) samples keyed by tokenizer SHA, max_len, limit, and raw data fingerprint.
• DPO cache. dpo_pairs.pt stores resumable (ci, cm, ri, rm) preference tuples with the same keying scheme.
• Graceful degradation. Disk-full or permission errors are logged; tokenisation continues in memory if caching is unavailable.

Notes

data/cache/ inherits the existing data/ gitignore entry, so cache files stay out of commits.