v0.1.4 — Memory-mapped token shards

Status: superseded by v0.2.0.

Version source: historical pyproject.toml state (version = "0.1.4").

---

What changed

Non-architectural storage optimisation for the tokenised data pipeline. v0.1.3 materialised the full tokenised corpus as Python objects before saving; v0.1.4 streams tokens directly to on-disk binary shards and memory-maps them at training time.

External interfaces stayed unchanged: PretrainDataset(...), SFTDataset(...), DPODataset(...), and the sample dictionaries returned by __getitem__ keep the same shape.

Storage layout

File	Dtype	Shape	Written by
pretrain_tokens.bin	int32	(total_tokens,) flat	PretrainDataset
pretrain_tokens.meta.json	JSON	metadata	PretrainDataset
sft_ids.bin	int32	(n_samples, max_len+1)	SFTDataset
sft_mask.bin	int8	(n_samples, max_len+1)	SFTDataset
sft.meta.json	JSON	metadata	SFTDataset
dpo_ids.bin	int32	(n_pairs, 2*(max_len+1))	DPODataset
dpo_mask.bin	int8	(n_pairs, 2*(max_len+1))	DPODataset
dpo.meta.json	JSON	metadata	DPODataset

Details

• Bounded construction memory. Pretrain flushes every 2,000 docs; SFT and DPO flush every 500 conversations. Peak construction memory is O(write batch), not O(corpus).
• Resume semantics. Metadata records processed indices and committed byte counts. On resume, shard files are truncated back to the last committed byte to drop any uncommitted tail writes.
• Dtype choices. int32 covers token IDs; int8 masks cut mask storage 4x vs int32. Conversion to int64 happens per sample in __getitem__.
• Migration from v0.1.3. Old .pt cache files are ignored in place. rm data/cache/*.pt is safe after v0.1.4 shards exist.
• Determinism caveat. DPO rejected-side random.shuffle(content) is still not seeded per conversation, so strict partial-rebuild determinism would require an index-derived seed.