v0.2.1 — Long-running process visibility
Status: active package release on the v0.2 architecture line.
Version source: pyproject.toml (version = "0.2.1").
What changed
Non-architectural ergonomics added to full_llm_pipeline.ipynb. The tokenizer, model family, training objectives, dataset registry, and checkpoint formats stay on the v0.2.0 architecture line.
- Quiet pre-flight helpers stay visible. Dataset disk-size scans and
load_dataset/load_from_diskcalls are wrapped inHeartbeat(...), so blocked metadata or local Arrow loads emit keep-alive messages instead of going silent. - Large list scans show progress. Text fingerprinting,
train_textscharacter counting, and effective-mix counting usetqdmwhere a full-corpus run can spend noticeable wall time. - Model setup and restore have heartbeat coverage. Model construction, parameter counting, checkpoint model rebuild, and checkpoint weight loading emit keep-alive messages if production configs run long.
- Speculative decoding reports progress. The demo adds a token progress bar plus heartbeat status for larger
max_new_tokensruns.
Verification
- Notebook JSON validated with
jq empty full_llm_pipeline.ipynb. - Notebook code cells compiled with
uv run python. uv run python -c "import download_data"passed.