Lavender-2

Appearance

v0.2.4 — Timestamped per-run checkpoints + milestone Bark pushes ​

Status: current baseline. Supersedes v0.2.3 as the default change_tokenizer state.

Version source: pyproject.toml (version = "0.2.4").

What landed ​

Three independent but complementary changes, driven by the owner's v0.2.4 brief:

  1. Checkpoint layout rewrite — run-subdirectory, timestamp filenames, latest-3 retention, no byte-identical _copy redundancy.
  2. Milestone Bark pushes — every stage boundary (run start, data loaded, tokeniser ready, each dataset built, each epoch save, final save, load) fires a silent Bark notification. The previous v0.2.3 wiring only pinged on swap-above-50%, which is why the owner saw no pushes during a run.
  3. Training-log upgrade — CheckpointManager.save now writes a SAVE <file> @ HH:MM:SS line into the per-run log, and a RETAIN dropped N old ckpt(s) line after each retention pass. The log now makes "when was the last save?" a one-grep answer.

Checkpoint layout (before → after) ​

Before (v0.2.3, retired) ​

text
checkpoints/
├── ckpt_pretrain_0001.pt          # 637 MB primary
├── ckpt_pretrain_0001_copy1.pt    # 637 MB byte-identical copy
├── ckpt_pretrain_0001_copy2.pt    # 637 MB byte-identical copy
├── ckpt_pretrain_0001_copy3.pt    # 637 MB byte-identical copy
├── ckpt_pretrain_0002.pt
├── ckpt_pretrain_0002_copy1.pt
├── ...                             # ad infinitum
└── tokenizer.json

Two long-standing complaints, both fixed in v0.2.4:

  • Numbered filenames (_0001, _0002, …) carried no timestamp — reading the directory required cross-referencing mtimes with process logs.
  • _copy1/2/3 suffixes suggested redundancy, but the copies were byte-identical on the same filesystem. They did not protect against disk failure (the filesystem already handles that) and cost 4× the disk.

After (v0.2.4, current) ​

text
checkpoints/
└── run-2026-04-24_16-08-45/        # one subdir per run, human-readable
    ├── pretrain-16-08-47-0003.pt   # latest 3 kept (by mtime)
    ├── pretrain-16-08-48-0004.pt
    ├── pretrain-16-08-49-0005.pt   # <stage>-<HH-MM-SS>-<counter>
    └── tokenizer.json
  • Run subdirectory: run-<YYYY-MM-DD_HH-MM-SS> fixes a start timestamp per run. Older runs stay browsable at their own subdirectories without cluttering the latest save.
  • Filenames: <stage>-<HH-MM-SS>-<counter>.pt. Stage is the training phase (pretrain / sft / dpo / final); time is the save clock; counter (4 digits, zero-padded) breaks ties when multiple saves land in the same second and preserves unambiguous order.
  • Retention: CheckpointManager._enforce_retention sorts by mtime and deletes everything older than the latest :data:CKPT_RETENTION (default 3) after each save. Smoke test verifies 5 saves → 3 kept, the 2 oldest removed.
  • No _copy suffix. One primary per save. The tokenizer is still written alongside so inference can reload the exact encoding from the same run directory.

Legacy cleanup (one-time) ​

The pre-v0.2.4 checkpoints/ held 105 files totalling 65 GB (26 numbered checkpoints × primary + three copies each). This round moved 101 of them to the system trash via gio trash, keeping the 3 latest primaries (ckpt_pretrain_0024.pt, _0025.pt, _0026.pt) plus tokenizer.json. Post-cleanup size: 1.9 GB. The kept files still load via CheckpointManager.load — it's a path-only API with no naming-pattern dependency.

Milestone Bark pushes ​

New helper in lavender_2/notify.py:

python
def notify_milestone(message, *, silent=True, prefix="[lavender-2] ", root="."):
    """Fire a silent Bark push for a training-pipeline milestone."""
    return send_bark(f"{prefix}{message}", silent=silent, root=root)

Wired into the notebook at every stage boundary:

StagePush body
Notebook run start[lavender-2] ipynb run started (device=cuda)
Data load complete[lavender-2] data loaded: 1,123,456 convs from 12 sources
Upsample complete[lavender-2] upsample complete: 2,345,678 texts
Tokeniser ready[lavender-2] tokenizer ready (200,023 vocab)
Pretrain dataset ready[lavender-2] pretrain tokenisation done: 9,876,543 chunks
SFT dataset ready[lavender-2] SFT tokenisation done: 1,234,567 samples
DPO dataset ready[lavender-2] DPO tokenisation done: 1,234,567 pairs
Checkpoint saved[lavender-2] checkpoint saved: pretrain-04-30-11-0002.pt, epoch 2/3, CE 1.234, ETA 3.2h
Checkpoint loaded[lavender-2] checkpoint loaded: pretrain-04-30-11-0002.pt
Final save[lavender-2] final checkpoint: final-05-12-22-0007.pt

Pushes are level=passive (silent) so they don't interrupt anything. A missing *.bark.env is a no-op — the notebook doesn't crash because a device key wasn't set.

ETA in save messages ​

CheckpointManager.save gained a shared _milestone_extra_bits helper that packs **extra into the push body when it carries any of:

  • epoch / epochs → epoch 2/3
  • ce_loss / sft_loss / dpo_loss → CE 1.234 etc.
  • remaining_hours → ETA 3.2h

The training loops feed remaining_hours from a new lavender_2.training._eta_hours(started, done, total) that linear- extrapolates from wall-clock elapsed. Crude but more useful than nothing for a one-number push body.

Log-file upgrade ​

The 1 KB per-run log (v0.2.3) gets three new line types:

  • CKPT run-dir checkpoints/run-2026-04-24_…/ — one per session.
  • SAVE <filename> @ HH:MM:SS — one per checkpoint save; carries the "time of last save" the owner asked for.
  • RETAIN dropped N old ckpt(s) — one per retention pass, when anything got deleted.

Terminal output unchanged — log-file additions are silent.

API shifts ​

These are the public-API changes worth knowing about when reading old code or scripting against :class:CheckpointManager:

Before (v0.2.3)After (v0.2.4)
checkpoints/ckpt_<stage>_NNNN.ptcheckpoints/run-<ts>/<stage>-<HH-MM-SS>-NNNN.pt
_copy1.pt/_copy2.pt/_copy3.pt fallback in load()removed; path-only load()
n_copies=3 ctor argretention=3 ctor arg (different semantics!)
CheckpointManager.list_checkpoints(ckpt_dir) flatsupports both run-dir and root; union of nested + flat
No list_runsCheckpointManager.list_runs(ckpt_dir) returns run-subdirs
No logger fieldlogger=... optional, writes SAVE/RETAIN lines
No notify fieldnotify=True by default; milestones fire on save + load

Saved .pt payload shape is unchanged, so files written by v0.2.3 still load via the new load() — just pass the legacy path directly.

AGENTS.md submodule ​

Probed and fetched; submodule was already at upstream 51b8f27 (docs: require progress for long-running work). No pointer bump this round.

What's not in this release ​

  • Live Bark server still not tested. Same caveat as v0.2.3: notify_milestone is covered by import tests and the offline path, not by a real push against the owner's phone. Any failure prints one short line and is non-fatal.
  • ETA is linear extrapolation only. It does not account for changing batch times as training warms up or cooldown schedules. Good enough for a one-glance ETA on a phone; not a training- planner replacement.
  • Retention is per-run. A box running many small back-to-back experiments accumulates one run-subdir each. Sweep them manually when disk gets tight; a prune_runs(keep=N) helper is deliberately out of scope this round.

Upstream references ​