Lavender-2

Appearance

v0.2.3 — Bark notifications, swap watchdog, per-run training log ​

Status: current baseline. Supersedes v0.2.2 as the default change_tokenizer / main state.

Version source: pyproject.toml (version = "0.2.3").

What landed ​

Two new modules and a hook into the existing :class:~lavender_2.utils.Heartbeat:

  • lavender_2/notify.py — Bark push-notification client + :class:SwapWatchdog background daemon.
  • lavender_2/logs.py — per-run :class:TrainingLogger with a hard 1 KB file-size cap.
  • lavender_2/utils.py — Heartbeat now records START / END markers to the installed logger (per-ping "still running" lines are intentionally dropped from the log to keep the byte budget; terminal output is unchanged).

All three are wired in the notebook's imports cell: a TrainingLogger is created at run start (seeded with the device + package version header), Heartbeat.set_logger(...) installs it globally, and a SwapWatchdog is started with the default cadence (10 min) and threshold (50%).

Notifications ​

  • Config files: *.bark.env at repo root. Each file is one-URL-template-per-line with $0 as the message placeholder. All *.env files are already covered by the repo .gitignore (*.env on line 37), so real device keys never reach a commit.
  • A .bark.env.example template is provided at repo root so new contributors see the expected shape without any real key.
  • lavender_2.notify.send_bark(message) URL-encodes the message, substitutes it into every template, and appends ?level=passive&isArchive=0 — silent, non-archived.
  • Network failures are swallowed (best-effort): each HTTP request gets a 5-second timeout, and an unreachable Bark server will not crash training. A short [bark] <file> failed: <ErrorClass> line is printed so the failure is visible but harmless.

Swap watchdog ​

  • Polls psutil.swap_memory() every 10 minutes; if used / total >= 0.5, fires a silent Bark push: [lavender-2] swap 54% full (16.3/32.0 GB) — consider intervening.
  • First poll runs synchronously on :meth:SwapWatchdog.start, so a host that's already paging gets the push immediately rather than 10 minutes later.
  • Threshold + cadence are constructor parameters; the notebook uses the defaults from the v0.2.3 brief.
  • The watchdog runs on a daemon thread, so it dies silently when the Python process exits — no need to .stop() at the end of the notebook.
  • Error guard: any exception inside the poll loop is caught and printed as [swap watchdog] error: <ErrorClass>: <msg> so a transient psutil issue never crashes training.

Training log ​

  • File per run at logs/run-<YYYYMMDD-HHMMSS>.log. Directory is .gitignore'd (logs/ on line 23 — pre-existing, verified).
  • Hard cap: 1024 bytes per run, no matter how long training lasts. On overflow, :meth:TrainingLogger._compact rewrites the file with head + "# [...compacted N lines, round M]" + tail. Verified: 100 log writes of ~80 bytes each produce a 432-byte file with 12 compaction rounds; the last four entries are preserved verbatim.
  • Heartbeat pings are dropped from the log — the 60-second "still running" spam would blow the budget on any multi-hour run. Only the START / END of each Heartbeat block is persisted. A 5h training epoch therefore contributes 2 lines (~70 bytes) to the log regardless of how many pings landed on the terminal.
  • Terminal output is untouched. tqdm bars, Heartbeat pings, print()s behave exactly as in v0.2.2. The user's brief was explicit: "Do not reduce anything for terminal logs."

Log entry shape ​

text
# lavender-2 run log 2026-04-24 00:41:41
# device=cuda lavender_2=v0.2.3
00:41:41 RUN start  device=cuda
00:41:52 START Building model
00:42:05 END   Building model 13s
00:45:18 BARK swap 51% sent ok=1 fail=0
00:45:18 START Pretrain 1/3
04:52:10 END   Pretrain 1/3 14812s

Every line is ≤80 characters; the full shape fits comfortably in the 1 KB budget for a typical pretrain → SFT → DPO run.

Notebook wiring ​

Imports cell now does, right after set_seed() / get_device():

python
training_logger = TrainingLogger(
    header=f"device={device} lavender_2=v{lv.__version__}"
)
Heartbeat.set_logger(training_logger)
training_logger.log(f"RUN start  device={device}")

swap_watchdog = SwapWatchdog(
    interval_sec=600, threshold=0.5, logger=training_logger
).start()

No other cell changed — every Heartbeat block inside :mod:lavender_2.training, :mod:lavender_2.checkpoints, :mod:lavender_2.model, :mod:lavender_2.inference picks up the logger for free.

What's not in this release ​

  • Bark notifications haven't been verified against a live Bark server in this round — send_bark is covered by import tests only. Any failure prints one short line and continues, so a broken config is visible but non-fatal.
  • The swap-watchdog threshold isn't tunable from the notebook (it's set in the imports cell constant). A follow-up could expose it as a module-level default.
  • Heartbeat ping dedup is binary (drop them all). A future refinement could keep one ping-per-hour for very long runs; the current behavior is the simplest thing that satisfies the budget.

Upstream references ​