Lavender-2

Appearance

LLM_CHECK.md — Lavender-2 end-of-round checklist ​

Derived from the canonical template at AGENTS.md/docs/LLM_CHECK.md. Read that file for the full rationale; the sections below are the Lavender-2-specific items the agent must confirm before declaring a round done.

1. Common mistakes seen in prior rounds ​

Scan this before starting and again before claiming done.

1.1 Dataset ingest ​

  1. Using streaming + rows.append(sample) + Dataset.from_list(rows) on a multi-million-row dataset (Wikipedia, WildChat, Soda). This accumulates the full dataset into RAM and OOMs on unlimited runs. Use streaming=False for datasets larger than ~1M rows.
  2. Forgetting to redirect the HF cache. The script pins HF_HOME and HF_DATASETS_CACHE to data/.hf_cache/; removing that makes downloads pollute ~/.cache/huggingface/.
  3. Changing the on-disk column schema of a dataset without updating its _norm_<name> function in full_llm_pipeline.ipynb. The notebook reads by column name; a silent rename breaks training.
  4. Running python download_data.py with no --limit and no streaming=False flag on large datasets. Check the registry before adding a new entry.

1.2 Secrets and git hygiene ​

  1. Committing hf-token.txt, or echoing its contents into logs, notebook output cells, or commit messages.
  2. Committing anything under data/, checkpoints/, or .venv/.
  3. Leaving notebook output cells that contain dataset rows with user PII (WildChat / lmsys_chat / arena conversations) committed.

1.3 Dependency drift ​

  1. Editing pyproject.toml without regenerating requirements.txt and uv.lock. The two files disagree silently until a new clone fails.
  2. Pinning pyarrow<17 — breaks zstd-compressed parquet reads used by several datasets.

1.4 Submodule handling ​

  1. Editing files inside AGENTS.md/ from this repo. The submodule tracks upstream; local edits get clobbered on git submodule update and confuse review.
  2. Committing a stale submodule pointer. After intentionally bumping the submodule, commit the pointer change explicitly.

1.5 Long-running process visibility ​

  1. Adding or changing work that can run longer than 1 minute without a progress bar, heartbeat, or other visible progress message. Use tqdm for Python loops where possible; use a tuned verbose / heartbeat message for blocking calls such as model load/save, dataset metadata fetches, or artifact copies.
  2. Adding or changing work that can run longer than 1 hour without a durable checkpoint / resume path. Model training, tokenisation, dataset processing, and large artifact generation must not lose all progress on interruption.
  3. Leaving any code path silent for 5 minutes. Across the owner's projects, such runs may be terminated as hung.

2. Round-end checklist ​

Run in order. Every item is either confirmed or called out as skipped-with-reason.

2.1 Truthfulness ​

  • [ ] Every command I claimed passed was actually executed in this environment.
  • [ ] Commands I did not run are called out with the reason (no GPU, no HF token, dataset too large to stream here, etc.).
  • [ ] Any bug fix has a named root-cause line (file.py:line), not just "symptom went away".

2.2 Docs and handoff ​

  • [ ] README.md still matches reality (one-liner blurb is fine; don't bloat it into a runbook — that's what CLAUDE.md is for).
  • [ ] CLAUDE.md reflects any change to repo layout, stack, build commands, or dataset-download invariants.
  • [ ] If a new dataset was added to download_data.py, its _norm_<name> and NORMALIZERS entry exist in full_llm_pipeline.ipynb, and the weights table in the notebook lists it.
  • [ ] Round logs were not added to this file. Versioned / incremental change documentation belongs in docs/versions/<semver>.md (for example 0.1.1.md, 0.1.2.md, 0.2.1.md); non-release project history belongs in docs/versions/project-history.md.

2.3 Code hygiene ​

  • [ ] Removed features have no dangling imports or references (rg across *.py and the notebook).
  • [ ] Unused deps removed from pyproject.toml, and requirements.txt and uv.lock regenerated.
  • [ ] .gitignore still covers data/, checkpoints/, hf-token.txt, .venv/, __pycache__/, .ipynb_checkpoints/.
  • [ ] New comments explain a non-obvious why, not a what.

2.4 Build / test / lint ​

No CI or test suite exists yet. Default bar for this repo:

  • [ ] python -c "import download_data" — at minimum the script still imports cleanly.
  • [ ] python download_data.py --only <changed-dataset> --limit 10 — smoke-test any registry change end-to-end; confirm the saved directory loads via load_from_disk.
  • [ ] If the notebook was edited: re-run the affected cells top-to-bottom in a clean kernel, or state explicitly that it was not executed and why.
  • [ ] Any process added or touched this round that may run longer than 1 minute emits progress at least once per minute; any process that may run longer than 1 hour has a checkpoint / resume path.

2.5 Secrets and infra ​

  • [ ] git status shows no hf-token.txt, no data/**, no checkpoints/**, no .venv/** staged.
  • [ ] No real HF token, email, or user-PII dataset row is pasted into committed files.

2.6 Multi-agent / submodule safety ​

  • [ ] Files inside AGENTS.md/ are unchanged (or the bump is intentional and the pointer commit is separate from project code changes).
  • [ ] No unrelated user changes were reverted, stashed, or force-pushed.

2.7 Handoff ​

  • [ ] Commit message drafted (action-oriented, scoped).
  • [ ] Push command prepared, or noted that the round should not be pushed yet.
  • [ ] CLAUDE.md updated if any rule or invariant changed during the round.