Lavender-2

Appearance

docs/AGENTS.md — Lavender-2 project-specific agent rules ​

These rules extend and override the canonical agent contract at AGENTS.md/AGENTS.md for the Lavender-2 repo. Read the root AGENTS.md/AGENTS.md first, then read this file. Any rule here wins when the two disagree.

Scope: anything specific to how Lavender-2 is built, versioned, or executed locally. Anything generic (commit hygiene, root-dir hygiene, secrets, size limits) stays in the root AGENTS.md and is not duplicated here.

1. Development environment: uv only ​

The project is managed with uv — the source of truth is pyproject.toml and uv.lock. The repo's own virtualenv lives at .venv/ (gitignored) and is built by uv sync.

Anaconda (base env or any conda env) must not be used to run this project's code. The two envs have different Python versions, different site-packages, and installing into the wrong one has already caused import failures for open_mythos and the OpenMythos git build.

1.1 Pre-flight every session ​

Before running any python, pip, jupyter, or notebook kernel in this repo, check whether conda is active:

bash
# If CONDA_DEFAULT_ENV is set, conda has auto-activated base (or another
# env). That will shadow the uv venv on PATH.
echo "conda: ${CONDA_DEFAULT_ENV:-not active}"
which python  # should be .../Lavender-2/.venv/bin/python3, not anaconda3/

If conda is active:

bash
conda deactivate                              # for the current shell
conda config --set auto_activate_base false   # durable fix for new shells

Neither command needs sudo. The durable change writes to ~/.condarc; it does not remove conda — you can still conda activate <env> manually when you actually want it.

1.2 Running commands ​

Always prefix with uv run (or activate the .venv once per session):

bash
uv sync                                 # install / update deps from uv.lock
uv run python download_data.py --help   # any Python script
uv run jupyter lab                      # JupyterLab in the uv env
uv add <package>                        # add a new dep (updates pyproject + uv.lock)
uv remove <package>                     # remove a dep

Never run pip install at the host / conda level for this repo's deps. If a dep needs to be added, it goes through uv add so pyproject.toml and uv.lock update atomically.

1.3 Jupyter kernel ​

The notebook (full_llm_pipeline.ipynb) must run against the uv venv, not the conda kernel. Register a kernelspec once per machine:

bash
uv run python -m ipykernel install --user \
  --name lavender-uv --display-name "Lavender (uv .venv)"

Then, in VSCode / JupyterLab, select "Lavender (uv .venv)" as the notebook kernel. Verify with !which python in a cell — it must point at <repo>/.venv/bin/python3.

1.4 Keeping requirements.txt in sync ​

requirements.txt exists for environments that cannot run uv sync (CI containers, remote training boxes with only pip). It is derived from uv.lock, not hand-maintained. When uv.lock changes, regenerate:

bash
uv export --format requirements-txt --no-hashes --no-emit-project \
  -o requirements.txt

This closes the drift flagged by docs/LLM_CHECK.md §1.3 #8. Ship the requirements.txt update in the same commit as the pyproject.toml / uv.lock change that caused it.

2. Architecture versioning ​

2.1 Baseline ​

The state on main as of the 0.1.0 release is the v0 baseline. Everything the current code does (data registry, byte-level BPE tokenizer, OpenMythos backbone, training loop, VitePress docs) is v0 by definition. Do not edit the baseline's behavior in place — any code-behavior change rides a version bump per §2.2.

2.2 Version scheme ​

0.MAJOR.MINOR while the project is pre-1.0:

  • Bump the MAJOR (e.g. 0.1.0 → 0.2.0) on every architectural baseline change. Examples from this repo's real history:
    • Swap the model family (hand-rolled MoE Transformer → OpenMythos) — commit 88fa6bc.
    • Swap the tokenizer algorithm (BPE → SentencePiece, or byte-level → char-level).
    • Swap the training objective or loss (next-token only → next-token + DPO, add RLHF stage, etc.).
    • Swap the data pipeline shape (streaming dialog → pretokenized binary shards, add a retrieval index).
    • Swap the runtime target (single-node dev-box → FSDP multi-node, switch precision class, add distillation stage).
  • Bump the MINOR (e.g. 0.1.0 → 0.1.1) on non-architectural changes: bug fixes, dataset additions to an existing registry shape, docs/infra work, dependency bumps, knob retunes inside the same architecture family.

pyproject.toml version = "..." is the single source of truth. Update it in the same commit as the architecture change.

2.3 Per-version release note ​

Every MAJOR and MINOR bump ships a release-note file at docs/versions/<semver>.md (e.g. docs/versions/0.2.0.md, docs/versions/0.2.1.md). Use docs/versions/0.1.0.md for baseline release shape and docs/versions/0.2.1.md for incremental change shape.

Minimum sections for a MAJOR baseline:

  • # v0.X.0 — <one-line label> — heading.
  • **Status**: active / superseded / deprecated.
  • **Version source**: link to pyproject.toml line.
  • ## What's shipped: bullet list of baselines (data ingest, normalization, tokenizer, model, training loop, inference).
  • ## What's fixed in this release: what broke or was sub-optimal in the prior MAJOR and is now resolved, with file:line references.
  • ## What's not in this release: known gaps the next agent should not rediscover (no test suite, untrained config, missing eval harness, etc.).
  • ## Upstream references: pointer to docs/techniques.md for paper-level provenance.

MINOR bumps get their own file too. Keep them concise: status, version source, what changed, and verification / migration notes when relevant. Do not put incremental change logs in docs/LLM_CHECK.md; that file is a checklist only. Non-release project history belongs in docs/versions/project-history.md.

2.4 Notebook snapshot policy ​

full_llm_pipeline.ipynb is the live working notebook. Today we keep exactly one copy of it at repo root — no parallel *_v0.ipynb, *_old.ipynb, or checkpoint files. That keeps diffs readable and the repo small.

When cutting a MAJOR bump (0.1.x → 0.2.0, etc.) that touches the notebook, snapshot the outgoing version before applying the new changes, so the prior architecture stays recoverable without digging through git history:

bash
# Run at the start of a MAJOR-bump round, while the notebook still
# reflects the outgoing major.
mkdir -p docs/versions/snapshots/0.1.x
cp full_llm_pipeline.ipynb docs/versions/snapshots/0.1.x/full_llm_pipeline.ipynb
git add docs/versions/snapshots/0.1.x/full_llm_pipeline.ipynb
# then edit the live notebook for 0.2.0 and commit both in the MAJOR-
# bump commit.

Rules for the snapshot:

  • MAJOR bumps only. MINOR bumps amend the live notebook in place — don't create snapshots for ergonomics tweaks.
  • Outgoing major, not the new one. Snapshot 0.1.x while editing 0.2.0 so readers can open the preserved 0.1.x copy and see what the prior architecture actually ran.
  • Strip notebook outputs before committing the snapshot if any cell output contains a dataset row, credential, or user PII (WildChat / lmsys / arena), per docs/LLM_CHECK.md §1.2 #7. Use jupyter nbconvert --clear-output --inplace on the snapshot before git add.
  • One snapshot per outgoing MAJOR, not per MINOR. The 0.1.x folder holds the final 0.1.<last> notebook; don't stack intermediate MINOR snapshots.
  • Snapshots are reference material. Don't import from them, don't wire CI against them, don't re-execute them. They exist so a future reader can recover what the prior major looked like.

2.5 docs/training/vN/ vs docs/versions/0.X.0.md ​

  • docs/training/v0/ is the pedagogical walkthrough of the v0-era architecture (tokenizer, MoE, training loop, etc.). It documents how the architecture works for someone learning.
  • docs/versions/0.X.Y.md is the changelog entry — what this specific release contains vs the prior one.

When an architecture baseline change happens, you typically need both: a new docs/training/v<MAJOR>/ tree if the teaching content diverges, and a new docs/versions/0.<MAJOR>.0.md entry.

Only fork docs/training/ into a new major when the old walkthrough no longer describes what the code does. Minor edits to existing pages are fine when the architecture is stable.

3. Host memory budget (128 GB ceiling) ​

The working training box Lavender-2 targets has a hard 128 GB of host RAM and a pool of GPU VRAM (size varies per machine). Any step of the pipeline — normalisation, tokenisation, cache construction, training, checkpointing, inference — must fit inside that ceiling. A run that exceeds 128 GB of process RSS is treated the same as an OOM crash for review purposes: it blocks a merge until fixed.

3.1 What counts against the ceiling ​

  • RSS of the Python process that runs the notebook / CLI — this is the number psutil.Process().memory_info().rss reports. Includes Python objects, PyTorch tensors on CPU, and mmap'd regions that have been touched (page cache that hasn't been pulled in is fine).
  • GPU VRAM across all devices — tracked separately but governed by the same rule: the pipeline must not exceed available VRAM on the configured device. Use torch.cuda.memory_allocated() and torch.cuda.max_memory_allocated() to introspect.

3.2 Concrete design invariants ​

  • Tokenised datasets go on disk in binary shards (data/cache/*.bin) and are mapped via np.memmap (v0.1.4+). They must not be materialised as a Python list of tensors during training. Per-iteration reads are the only allowed path.
  • train_data (normalised [{role, content}, ...] list) and train_texts (upsampled flat strings) are the largest in-RAM structures. Their size scales with SAMPLE_LIMIT. On a real corpus, expect ~1–2 GB per 1M conversations. Check that the intended SAMPLE_LIMIT projected against this ratio stays comfortably under 128 GB; otherwise cap the sample limit or switch to a streaming normaliser.
  • BPE / tokenizer training (retired in v0.2.0) used 5–8× corpus bytes in RAM. Any future trainer that touches RAM on the corpus scale must document a peak-RAM figure in its release-note file.
  • Model instantiation for PRODUCTION_32B_CFG allocates ~60 GB+ of weight tensors on its own — it must not run on a box without matching VRAM/RAM. Guard it behind the config toggle (MODEL_CFG = PRODUCTION_32B_CFG is commented out for a reason).
  • Snapshot paths like docs/versions/snapshots/0.1.x/ are outputs-stripped, so they contribute ~1 MB each regardless of training state.

3.3 What the notebook / package must do ​

  • lavender_2.utils.log_memory(label) is the canonical probe; it prints RSS / available RAM / VRAM (if CUDA) via tqdm.write. Call it at stage boundaries: after data load, after tokenisation, before and after each training epoch, before and after each checkpoint save.
  • lavender_2.utils.check_memory_budget(ceiling_gb=128.0) warns via tqdm.write when the process RSS crosses 90% of the ceiling and raises MemoryBudgetExceeded when it crosses 100%. Callers decide whether to catch or let it kill the run.
  • Every per-round release note (docs/versions/<semver>.md) for changes to data / training / inference code must state a peak-RAM claim and, where relevant, a peak-VRAM claim, with the measurement method (observed vs. computed).

3.4 When the ceiling has to move ​

If a real training run legitimately needs more than 128 GB, open the conversation before the work lands, update this section in the same commit that changes the ceiling, and record the new target in the release note. No silent raises.

4. docs/TODO.md ​

Per root AGENTS.md §2.5, owner-visible multi-round work lives in docs/TODO.md. Keep it lightweight: one line per item, grouped by status (Next / Later / Done). Per-round agent scratch does not go in docs/TODO.md — that goes in the agent's own live task tool and never gets committed.

5. Cross-references ​