Training pipeline — v0

This section walks through the v0 training pipeline one stage at a time. The intended reader is an undergraduate who has taken a basic machine-learning course (linear algebra, gradient descent, forward and backward passes, cross-entropy loss) and wants to follow a modern chat-LM recipe end-to-end without skipping over the engineering pieces.

What the pipeline does

Given nothing but a Python environment, the pipeline produces a trained, chat-capable language model in four conceptual stages:

1. Get text — download ~12 HuggingFace datasets covering dialogue, reasoning traces, and encyclopedic content. The Wikipedia entry is capped at 32 GiB of raw text payload so a future corpus refresh can't silently blow past the RAM and training budget downstream.
2. Map text to integers — train a byte-level BPE tokenizer on the normalized corpus.
3. Build and train the model — an OpenMythos recurrent-depth transformer (prelude → single recurrent block looped T times with ACT early-exit → coda) with RMSNorm + RoPE
   • MLA or GQA attention and a SwiGLU MoE with routed plus shared experts. Trained with AdamW and cosine learning-rate schedule in bf16 mixed precision.
4. Generate text — load a checkpoint and run an interactive chat loop that reuses the training model's attention path with a KV cache for speed.

System shape

HuggingFace Hub ─┐
                 ├─> download_data.py ─> data/<dataset>/  (save_to_disk, arrow)
                 │                        │
                 │                        ▼
                 │          full_llm_pipeline.ipynb
                 │            ├── normalize to chat-turn schema
                 │            ├── train BPE tokenizer (tokenizers lib)
                 │            ├── build OpenMythos (recurrent-depth MoE transformer)
                 │            ├── pretrain (AdamW + cosine LR, bf16 AMP)
                 │            ├── finetune (same loop, smaller LR)
                 │            └── save to checkpoints/
                 │                        │
                 │                        ▼
                 └─────────────────────> chat.py  (interactive inference)

Stage-by-stage

1. Data ingest & normalization — which datasets, how they're downloaded, how different input formats are unified into chat turns.
2. Tokenizer — why text has to become integer IDs and how byte-level BPE does that losslessly.
3. Architecture — the OpenMythos recurrent-depth transformer: prelude, single recurrent block looped with LTI-stable injection and ACT halting, coda; RMSNorm pre-norm, RoPE, and either MLA (latent-KV) or GQA attention.
4. SwiGLU & Mixture-of-Experts — the feed-forward path is a sparse MoE with n_experts_per_tok routed experts plus n_shared_experts always-on shared experts, load-balanced by the router softmax.
5. Training loop — optimizer, learning-rate schedule, mixed precision, gradient clipping.
6. Checkpointing — triple-redundant saving so a crash doesn't kill a long run.
7. Inference — generating tokens autoregressively with a KV cache, used by chat.py.

Two shipped configs

Field	Dev	32B reference
d_model (dim)	256	5120
n_heads / n_kv_heads	8 / 4	32 / 8
prelude_layers / coda_layers	2 / 2	2 / 2
max_loop_iters	4	28
d_ff (expert_dim)	512	6144
num_experts / n_shared_experts / top_k	4 / 1 / 2	64 / 2 / 4
attn_type	gqa	mla
vocab_size	32,000	128,256
max_seq_len	256	8192

The dev config is small enough to smoke-test on a single GPU or CPU (~13M params). The 32B reference config is a design target (~32B total, ~3.7B active per token); v0.1.0 has not trained it end-to-end. See ../../versions/0.1.0 for what's actually been run.