From bare-metal microcontroller registers to large-scale LLM inference pipelines. We obsess over every microsecond, every cache miss, every memory access pattern.
$ perf stat -e cycles,instructions,cache-misses ./inference
# Performance counter stats:
42,891,234,567 cycles
98,234,567,890 instructions # 2.29 IPC
1,234,567 cache-misses # 0.001%
✓ 3.2x throughput improvement How to build a simulated marketplace where 10,000 LLM-powered agents shop, negotiate, write reviews, and ask questions — generating millions of realistic training examples. Agent architecture, environment design, persona systems, data extraction pipelines, quality scoring, and the full cost analysis for 1M synthetic examples.
How to build a training dataset for code LLMs from raw GitHub data — license filtering with SPDX headers and permissive-only policies, exact and near-deduplication with MinHash at function-level granularity, quality scoring with lint, cyclomatic complexity, and documentation ratio, language distribution strategies, and PII removal. Based on The Stack v2 methodology.
How DCLM and FineWeb-Edu transform raw web crawls into high-quality LLM training data. The exact heuristic filters, quality classifiers, deduplication algorithms, and the pipeline that converts 100TB of HTML into 1TB of clean tokens.
A deep dive into data mixing for LLM pretraining -- how to allocate 15T tokens across web text, code, math, and books. Covers empirical results from Llama 3, the 'code helps everything' finding, synthetic math data generation, curriculum learning schedules that shift proportions during training, and a complete implementation of a configurable multi-source data mixer with dynamic scheduling.
How to construct evaluation datasets that reliably measure model capability — benchmark design principles, contamination detection, the difference between memorization and understanding, and implementing a custom evaluation pipeline.
How to construct preference datasets for alignment training — human annotation protocols, AI-generated preferences, quality control, and the exact data format for DPO, KTO, and RLHF training. Includes a complete annotation pipeline implementation.
How to generate high-quality synthetic training data for LLMs — the Magpie technique for instruction generation, Nemotron-4's reward-model-guided synthesis, quality filtering, and building a complete pipeline from raw model outputs to training-ready datasets.
The evolution from RLHF to simpler alignment methods. Implementation of DPO, KTO, and ORPO loss functions — why each works, when to use each, and the exact training code.
A comprehensive technical comparison of parameter-efficient fine-tuning methods: LoRA's low-rank adaptation, DoRA's magnitude-direction decomposition, QLoRA's 4-bit quantized training, GaLore's gradient projection for full-weight updates, and LISA's random layer freezing. Includes memory budgets, quality benchmarks, and a decision matrix.