Understanding language models
from first principles.
"The Machine" is an educational and personal research project exploring how generative AI and neural language models work under the hood. Rather than merely consuming cloud APIs, it investigates tokenization, vocabulary construction, PyTorch neural architectures, and pattern learning directly.
“What happens inside an AI model when text is transformed into weights, vectors, and predictions?”
Modern developers are often taught to call external AI APIs through client libraries without ever seeing how tokenizers convert text into indices, how multi-head self-attention computes query-key affinity, or how backpropagation updates gradients. The Machine is a personal quest to master generative AI by writing the fundamental pieces from scratch.
Topics Explored from First Principles
Tokenization & Vocab
- Byte-pair encoding (BPE) & character-level tokenizers
- Vocabulary construction and special token handling
- Out-of-vocabulary fallback strategies
- Token distribution across text corpora
Neural Mechanics
- Multi-head self-attention layer implementations
- Positional embeddings (sinusoidal & rotary)
- Feed-forward blocks and LayerNorm stability
- Loss computation with cross-entropy
Training on Constrained Hardware
- Training small parameter models on Intel Core i5 CPU / Iris Xe
- Batch size optimization for 8 GB RAM headroom
- Observing perplexity drops over text epochs
- Translating lessons into quantized SLM deployment in Zeta
What This Research Powers in Berry Tech
Building and testing small models from scratch provides the intuition needed across the rest of the Berry Tech studio. It informs how Synapse clusters code symbols using quantized Qwen SLMs, and how Zeta caches context buffers efficiently without blowing through host memory.