← Berry Tech
Berry Tech Independent builder's lab
The Machine · Research & Learning
Research Notes ↓
Berry Tech Ecosystem
Status: Research & Learning Experiment

Understanding language models
from first principles.

"The Machine" is an educational and personal research project exploring how generative AI and neural language models work under the hood. Rather than merely consuming cloud APIs, it investigates tokenization, vocabulary construction, PyTorch neural architectures, and pattern learning directly.

PyTorchDeep Learning
Custom TokenizersVocab Construction
First PrinciplesResearch Purpose
EducationalNo Fake Scale Claims
the_machine — model_scratch.py
PyTorch NLP
# Fundamental Tokenizer & Vocab Test >>> vocab = build_vocab(corpus, min_freq=2) ✓ Vocabulary size: 4,096 distinct tokens >>> tokens = tokenize("Start with an ambitious question") Tensor: [104, 3812, 42, 891, 1204] (len=5)   # Transformer Block Initialization >>> model = TinyTransformer(d_model=128, n_heads=4) ✓ Parameters: ~420K | Designed for 8GB rig training >>>
// 01 · The Motivation

“What happens inside an AI model when text is transformed into weights, vectors, and predictions?”

Modern developers are often taught to call external AI APIs through client libraries without ever seeing how tokenizers convert text into indices, how multi-head self-attention computes query-key affinity, or how backpropagation updates gradients. The Machine is a personal quest to master generative AI by writing the fundamental pieces from scratch.

// 02 · Core Research Areas

Topics Explored from First Principles

Area 01

Tokenization & Vocab

  • Byte-pair encoding (BPE) & character-level tokenizers
  • Vocabulary construction and special token handling
  • Out-of-vocabulary fallback strategies
  • Token distribution across text corpora
Area 02

Neural Mechanics

  • Multi-head self-attention layer implementations
  • Positional embeddings (sinusoidal & rotary)
  • Feed-forward blocks and LayerNorm stability
  • Loss computation with cross-entropy
Area 03

Training on Constrained Hardware

  • Training small parameter models on Intel Core i5 CPU / Iris Xe
  • Batch size optimization for 8 GB RAM headroom
  • Observing perplexity drops over text epochs
  • Translating lessons into quantized SLM deployment in Zeta
// 03 · Insights & Synthesis

What This Research Powers in Berry Tech

Building and testing small models from scratch provides the intuition needed across the rest of the Berry Tech studio. It informs how Synapse clusters code symbols using quantized Qwen SLMs, and how Zeta caches context buffers efficiently without blowing through host memory.

Loading The Machine Research...