Please enable JavaScript.
Coggle requires JavaScript to display documents.
How AI Is Built - Full Tree - Coggle Diagram
How AI Is Built - Full Tree
PART 1 - The Universal Build Pipeline (applies to ALL modern AI)
Step 1 - Define the problem precisely
What is the input (text, pixels, audio, sensor readings)
What is the output (a class, a number, a sequence, an action)
What does better mean (this becomes your metric)
Step 2 - Collect data
Scrape the web
Buy datasets
Record sensors
Run simulations
Crowdsource labels
Generate synthetic data
Example - ImageNet = 14 million images labeled via Amazon Mechanical Turk
Step 3 - Clean and preprocess the data
Remove duplicates
Fix errors
Normalize values (scale to similar ranges)
Handle missing values
Split the data - typically 80% training / 10% validation / 10% testing
Test set must NEVER be seen during training (honest final exam)
Step 4 - Represent the data (the secret sauce)
Computers only understand numbers - everything must become vectors
Text goes to tokens goes to numbers
Images go to grids of pixels (e.g. 224x224x3 numbers)
Tabular data goes to feature vectors
Historically humans hand-crafted features (feature engineering)
Deep learning learned to do this automatically
Step 5 - Choose a model architecture
Model = mathematical function with adjustable internal numbers
Parameters = weights and biases
Architecture defines the shape of the function (how parameters connect)
Step 6 - Define a loss function
Single number measuring how wrong predictions are
Regression goes to Mean Squared Error
Classification goes to cross-entropy
LLMs goes to next-token prediction error
Step 7 - Optimization - gradient descent
Compute how the loss changes if you nudge each parameter slightly (the gradient)
Adjust all parameters slightly in the direction that reduces loss
Repeat millions of times in small batches (mini-batch gradient descent)
Optimizer like Adam controls step size
Backpropagation (1986, Rumelhart/Hinton/Williams) computes gradients efficiently through layers - this made deep learning possible
Step 8 - Evaluate
Measure on held-out test set with task metrics
Accuracy, precision/recall, F1
BLEU (translation)
Perplexity (language)
Elo ratings (games)
Human preference (chatbots)
Step 9 - Regularize and iterate
Prevent overfitting (memorizing instead of generalizing)
Dropout
Weight decay
Early stopping
Data augmentation
Step 10 - Deploy and monitor
Serve model as an API
Compress it (quantization, distillation)
Watch for data drift in production
Retrain periodically
PART 2 - How Rule-Based / Symbolic AI Was Built
Step 1 - Interview domain experts
Doctor, chemist, engineer
Step 2 - Extract rules as if-then statements
Example - MYCIN - IF fever AND elevated white cells THEN consider bacterial infection (confidence 0.7)
Step 3 - Encode a knowledge base
Thousands of facts in formal logic
Step 4 - Build an inference engine
Forward chaining - facts to conclusions
Backward chaining - goal to what facts would prove it
Step 5 - Handle uncertainty
Certainty factors
Bayesian networks
Fuzzy logic
Examples built this way
MYCIN (medical diagnosis, ~1972)
DENDRAL (chemistry)
Early chess programs (hand-written rules + search tree)
Why it failed at scale
Experts cannot articulate all tacit knowledge
Rule systems became brittle (one unexpected input breaks the chain)
Caused the first AI winter (~1974)
Learning from data replaced hand-writing rules
PART 3 - How Classical Machine Learning Was Built (1980s-2010s)
Linear and Logistic Regression
Fit a line/plane through data
Used everywhere - credit scoring, pricing
Decision Trees
Split data with questions (Is income greater than $50k?)
Built by greedily choosing splits that best separate classes
SVMs - Support Vector Machines (1990s)
Find hyperplane maximizing margin between classes
Kernel trick handles non-linear data
Dominated text classification and bioinformatics for a decade
Ensembles - combine many weak models
Random Forests - hundreds of trees on random subsets, majority vote
Gradient Boosting (XGBoost, LightGBM) - trees built sequentially, each correcting previous errors
Won virtually every Kaggle tabular-data competition
Naive Bayes
Probabilistic classifier
Spam filters used it for years
Unsupervised
k-Means - clustering
PCA - dimensionality compression
The actual build process
Massive human effort in feature engineering
Convert raw inputs into informative numbers
Example - fraud detection - number of transactions in last hour, distance between purchase locations
The algorithm was often the easy part - the features were the product
PART 4 - How Deep Learning Was Built (the 2012 revolution)
4.1 The unit - the artificial neuron
Inputs x weights goes to sum goes to add bias goes to apply activation function
ReLU - max(0,x)
Sigmoid
Stack layers of neurons = neural network
Deep network = many layers
4.2 The training loop (every deep net ever trained)
Repeat millions of times
Grab a mini-batch of examples
Forward pass - run data through network to predictions
Compute loss (how wrong?)
Backpropagation - compute gradient of loss w.r.t. every weight
Optimizer step - adjust weights to reduce loss
Log metrics, validate periodically
4.3 What made it suddenly work around 2012
Data at scale (ImageNet)
GPUs - graphics chips perfect for matrix math (why NVIDIA dominates AI today)
Better activation functions (ReLU)
Better initialization
Dropout (Hinton et al.)
Batch normalization
AlexNet (2012) crushed ImageNet competition - deep learning exploded
4.4 Key architectures and how each was built
CNNs - Convolutional Neural Networks
Idea - Fukushima 1980 to LeCun 1989 to AlexNet 2012
Slide small filters (kernels) across image to detect edges, textures, shapes, objects - hierarchical features learned automatically
Built with - convolution layers + pooling (downsampling) + fully connected layers
ResNet (2015) - skip connections solved vanishing gradient in very deep nets (now everywhere)
RNNs to LSTMs to GRUs (sequence models)
For ordered data - text, speech, time series
Feed network one element at a time, carrying a hidden state forward as memory
LSTM (1997, Hochreiter and Schmidhuber) - gates decide what to remember and forget, solved forgetting long sequences
Powered early Google Translate and Siri speech recognition
Transformers (2017) - Attention Is All You Need (Google) - architecture behind ChatGPT, Claude, everything modern
Tokenization - split text into subword units (unbelievable to un, believ, able), each mapped to an ID
Embedding - each token ID becomes a vector in ~thousands-dimensional space with learnable parameters
Positional encoding - inject order info (transformers process tokens in parallel, need sequence position)
Attention mechanism - for each token compute query, key, value vectors; each token attends to all previous tokens weighted by relevance - core innovation: dynamic context-dependent weighting of every word against every other word
Multi-head attention - many attention processes in parallel, each catching different relationship types (grammar, reference, meaning)
Feed-forward layers + residual connections + layer normalization - stabilize and enrich representations
Stack this block 12-120+ times = decoder-only LLM
Output head - map final vector to probability distribution over vocabulary; trained to put probability on correct next token
PART 5 - How Modern LLMs Are Built (full industrial pipeline)
Phase 1 - Data engineering (months of work)
Crawl the web at petabyte scale (Common Crawl, plus books, code, Wikipedia, scientific papers)
Filter aggressively - remove garbage, NSFW, personal data, low-quality content (heuristics + classifiers)
Deduplicate (near-dupes hurt performance)
Classify quality (perplexity-based scoring) and upweight good sources (textbooks, encyclopedias)
Result - e.g. 10-15 trillion tokens after processing (1 token = about 3/4 of a word)
Phase 2 - Pretraining (the expensive part)
Randomly initialize billions of parameters
Train on next-token prediction over trillions of tokens - show sequence, hide last word, predict it, penalize wrong probabilities
Run on clusters of thousands of GPUs/TPUs for weeks to months (GPT-4-class = tens of millions of dollars in compute)
Scaling laws (OpenAI/Kaplan 2020, Chinchilla 2022) - loss improves predictably with more compute, data, parameters; optimal ratio ~20 tokens per parameter (budget-allocation problem)
Result - base model - a world-model that can complete text but does not yet behave like an assistant
Phase 3 - Alignment (making it useful and safe)
Supervised Fine-Tuning (SFT) - humans write ideal responses to example prompts; model trained on them
Reward modeling (RLHF) - humans compare pairs of model responses (which is better?); train separate reward model to predict human preference
RL fine-tuning (PPO, or newer DPO) - base model optimized to maximize reward-model scores
Result - chatbot personality - helpful, harmless, refuses dangerous requests
Add system prompts, guardrails, content filters at deployment
Phase 4 - Inference (how it answers you)
Prompt tokenized to embeddings
Model runs forward pass, producing probability for each possible next token
Decoding strategy picks one - greedy (most likely) or sampling with temperature/top-p (more variety)
Chosen token appended, whole sequence fed back in - auto-regressively, one token at a time (why LLMs type word by word)
Speedups - KV-caching (do not recompute past tokens), speculative decoding, quantization
Phase 5 - Post-training enhancements
Tool use / function calling - train model to output structured calls (call calculator(x)) that the system executes
Retrieval-Augmented Generation (RAG) - at query time retrieve relevant documents from database and insert into prompt; fresh private knowledge without retraining
Distillation - train small model to mimic big one (why Llama/Phi are cheap)
Agents - wrap LLM in a loop - plan, call tools, observe results, iterate (Devin, Claude with tools, AutoGPT)
PART 6 - How Generative Image/Audio/Video AI Is Built
GANs (2014, Goodfellow) - two networks in a duel
Generator - turns random noise into fake images, trying to fool discriminator
Discriminator - tries to tell real vs fake
Trained alternately until fakes are indistinguishable
Examples - StyleGAN faces, early deepfakes
VAEs (2013)
Compress data into probability distribution in latent space
Sample and decode back
Built by maximizing lower bound on data likelihood (the ELBO)
Diffusion models (2020 to DALL-E 3, Stable Diffusion, Sora) - the modern winner
Take an image
Forward process - add a little Gaussian noise repeatedly over hundreds of steps until pure static
Training - train neural net (U-Net or transformer) to predict the noise added at each step
Generation - start from pure noise, iteratively denoise step-by-step, guided by text embedding (CLIP encodes prompt into vector that steers denoising)
Variants - latent diffusion (operate in compressed space for speed), flow matching (faster training)
PART 7 - How Reinforcement Learning Systems Are Built
The framework
Agent observes state, takes action, environment returns reward + new state
Goal - learn a policy (action-choosing function) maximizing long-term reward (discounted sum of future rewards)
How AlphaGo was built (the landmark case)
Supervised warm-up - train on 30 million human game positions to predict expert moves
Self-play RL - model plays millions of games against itself; winner gets reward
Monte Carlo Tree Search (MCTS) - at each move simulate many possible futures down search tree, pick move with best win probability
Iterate - new champion generates training data for next iteration
AlphaZero later dropped human data entirely - learned purely from self-play in chess, shogi, Go
Core RL algorithms
Q-learning - learn value for each state-action pair; act greedily on it
Deep Q-Networks (DQN, DeepMind 2013) - Q-learning with neural net - learned Atari from pixels
Policy gradients / PPO / A3C - directly optimize the policy by gradient ascent on expected reward
RLHF - RL used to align language models
How robots are built (physical AI)
Sensors - cameras, LiDAR, IMUs, joint encoders, force sensors
Perception - CNNs/transformers turn raw sensor data into understanding (person 2m ahead)
Localization and mapping (SLAM) - build map of environment while tracking position within it
Planning - search algorithms (A*, rapidly-exploring random trees) find collision-free paths
Control - turn plans into motor commands (PID controllers, model predictive control, learned policies)
Sim-to-real - train in physics simulators (MuJoCo, Isaac Sim), transfer to real robot, close reality gap with domain randomization
Modern approach - imitation learning from human teleoperation data + RL fine-tuning (how modern humanoid robots learn manipulation)
PART 8 - How the Alternative AI Families Are Built
Evolutionary algorithms
Create population of random candidate solutions (often neural nets - NEAT evolves network structure too)
Evaluate each with a fitness function
Select the best - crossover (combine) + mutate (random tweaks)
Repeat for thousands of generations
Swarm intelligence
Ant colony optimization - virtual ants lay pheromone on paths; shorter paths accumulate more pheromone, converge on optimal routes - applied to logistics, network routing
Particle swarm - candidate solutions fly through solution space, each pulled toward personal best and swarm best
Fuzzy systems
Fuzzification - convert crisp inputs to degrees of membership (temperature = 70% hot)
Rule evaluation across rule base
Defuzzification - collapse back to crisp output (fan speed = 60%)
Neuro-symbolic AI
Extract symbolic rules/knowledge graph from experts or text
Attach neural components (embeddings, LLMs) for perception/fuzzy reasoning
Attach symbolic solvers/logic engines for exact reasoning
Train jointly or pipeline between the two - goal: LLM flexibility + calculator-like reliability
Bayesian AI
Define prior beliefs as probability distributions
Collect evidence
Apply Bayes theorem - posterior proportional to likelihood x prior
Update continuously as new data arrives; quantify uncertainty explicitly
PART 9 - The Engineering Layer (lab to product)
Frameworks
PyTorch (research standard)
JAX
TensorFlow
Distributed training
Data parallelism - split batch across GPUs
Tensor/pipeline parallelism - split the model itself
Gradient accumulation
Mixed precision (FP16/BF16) for speed
Experiment tracking
Weights & Biases
TensorBoard
Hyperparameter search - learning rate, batch size, architecture width/depth
Serving
ONNX
TensorRT
vLLM
Quantization (FP32 to INT8/INT4) to shrink models 2-4x
Batching requests
Monitoring
Track accuracy drift, latency, cost
Shadow deployments
A/B tests
Human feedback loops feeding the next training run
The One-Sentence Summary
Every AI ever built reduces to
Represent the world as numbers
Define a function with adjustable parameters
Define wrongness as a loss
Use calculus (gradients) to tune parameters until wrongness is minimized
Validate honestly
Wrap in engineering
Differences between a 1958 perceptron, AlphaGo, and GPT-class models
Architecture of the function
Scale of data and compute
Objective they are optimized for