When I started writing about AI, I thought the “model” was whatever lived on the GPU and spat out answers.
Simple.
Done.
Onward.
Turns out, not quite. There’s something that sits before training, before inference, and before any cleverness shows up. It’s called architecture, and it’s the part we almost never talk about outside research circles.
Architecture = The Brain’s Blueprint
Architecture is just the design of how the model thinks.
Not the data. Not the training. Not the GPUs. Not the math.
Just the layout.
The wiring diagram.
For decades, this wiring diagram came in different shapes:
-
RNNs (models that remembered yesterday)
-
LSTMs (models that remembered yesterday slightly better)
-
CNNs (models that inspected images like bouncers checking IDs)
One architecture changed everything: The Transformer.
What Makes a Transformer a Transformer?
It has two big superpowers:
-
Attention :the ability to look at every part of a sentence and decide what matters
-
Parallelism : the ability to think about many things at once, without step-by-step bottlenecks
And inside each Transformer block, you mostly have two components:
-
Multi-Head Attention
-
Feed Forward Networks (FFN)
Stack a number of those and suddenly you’ve built a modern LLM. This stack of blocks is called as architecture. Think of it as the playing conditions before the cricket match even starts:
-
the pitch
-
the boundaries
-
the field layout
-
the weather
The players (your neurons) and the coach (the optimizer) are important, but the ground determines how the match will flow.
Why This Matters for My Writing Journey
I realized that if I wanted to understand AI deeply enough to explain it to my kids one day, I needed to understand the blueprint first. Everything else (training, inference, MoE, GPUs, CUDA, Randy Johnson and Curt Schilling, and yes, Travis Head ) all sit on top of this architecture . This is the moment in the story where the camera pans out and you finally see the whole cricket ground.
Next up:
Mixture of Experts the specialist lineup inside the Transformer.
And after that?
A man named Travis Head walks into a game and becomes an expert layer all by himself.
Stay tuned.