Task
1.00
Sampling k = 5
It picked le 100.0%

drawn at random from the 5 words that survive top-k out of 81. Lower the temperature and it keeps landing on the same word; raise it and it starts wandering.

Full model
Input
Encoder block 1
Encoder block 2
Decoder
Output

The Transformer model

This is the architecture from the 2017 paper that started all of it. Solid boxes are the parts this module has built, and you can click any of them to jump there. Faded boxes are real parts of the model that are not built here yet, shown so you can see how much of the whole you are looking at. Dotted lines are residual connections, where a block's input is added back onto its output. The purple line is the one wire joining the two halves.

Encoder ×NDecoder ×NEach input word becomes a list of 64 numbers.EmbeddingsA wave pattern that says where each word sits.Positional encodingEvery word looks at every other word, four ways at once.Multi-head AttentionAdd the input back on, then rescale.Add & NormEach word thinks about what it just heard, alone.MLPsfeed-forwardAdd and rescale again. One block is now complete.Add & NormThe output built so far, turned into vectors.EmbeddingsThe same wave pattern, applied to the output side.Positional encodingLike attention, but a word may not look ahead at what has not been written yet.Masked Multi-headAttentionAdd and rescale.Add & NormQuestions come from the output side, answers from the encoder. This is where the two halves meet.Multi-head Attentioncross-attentionAdd and rescale.Add & NormThe same thinking step as the encoder side.MLPsfeed-forwardAdd and rescale.Add & NormTurn the final vector into one score per word in the vocabulary.LinearTurn those scores into percentages that add up to 100.SoftmaxInput sentenceOutput so farNext-word probabilities
English in · encoderFrench out · decodertokens<fr>thecatsatonthematembeddingQ K VQKVattentionheads + W_oMLPwritten so far<bos>embeddingmaskedcross-attnMLPnext wordlelanewencoder output feeds cross-attentionwrites “le”

Tokenize

Split the sentence into pieces and turn each piece into a number.

text → tokens → row IDs

A model only works with numbers. So the first step turns your sentence into numbers.

It happens in two parts. First the sentence is split into pieces called . In this model a token is a whole word, split on spaces and punctuation. Then each token is looked up in a list called the . The vocabulary holds every token the model knows, numbered from 0.

The number is what gets passed on. The word itself is thrown away and never comes back.

The size of the vocabulary is a real design choice. If it is too small, common words get broken into pieces and sequences get longer. If it is too big, the model gets bigger too, because the embedding table and the output layer both have one row per token. Real models use somewhere between 30,000 and 200,000 tokens. This one uses 81, which is all it needs for its small phrasebook.

Look at the first token. It is not a word from your sentence. It is a task tag, added automatically at the front of the input, and it tells the model which job to do.

This model was trained on two jobs, so it has two tags. <fr> means translate the rest into French. <next> means continue the sentence in English. Switching the Task control at the top swaps which tag gets added, and nothing else about the model changes. The same weights, the same encoder, the same decoder produce a different kind of answer because the first token is different.

The model was never told what the tag means. It saw thousands of examples that began with <fr> and ended in French, and thousands that began with <next> and ended in English, and it worked out the pattern the same way it worked out everything else.

Large models do the same thing with plain words instead of a special tag. T5 is trained on inputs that literally begin "translate English to German:" or "summarize:", and to the model that instruction is just more tokens, no different from the text that follows it. That is also what a system prompt is: text pasted in front of yours, which the model has learned to treat as instructions.

How to read this

One card per token, in order. The number under each word is its position in the vocabulary list, and the same word always gets the same number. The first card is the task tag rather than part of your sentence, so it is marked separately.

What to notice

"the" shows up more than once and lands on the same row number every time. That is why identical words begin as identical lists of numbers. Everything that later tells them apart comes from where they sit and what surrounds them.

  1. task <fr> row 3
  2. 1 the row 69
  3. 2 cat row 14
  4. 3 sat row 60
  5. 4 on row 49
  6. 5 the row 69
  7. 6 mat row 43

<fr> is a task tag, not part of your sentence. It is added at the front of the input and tells the model which job to do: <fr> to translate into French, <next> to continue the sentence. Everything after it is your text.

tokens
7
in vocabulary
7
hashed
0
vocabulary size
81

the appears more than once and lands on the same row every time. Identical words start out as identical vectors, and only position tells them apart.

Every word here is in the vocabulary, so nothing needed hashing. Try a rare or invented word to see the fallback.

What real models use instead

This module splits on whitespace, which keeps the panels readable but means a 81-word vocabulary can never cover real text. Production models tokenize into subword pieces instead: a vocabulary of 30k to 200k pieces covers any input, because anything unfamiliar decomposes into fragments it already knows. Worth trying these on your own sentence to see where the splits land.

  • tiktoken OpenAI · Byte-pair encoding Used by GPT-3.5, GPT-4 and o-series. Merges the most frequent byte pairs until the vocabulary is full, so common words stay whole and rare ones split.
  • SentencePiece Google · Unigram or BPE Used by T5, LLaMA and Gemma. Trains straight from raw text with no pre-splitting on spaces, so it works the same for languages that do not use them.
  • WordPiece Google · Likelihood-greedy Used by BERT. Picks the merge that most improves the likelihood of the training corpus rather than the one that is simply most frequent.
Input
Position
Self-attention
Output
Self-attention
Block
Decoder
Output
1/24

A real trained model. This encoder-decoder was trained on two jobs, translating English into French and continuing an English sentence, and gets 98.8% of held-out sentences exactly right. Every attention pattern and every prediction here is learned behaviour. It is small and only knows the words in its phrasebook, but nothing on screen is faked. Its sizes are smaller than the 2017 paper's, and that paper is where the architecture comes from.

About this model

This is a real transformer, trained here rather than downloaded, and it follows the architecture in the 2017 paper. It is much smaller than anything in that paper, because the point is to be small enough to watch every number. Where the sizes differ, they differ only in scale.

Sizes, next to the paper's base model

SettingThis modelPaper, base
Encoder blocks16
Decoder blocks16
d_model64512
Attention heads48
d_k per head1664
d_ff2562048
Vocabulary81 whole words37,000 subword pieces
Parameters121,15265 million
Dropoutnone0.1
Training data19,440 generated pairs4.5M WMT sentence pairs

Every one of these is a size, not a structural change. A wider model has more numbers per token; a deeper one repeats the same block more times. Nothing in the list alters how a block works.

What is unchanged from the paper

  • Sinusoidal positional encoding, the exact formula from section 3.5.
  • Scaled dot-product attention, including the division by the square root of d_k.
  • Multi-head attention with per-head projections and a combining output projection.
  • Residual connection around every sublayer, with layer normalization applied after the add, which is the post-norm arrangement the paper uses.
  • A position-wise feed-forward network of two linear layers with a nonlinearity between them.
  • Causal masking in the decoder, applied before softmax.
  • Cross-attention taking queries from the decoder and keys and values from the encoder.
  • Embedding weights shared with the output projection, as described in section 3.4.

How it was trained

Two tasks at once: translate English into French, and continue an English sentence. 19,440 sentence pairs, generated from a small grammar so that a model this size can actually learn them. Trained with Adam and a warmup schedule, and it reaches 98.8% correct on held-out sentences it never saw. Training ran once, offline. Your browser only loads the finished weights and runs the forward pass.

The paper

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30. arxiv.org/abs/1706.03762

 space plays · ←→ step · click any stage above to jump there.