Tokenization
Text is split into learned units called tokens. A token is not always a word; spelling, language and formatting change token counts and costs.
Aura Academy
Section 2 · Mechanisms
How large language models work
An LLM predicts tokens from context. Scale and training make that simple objective surprisingly capable, but the model still generates likely continuations rather than consulting a guaranteed store of facts.
Beginner to intermediate25 minutes
By the end, you can
- Explain tokens, embeddings, attention and the Transformer at a practical level.
- Separate pretraining, post-training and inference.
- Predict why context length, sampling and external memory affect results.
Five stages of the LLM pipeline
Embeddings
Tokens become vectors. During training, useful geometric relationships emerge so the network can operate on contextual representations rather than raw characters.
Attention and Transformers
Self-attention lets each position weigh information from other positions. Transformer blocks combine attention and feed-forward computation repeatedly.
Pretraining and post-training
Pretraining learns broad patterns by prediction. Supervised fine-tuning and preference methods such as RLHF or DPO shape instruction following and behavior.
Inference
At runtime, the model calculates a probability distribution and selects the next token repeatedly. Sampling settings trade repeatability against variation; they do not add knowledge.
From text to generated text
- 01
Tokenize
- 02
Embed
- 03
Attend
- 04
Predict next token
- 05
Repeat with context
The context window is working state, not durable memory. Longer context helps only when relevant information is present, accessible and correctly used.
Common confusions
- A token is not a stable word unit, so word counts and token counts are not interchangeable.
- A larger context window does not guarantee uniform attention; important evidence can still be missed.
- Post-training can shape behavior, but it does not turn generated probabilities into guaranteed truth.
Practice: trace one answer
Take a short LLM question and identify what influences the response.
- Mark the instruction, supplied context and information the model must infer from parameters.
- List one ambiguity that could change token prediction.
- Change sampling or context once and compare stability without assuming causality.
Quick check
Does an LLM retrieve a verified sentence before every answer?
No. By default it generates tokens from learned parameters and current context; retrieval is a separate system component.
What does self-attention do?
It lets a token representation weight information from other positions to build a context-dependent representation.
What is the difference between pretraining and inference?
Pretraining changes model parameters from large datasets; inference uses fixed parameters to generate an output for a new input.
Keywords to know
- token
- tokenizer
- embedding
- attention
- Transformer
- pretraining
- SFT
- RLHF
- DPO
- inference
Primary research
Attention Is All You NeedPrimary paper introducing the Transformer architecture.SentencePiecePrimary paper on language-independent subword tokenization from raw text.Training language models to follow instructionsPrimary paper on supervised fine-tuning and reinforcement learning from human feedback.Direct Preference OptimizationPrimary paper presenting direct optimization from preference pairs.Lost in the MiddleStudy of how language models use information placed at different context positions.