Concept 1 / 5
Tokenization
Text is split into learned units called tokens. A token is not always a word; spelling, language and formatting change token counts and costs.
Aura Academy
Section 2 · Mechanisms
How large language models work
An LLM predicts tokens from context. Scale and training make that simple objective surprisingly capable, but the model still generates likely continuations rather than consulting a guaranteed store of facts.
Beginner to intermediate25 minutes
By the end, you can
- Explain tokens, embeddings, attention and the Transformer at a practical level.
- Separate pretraining, post-training and inference.
- Predict why context length, sampling and external memory affect results.
Vue interactive
Explorez cette leçon à votre rythme
Choisissez une vue. Le détail n’apparaît que lorsque vous le demandez.
Interaction locale · aucune donnée envoyée
Common confusions
Afficher +Réduire −
Common confusions
- A token is not a stable word unit, so word counts and token counts are not interchangeable.
- A larger context window does not guarantee uniform attention; important evidence can still be missed.
- Post-training can shape behavior, but it does not turn generated probabilities into guaranteed truth.
Keywords to know
Afficher +Réduire −
Keywords to know
- token
- tokenizer
- embedding
- attention
- Transformer
- pretraining
- SFT
- RLHF
- DPO
- inference
Primary research
Voir les sources +Réduire −
Primary research
Voir les sources +Réduire −Attention Is All You NeedPrimary paper introducing the Transformer architecture.SentencePiecePrimary paper on language-independent subword tokenization from raw text.Training language models to follow instructionsPrimary paper on supervised fine-tuning and reinforcement learning from human feedback.Direct Preference OptimizationPrimary paper presenting direct optimization from preference pairs.Lost in the MiddleStudy of how language models use information placed at different context positions.