Concept 1 / 5
Tokenization
Text is split into learned units called tokens. A token is not always a word; spelling, language and formatting change token counts and costs.
Aura Akademie
Abschnitt 2 · Mechanismen
Wie große Sprachmodelle funktionieren
Ein LLM sagt Tokens aus dem Kontext voraus. Es erzeugt wahrscheinliche Fortsetzungen und greift standardmäßig nicht auf einen garantierten Faktenspeicher zu.
Einstieg bis Mittelstufe25 Minuten
Danach kannst du
- Explain tokens, embeddings, attention and the Transformer at a practical level.
- Separate pretraining, post-training and inference.
- Predict why context length, sampling and external memory affect results.
Vue interactive
Explorez cette leçon à votre rythme
Choisissez une vue. Le détail n’apparaît que lorsque vous le demandez.
Interaction locale · aucune donnée envoyée
Häufige Verwechslungen
Afficher +Réduire −
Häufige Verwechslungen
- A token is not a stable word unit, so word counts and token counts are not interchangeable.
- A larger context window does not guarantee uniform attention; important evidence can still be missed.
- Post-training can shape behavior, but it does not turn generated probabilities into guaranteed truth.
Schlüsselbegriffe
Afficher +Réduire −
Schlüsselbegriffe
- token
- tokenizer
- embedding
- attention
- Transformer
- pretraining
- SFT
- RLHF
- DPO
- inference
Primärforschung
Voir les sources +Réduire −
Primärforschung
Voir les sources +Réduire −Attention Is All You NeedPrimary paper introducing the Transformer architecture.SentencePiecePrimary paper on language-independent subword tokenization from raw text.Training language models to follow instructionsPrimary paper on supervised fine-tuning and reinforcement learning from human feedback.Direct Preference OptimizationPrimary paper presenting direct optimization from preference pairs.Lost in the MiddleStudy of how language models use information placed at different context positions.