Tokenization
Text is split into learned units called tokens. A token is not always a word; spelling, language and formatting change token counts and costs.
Aura Akademie
Abschnitt 2 · Mechanismen
Wie große Sprachmodelle funktionieren
Ein LLM sagt Tokens aus dem Kontext voraus. Es erzeugt wahrscheinliche Fortsetzungen und greift standardmäßig nicht auf einen garantierten Faktenspeicher zu.
Einstieg bis Mittelstufe25 Minuten
Danach kannst du
- Explain tokens, embeddings, attention and the Transformer at a practical level.
- Separate pretraining, post-training and inference.
- Predict why context length, sampling and external memory affect results.
Fünf Stufen der LLM-Pipeline
Embeddings
Tokens become vectors. During training, useful geometric relationships emerge so the network can operate on contextual representations rather than raw characters.
Attention and Transformers
Self-attention lets each position weigh information from other positions. Transformer blocks combine attention and feed-forward computation repeatedly.
Pretraining and post-training
Pretraining learns broad patterns by prediction. Supervised fine-tuning and preference methods such as RLHF or DPO shape instruction following and behavior.
Inference
At runtime, the model calculates a probability distribution and selects the next token repeatedly. Sampling settings trade repeatability against variation; they do not add knowledge.
From text to generated text
- 01
Tokenize
- 02
Embed
- 03
Attend
- 04
Predict next token
- 05
Repeat with context
The context window is working state, not durable memory. Longer context helps only when relevant information is present, accessible and correctly used.
Häufige Verwechslungen
- A token is not a stable word unit, so word counts and token counts are not interchangeable.
- A larger context window does not guarantee uniform attention; important evidence can still be missed.
- Post-training can shape behavior, but it does not turn generated probabilities into guaranteed truth.
Practice: trace one answer
Take a short LLM question and identify what influences the response.
- Mark the instruction, supplied context and information the model must infer from parameters.
- List one ambiguity that could change token prediction.
- Change sampling or context once and compare stability without assuming causality.
Quick check
Does an LLM retrieve a verified sentence before every answer?
No. By default it generates tokens from learned parameters and current context; retrieval is a separate system component.
What does self-attention do?
It lets a token representation weight information from other positions to build a context-dependent representation.
What is the difference between pretraining and inference?
Pretraining changes model parameters from large datasets; inference uses fixed parameters to generate an output for a new input.
Schlüsselbegriffe
- token
- tokenizer
- embedding
- attention
- Transformer
- pretraining
- SFT
- RLHF
- DPO
- inference
Primärforschung
Attention Is All You NeedPrimary paper introducing the Transformer architecture.SentencePiecePrimary paper on language-independent subword tokenization from raw text.Training language models to follow instructionsPrimary paper on supervised fine-tuning and reinforcement learning from human feedback.Direct Preference OptimizationPrimary paper presenting direct optimization from preference pairs.Lost in the MiddleStudy of how language models use information placed at different context positions.