Concept 1 / 5
Tokenization
Text is split into learned units called tokens. A token is not always a word; spelling, language and formatting change token counts and costs.
Academia Aura
Seção 2 · Mecanismos
Como grandes modelos de linguagem funcionam
Um LLM prevê tokens a partir do contexto. Ele gera continuações prováveis, não consulta por padrão um repositório garantido de fatos.
Iniciante a intermediário25 minutos
Ao final você poderá
- Explain tokens, embeddings, attention and the Transformer at a practical level.
- Separate pretraining, post-training and inference.
- Predict why context length, sampling and external memory affect results.
Vue interactive
Explorez cette leçon à votre rythme
Choisissez une vue. Le détail n’apparaît que lorsque vous le demandez.
Interaction locale · aucune donnée envoyée
Confusões comuns
Afficher +Réduire −
Confusões comuns
- A token is not a stable word unit, so word counts and token counts are not interchangeable.
- A larger context window does not guarantee uniform attention; important evidence can still be missed.
- Post-training can shape behavior, but it does not turn generated probabilities into guaranteed truth.
Palavras-chave
Afficher +Réduire −
Palavras-chave
- token
- tokenizer
- embedding
- attention
- Transformer
- pretraining
- SFT
- RLHF
- DPO
- inference
Pesquisa primária
Voir les sources +Réduire −
Pesquisa primária
Voir les sources +Réduire −Attention Is All You NeedPrimary paper introducing the Transformer architecture.SentencePiecePrimary paper on language-independent subword tokenization from raw text.Training language models to follow instructionsPrimary paper on supervised fine-tuning and reinforcement learning from human feedback.Direct Preference OptimizationPrimary paper presenting direct optimization from preference pairs.Lost in the MiddleStudy of how language models use information placed at different context positions.