Concept 1 / 5
Tokenization
Text is split into learned units called tokens. A token is not always a word; spelling, language and formatting change token counts and costs.
Academia Aura
Sección 2 · Mecanismos
Cómo funcionan los grandes modelos de lenguaje
Un LLM predice tokens a partir del contexto. Genera continuaciones probables: no consulta por defecto una reserva garantizada de hechos.
Principiante a intermedio25 minutos
Al terminar podrás
- Explain tokens, embeddings, attention and the Transformer at a practical level.
- Separate pretraining, post-training and inference.
- Predict why context length, sampling and external memory affect results.
Vue interactive
Explorez cette leçon à votre rythme
Choisissez une vue. Le détail n’apparaît que lorsque vous le demandez.
Interaction locale · aucune donnée envoyée
Confusiones frecuentes
Afficher +Réduire −
Confusiones frecuentes
- A token is not a stable word unit, so word counts and token counts are not interchangeable.
- A larger context window does not guarantee uniform attention; important evidence can still be missed.
- Post-training can shape behavior, but it does not turn generated probabilities into guaranteed truth.
Palabras clave
Afficher +Réduire −
Palabras clave
- token
- tokenizer
- embedding
- attention
- Transformer
- pretraining
- SFT
- RLHF
- DPO
- inference
Investigación primaria
Voir les sources +Réduire −
Investigación primaria
Voir les sources +Réduire −Attention Is All You NeedPrimary paper introducing the Transformer architecture.SentencePiecePrimary paper on language-independent subword tokenization from raw text.Training language models to follow instructionsPrimary paper on supervised fine-tuning and reinforcement learning from human feedback.Direct Preference OptimizationPrimary paper presenting direct optimization from preference pairs.Lost in the MiddleStudy of how language models use information placed at different context positions.