HarmonyFidelisHarmonyFidelis
Login
Skip to content
AURA
LearnConfigurePublic programs
Learn

First principles

LearningInference

Mechanisms

RepresentationsContext & memoryArchitecturesDiffusion

Advanced research

Reasoning & agentsWorld modelsAGI & ASIEvaluation

Explore each space independently. More detail is always within reach.

Aura home
AuraUnderstand AIArchitectures

Transformers: understanding attention

Compare architectures, training methods and complete systems at the right level.

EssentialsExperimentGo deeper
Scientific sources

Teaching simulation · fictional values

Numerical example: one query, three keys

K = [[1,0],[0,1],[1,1]], V = [[1,0],[0,1],[1,1]], Q = [a,1] and dₖ = 2. These numbers are chosen for the exercise.

Attention weights

K₁24.8 %
K₂24.8 %
K₃50.3 %
0 %25 %50 %75 %100 %

Output = [0.752 ; 0.752]

A causal mask excludes future positions in autoregressive prediction. Projections, multiple heads, local layers and residual connections complete a Transformer. Attention weights alone are not causal proof of its behaviour.

Check my understanding

Before answering, form your own explanation.

Transformer, diffusion and MuZero describe exactly the same design level.

Takeaway

Diffusion is a modelling and generation method that can use different networks. DreamerV3 and MuZero organise components into learning and decision systems. These names are not interchangeable categories.

Go deeper
From concept to practicePut it into practice

Scientific sources

  • Vaswani et al. · 2017Attention Is All You NeedarXiv v7 · 2023
    Original publication
  • Hochreiter & Schmidhuber · 1997Long Short-Term Memory
    Original publication
  • Ho, Jain & Abbeel · 2020Denoising Diffusion Probabilistic Models
    Original publication
  • Jain & Wallace · 2019Attention is not Explanation
    Original publication

Teaching synthesis of the cited sources. Published results remain tied to their tasks, protocols and budgets.

Explore nextDiffusionInference

Search Aura

18 results

LearnLearning, then using a modelSeparate learned parameters from information supplied with a request.
LearnFrom context to the next tokenSeparate the model’s probabilities from the rule that selects an output.
LearnFrom text to representationsUnderstand why words become vectors without equating a vector with human meaning.
LearnWhat kind of memory?Separate working context, recurrent state, learned parameters and persistent storage.
LearnTransformers: understanding attentionCompare architectures, training methods and complete systems at the right level.
LearnGenerating by denoisingDistinguish corruption during learning from progressive generation.
LearnFrom a model to a system that actsUnderstand the harness around a model and its tools.
LearnPredicting to choose an actionExplore longer horizons and the risk of inaccurate predictions.
LearnDefinitions, not a single verdictExamine generality, performance and adaptation before applying a label.
LearnWhat can a result establish?Move from a convincing impression to a reproducible comparison.
ConfigurePrepare a ChatGPT projectA clear objective, useful context and instructions you can actually use.
ConfigureChoose for your taskStart with your needs. Compare access and outcomes without a universal model ranking.
ConfigureMake your request verifiableTurn an intention into an objective, context and success criteria.
ConfigurePut the right information in the right placeDistinguish working rules, task resources and memory managed by the product.
ConfigurePrepare a development environmentDefine the model’s role, harness capabilities and the access you actually need.
ConfigureCompose a development workflowOrganise preparation, changes, verification and reporting. Reuse the steps that help.
ConfigurePrepare a mission, then check the resultA reproducible example, a checklist and a record of your observations.
Public programsTentacularPublicly available uses, features and resources.