Skip to contentExperiment
What can a result establish?
Move from a convincing impression to a reproducible comparison.
Essentials
How it works
Define the task, data, baseline and criteria before an experiment. Reserve examples that were not used to tune the configuration. Record errors, resources and between-run variation for stochastic outputs.
Takeaway
Narrow samples, contamination and test adaptation can distort conclusions. Include new cases and failures. A strong score does not guarantee performance in another domain.
From concept to practicePut it into practice
Scientific sources
- Chollet · 2019On the Measure of IntelligenceOriginal publication
- Morris et al. · 2023Levels of AGIarXiv v5 · 2025Original publication
Teaching synthesis of the cited sources. Published results remain tied to their tasks, protocols and budgets.