? Autoresearch and the experimental society
The most important thing happening in AI right now is not just the intelligence of the models, but the harnesses that make that intelligence usable.
In the rapidly evolving world of artificial intelligence, the focus is no longer solely on the intelligence of the models themselves but on the tools that make this intelligence accessible and usable. This shift is exemplified by a recent development in the field: autoresearch, a Python codebase released by Andrej Karpathy that has the potential to revolutionize how knowledge is produced, particularly in scientific research.
Autoresearch is an autonomous experimental loop designed to streamline the process of generating knowledge. In this system, a human sets a strategic direction and defines what constitutes success, while the agent iteratively works towards achieving that goal within predefined boundaries. Karpathy's initial experiment with autoresearch involved training a GPT-2-level model over just two days, achieving a 11% faster training time and identifying 20 genuine improvements. This demonstrates the power of autoresearch in accelerating the development of advanced AI models.
The impact of autoresearch was quickly recognized beyond its initial scope. Shopify's CEO, Toby Lütke, utilized the tool on the company's internal model, QMD. Running 37 experiments overnight, Toby woke up to a 0.8-billion-parameter model that outperformed the previous 1.6-billion-parameter version by 19%. Notably, Toby is not a machine learning engineer, highlighting the accessibility and versatility of autoresearch.
The strength of autoresearch lies in its ability to address two critical challenges simultaneously. Firstly, it automates parts of the knowledge-production process, making it more efficient and scalable. Secondly, it solves the agent control problem, ensuring that AI systems remain focused on their intended tasks. Traditional AI models often drift when given open-ended briefs or when optimized for the wrong metrics. Autoresearch mitigates this risk by design, as the human sets the strategic direction, while the system ensures the model stays on track.
Recognizing the broader applicability of autoresearch, Karpathy spent the past month adapting the tool for knowledge work beyond machine learning. His goal was to create a system capable of running structured, low-cost experiments on the types of decisions teams make weekly. This new version, named AutoBeta, aims to democratize the process of generating actionable insights across various domains. Karpathy is making the full playbook and skillset available to paying members, marking a significant step towards widespread adoption.
The initial reaction to autoresearch was that its principles could extend beyond machine learning. The core loop of hypothesize, test, score, and iterate is generic and applicable to a wide range of knowledge-intensive fields. When Karpathy began experimenting with autoresearch on other aspects of his work, he encountered unexpected challenges. The process did not unfold exactly as anticipated, but this only served to underscore the potential of the tool to transform various sectors.
In conclusion, autoresearch represents a groundbreaking development in the field of AI, offering a powerful solution to the challenges of knowledge production and agent control. By harnessing the power of autonomous experimental loops, it empowers individuals and organizations to generate insights more efficiently and effectively. As the tool continues to evolve and expand into new domains, it promises to reshape the way knowledge is created and utilized, fostering an experimental society where innovation thrives.
