Jakarta, ThedailyID — AI researchers, including Yoshua Bengio and Geoffrey Hinton, have warned governments to prepare for a possible “intelligence explosion” driven by AI systems improving AI research.
The warning comes from a study titled What If Automating AI R&D Triggers an Intelligence Explosion? The research involved 22 AI researchers, including Anthropic co-founder Jack Clark and OpenAI Chief Scientist Jakub Pachocki.
The study was published by the Cambridge Programme on AI Science & Policy. Its authors urged policymakers to act before AI development accelerates beyond effective human oversight.
The researchers define an “intelligence explosion” as a rapid acceleration in AI progress caused by AI systems automating research and development. They link the process to recursive self-improvement, or RSI.
According to the study, automated AI research could allow one developer to operate a workforce equivalent to millions of top human researchers. Such a shift could compress years of technological progress into months or less.
“An AI intelligence explosion could be the most important technological development in human history,” the researchers wrote. They warned that it could challenge human control over AI systems and weaken oversight across governments, companies and other institutions.
The researchers outlined three policy priorities. These include transparent reporting on AI R&D progress, measures to limit extremely rapid AI development, and preparations for adapting to a potential intelligence explosion.
Anthropic and OpenAI have also agreed to use independent evaluators to assess their AI models. Both companies have warned about risks linked to increasingly capable AI systems.
Anthropic says AI now produces 80 percent of its own code. OpenAI, meanwhile, uses autonomous AI agents in areas including the training of new models.
The study also identifies several potential risks. These include biological and cyber threats, reduced human control and involvement, and the possibility of conflict between countries.
However, the researchers stressed that these outcomes are not certain. They also noted that technological breakthroughs may take time to translate into real-world systems.
The researchers said AI could also accelerate efforts to reduce these risks. They therefore urged governments to make risk mitigation part of their preparations.
The proposed measures include greater transparency, limits on development speed, cooperation on halting projects involving data centers, isolation of automated systems and emergency response plans.
“Productivity gains from automated AI R&D have not yet reached the threshold needed to trigger an intelligence explosion,” the researchers wrote. However, they said improvements in newer AI systems could be bringing that threshold closer.
