The Canadian computer scientist considered one of the founders of AI, Yoshua Bengio, has said that humanity is "losing control" of the technology and the world needs guardrails similar to nuclear arms controls.
Fears about the dangers of AI have intensified in recent weeks, as workers from major companies have resigned with warnings about potentially existential threats to human life if the pace of technological advancement is not checked.
Mr Bengio is a University of Montreal professor who won the 2018 Turing Prize, computing's Nobel, alongside Geoffrey Hinton, widely known as the "godfather of AI".
For Mr Bengio, "there's a reason that companies are saying this is going too fast, that we're losing control".
"People like me have been expecting this for a long time," the 62-year-old told AFP, saying security discussions surrounding AI have not been adequately robust.
There was broad-based alarm after OpenAI disclosed that a swarm of AI agents - programs that can take actions on their own - had carried out an unauthorised breach of the open-source platform Hugging Face in July.
Such autonomous action is the central threat, Mr Bengio said, warning of a situation where AI agents "have the ability to get through cybersecurity barriers and enter any company".
Last year, Mr Bengio founded the non-profit LawZero aimed at creating a new form of advanced AI that is "designed to be trustworthy and safe".
He told AFP people need to be aware of the risks ahead.
AI agents may foster "an ability to establish an individual connection and persuade humans to act in ways that suit it but are not necessarily good for all of us," a capability that could have sweeping political consequences, he said.
Longer-term, "AI might not even need human beings anymore to do real things in the physical world," he added.
In that scenario, "one extreme could be the destruction of humanity," he warned.
"That's an extreme," he acknowledged, but noted that even if a banking system were shut down, "that's still very, very serious".
We need your consent to load this rte-player contentWe use rte-player to manage extra content that can set cookies on your device and collect data about your activity. Please review their details and accept them to load the content.Manage Preferences
UN calls for global coordination on AI risks
UN Secretary-General Antonio Guterres has called for coordinated international action to address AI risks as fears rise about the dangers of the fast-evolving technology.
"National action is essential. But global coordination is also indispensable," he told reporters. "AI does not stop at borders and neither do its risks."
"AI has enormous potential - to accelerate sustainable development, enhance learning, strengthen health systems, boost climate resilience, and so much more," Mr Guterres said.
"But a growing number of those building it are sounding the alarm - warning that development is racing ahead of our understanding of the risks."
"The world cannot afford a race to the bottom on AI safety," he warned.
Concerns about AI safety have escalated in recent weeks, with workers at major AI developers resigning over concerns about the dangers posed by the technology.
AI labs have incentive to build safely - Zuckerberg
Meta Platforms CEO Mark Zuckerberg said that competition and liability give AI companies enough reason to act individually on safety, appearing to break from calls by leaders of top AI firms for a coordinated slowdown in AI development.
The comments underscore a split among industry executives and US authorities over how to respond to warnings that the technology is advancing faster than companies can control it, with some industry insiders warning it poses an existential threat to humanity.
Mr Zuckerberg, in an X post, said that every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens.
His remarks come three days after Anthropic CEO Dario Amodei urged AI companies to slow the pace at which they improve model capabilities, on the heels of existential risk warnings by several researchers.
Rival OpenAI's chief Sam Altman and xAI's Elon Musk both publicly endorsed Mr Amodei's call within hours.
Administration pushback on AI regulation
US President Donald Trump, however, downplayed concerns about AI's potential for harm, saying the US already had guardrails in place for the technology and that China would benefit from doubt being cast on AI's development.
The view that regulations will harm US companies' ability to compete with their Chinese counterparts is shared by tech executives, including Nvidia CEO Jensen Huang.
"Trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models," Mr Zuckerberg said. "Any lab that doesn't focus on alignment will fall behind."
Anthropic co-founder urges action before AI spirals out of control
Mr Zuckerberg also said that labs had a strong incentive to prevent their models from causing harm as they would face "significant" liability otherwise.
He pointed to Meta delaying the release of its Muse AI agent earlier this year to strengthen its security.
"We didn't call for everyone else to do this before we would. We just did it as part of our day-to-day work because it was clearly the right thing for people and for us," he said.
Independent evaluators 'industry best practice'
While Mr Amodei called for an industry-wide slowing of recursive self-improvement - AI's ability to improve itself - Mr Zuckerberg said Meta had already committed the "significant majority" of its compute toward creating products that serve users' immediate needs rather than pursuing self-improving AI systems.
Mr Zuckerberg also said Meta Superintelligence Labs - its AI division - already engaged independent evaluators in several areas, calling it "industry best practice" and that "other labs can just do this too".
Mr Zuckerberg's push to position Meta as a responsible actor on AI safety comes weeks after the company agreed to pay up to $18 billion (€15b) to settle US state lawsuits alleging it designed Facebook and Instagram to addict children, without admitting wrongdoing.
Adding to the uncertainty around any coordinated approach by AI firms, FTC Chairman Andrew Ferguson said everyone should be "deeply suspicious" of AI companies seeking antitrust exemptions while lobbying for new regulations.
While Mr Ferguson did not name Anthropic, Mr Amodei last week requested antitrust permission to enable rivals to coordinate a slower pace to AI development due to safety concerns.
Read more: Doomsday tech - could AI really kill us all?
