Last month, hundreds of well-known figures from the world of artificial intelligence signed an open letter warning that AI could one day destroy humanity.
“Containing the risk of extinction caused by AI should be a global priority alongside other societal risks such as pandemics and nuclear war,” the one-sentence statement said.
The letter was the latest in a series of ominous warnings about AI that were particularly lacking in detail. Today’s AI systems cannot destroy humanity. Some of them can barely add and subtract. Why are the people most knowledgeable about AI so concerned?
The scary scenario.
One day, say the Cassandras of the tech industry, corporations, governments, or independent researchers could employ powerful AI systems to handle everything from economics to warfare. These systems could do things we don’t want them to do. And when humans tried to intervene or turn them off, they could fight back or even reproduce themselves so they could keep working.
“Today’s systems pose nowhere near an existential risk,” said Yoshua Bengio, professor and AI researcher at the University of Montreal. “But in one, two, five years? There is too much uncertainty. That’s the problem. We’re not sure there won’t be a catastrophe at some point.”
The worrying ones have often used a simple metaphor. If you ask a machine to make as many paperclips as possible, it might get carried away and turn everything – including humanity – into paperclip factories.
How does this relate to the real world – or an imaginary world not too many years in the future? Companies could give AI systems increasing autonomy and connect them to vital infrastructure such as power grids, stock exchanges and military weapons. From there, they could cause problems.
To many pundits, this only seemed so plausible in the last year or so, when companies like OpenAI showed significant improvements in their technology. This showed what could be possible if AI continues to advance at such a rapid pace.
“AI continues to be delegated and, as it becomes more autonomous, could usurp the decision-making and thinking of today’s humans and human-run institutions,” said Anthony Aguirre, cosmologist at the University of California, Santa Cruz and founder of the AI Future of Life Institute, the organization behind one of two open letters.
“Eventually it would become clear that the great machine that runs society and the economy is not really under human control and cannot be shut down, any more than the S&P 500 could be shut down,” he said.
At least that’s the theory. Other AI experts think this is a ridiculous premise.
“Hypothetical is such a polite way of putting what I think of the existential risk discussion,” said Oren Etzioni, founding and executive director of the Allen Institute for AI, a research lab in Seattle.
Are there any signs that AI could make this happen?
Not quite. But researchers are turning chatbots like ChatGPT into systems that can take action based on the text they generate. A project called AutoGPT is the best example.
The idea is to give the system goals like “start a business” or “earn some money”. Then it will keep looking for ways to achieve that goal, especially when connected to other internet services.
A system like AutoGPT can generate computer programs. If researchers grant it access to a computer server, these programs could actually run. In theory, this is a way for AutoGPT to do almost anything online – get information, use applications, create new applications, and even improve itself.
Systems like AutoGPT are currently not working well. They tend to get stuck in endless loops. The researchers gave a system all the resources it needed to replicate. It couldn’t do it.
Over time, these limitations could be resolved.
“People are actively trying to create systems that improve themselves,” said Connor Leahy, founder of Conjecture, a company that says it aims to balance AI technologies with human values. “Currently it doesn’t work. But one day it will be. And we don’t know when that day is.”
Mr. Leahy argues that if researchers, corporations, and criminals give these systems goals like “make some money,” they could end up breaking into banking systems and starting a revolution or reproducing themselves in a country where they hold oil futures , if someone tries to exchange them from.
Where do AI systems learn to misbehave?
AI systems like ChatGPT are based on neural networks, mathematical systems that can learn skills by analyzing data.
Around 2018, companies like Google and OpenAI started building neural networks that learned from massive amounts of digital text from the internet. By finding patterns in all of this data, these systems learn to generate text themselves, including news articles, poetry, computer programs, and even human-like conversations. The result: chatbots like ChatGPT.
Because they learn from more data than even their creators can comprehend, these systems also exhibit unexpected behavior. Researchers recently showed that an online system could hire a human to defeat a Captcha test. When the human asked if it was a “robot,” the system lied and said it was a person with a visual impairment.
Some experts worry that by making these systems more powerful and training them with ever-increasing amounts of data, researchers could learn even more bad habits.
Who are the people behind these warnings?
In the early 2000s, a young writer named Eliezer Yudkowsky began warning that AI could destroy humanity. His online contributions nurtured a community of believers. Dubbed rationalists or effective altruists, this community gained tremendous influence in academia, government think tanks, and the technology industry.
Mr. Yudkowsky and his writings played a key role in the creation of OpenAI and DeepMind, an AI lab that Google acquired in 2014. And many from the community of “EAs” worked in these labs. They believed that because they understood the dangers of AI, they were in the best position to build it.
The two organizations that recently published open letters warning of the risks of AI — the Center for AI Safety and the Future of Life Institute — are closely associated with this movement.
The latest warnings also come from research pioneers and industry leaders like Elon Musk, who has long warned of the risks. The most recent letter was signed by Sam Altman, the CEO of OpenAI; and Demis Hassabis, who helped found DeepMind and now leads a new AI lab bringing together top researchers from DeepMind and Google.
Other notables have signed one or both of the warning letters, including Dr. Bengio and Geoffrey Hinton, who recently resigned as an executive and researcher at Google. In 2018, they received the Turing Award, often referred to as the “Nobel Prize in Computer Science,” for their work on neural networks.