Stanford University researchers have developed their own Alpaca AI model, which is similar to ChatGPT. This model would have cost only $600 to train since it was built on Meta’s open-source LLaMA platform. Their experience shows that new advanced AI systems can be easily replicated and do not even require huge costs. However, a new report this week indicates that researchers have taken their AI offline due to severe “hallucinations” and increased costs. Also, the researchers reportedly failed to get the AI to behave well.
Google, Meta, OpenAI, etc. and Microsoft have each released at least one advanced AI system in recent years, and in each case the cost of the product is in the millions of dollars. For example, Microsoft invested several billion dollars in OpenAI this year to maintain its exclusive access to advanced language models developed by the AI lab. In return, this partnership enables OpenAI to leverage the computing power of Microsoft’s Azure cloud required to run ChatGPT and other products. OpenAI can thus optimize its spending on digital infrastructure.
Recently, however, researchers at Stanford University announced that they have developed a cheap AI model that works just like OpenAI’s ChatGPT at just $600. According to the study report, the base cost the researchers $500, and they spent about $100 to create the AI, which took eight computers with 80GB NVIDIA A100 accelerators for three hours; they have “rented” this service in the cloud infrastructure. The researchers based their AI on the LLaMA 7B open language model, which is the smallest and cheapest of the LLaMA series developed by Meta. They named their model AI Alpaca.
Its capabilities are actually quite limited and it performs worse than ChatGPT on most tasks. This is not surprising as training the GPT AI models took more time and resources. ChatGPT has read billions of books, while Alpaca has learned a few questions and answers from humans, albeit few. On the other hand, the AI model Alpaca does some tasks quite well, and sometimes even better than its rival ChatGPT. In the first test, Alpaca would have passed 90 tests (writing emails, posting on social media, helping at work, etc.), while ChatGPT would have passed only 89 tests.
The researchers wrote: Given the small size of the model and the modest amount of instruction trace data, we were quite surprised by this result. In addition to using this set of static evaluations, we also tested the Alpaca model interactively and found that Alpaca often behaves similarly to text-davinci-003 (GPT-3.5) on a diverse set in between. We recognize that our review may be limited in scope and variety. The team believes it probably would have been cheaper if they tried to streamline the process.
The team has published on Github the 52,000 questions used in this study, along with code to generate others and code used to refine the LLaMA model. There are other tweaks to ensure this model works safely and ethically. So what’s stopping someone from building their own AI for a hundred euros and training it at will? In theory, anyone with the appropriate technical training and at least $600 can replicate the Stanford researchers’ experiment. But in reality things are a bit complicated.
The OpenAI license does not allow, or rather forbids, using data from their models to develop competing systems. Meta, on the other hand, grants researchers and academics a non-commercial license to use its models, although this is a moot point since the entire LLaMA model was leaked on 4chan a week after it was announced. Another group claims to have successfully eliminated the cost of the cloud by releasing additional code on Github that can run on a Raspberry Pi and complete the training process in five hours using a single high-end Nvidia RTX 4090 graphics card.
I don’t know what to think of this development. Alpaca is surprisingly very good. The claim here is that the workout can be done in 5 hours on a single RTX 4090. Were GPT-like models democratized overnight?! https://t.co/ysfn5u6xwI
— Carlos E. Perez (@IntuitMachine) March 16, 2023
However, a report published this week says researchers have taken alpaca offline due to increased costs, safety concerns and “hallucinations,” a term the AI community has struggled with. Agreement when a chatbot confidently claims false information and dreams of a fact does not exist. In a press release announcing the first Alpaca launch, lead author Rohan Taori, a Stanford computer science graduate student, acknowledged that there are risks in a public test. However, it’s unclear what exactly went wrong during Alpaca’s interactive demo.
The original purpose of releasing a demo was to spread our research in an accessible way. “We believe we’ve largely achieved that goal, and given the cost of hosting and the inadequacies of our content filters, we’ve decided to withdraw the demo,” said a spokesman for Stanford Human-Centered Artificial Intelligence (Stanford HAI). The department did not immediately respond to a request for comment. You can no longer access a working copy of Alpaca, but the underlying code and data is still live on GitHub.
We encourage users to help us identify new types of bugs by reporting them in the web demo. Overall, we hope that the release of alpaca will facilitate further research into instruction-following patterns and how they align with human values,” the researchers said in the release. Despite its apparent failures, alpaca has some interesting aspects that make the research project worthwhile. Particularly noteworthy are the low acquisition costs in contrast to the multi-million dollar supercomputers from Microsoft.
What does it all mean? This means that an unlimited number of unverified language models can now be implemented, especially by people with machine learning skills who don’t care about terms of service or software piracy – for next to nothing. If a lot of time and money is invested in the post-training phase, and that work can more or less be stolen in the time it takes to answer 50 or 100,000 questions, does it make sense for companies to keep spending that money?
Besides, the impressive abilities of this software could surely come in handy for an authoritarian regime, phishing operation, spammer or any other dubious actor. The genie is out of the bottle and seems extremely easy to reproduce and bring back. Hats off, the experts warn.
Sources: Stanford researchers press release, project GitHub repository, AlpacaPi
And you ?
What is your opinion on the topic?
What do you think of the insignificant cost of creating and training the alpaca AI model?
If it’s so easy to replicate and train ChatGPT, why is Microsoft spending billions on it?
Do you think there is an advantage to injecting so much money?
Do you think it’s a good idea to make Alpaca’s code and underlying data public?
Do you think this could lead to a proliferation of amateur AI models in the next month?
What impact could such a situation have on the internet and the AI sector?
See also
A search using Google’s Bard and Microsoft’s ChatGPT will likely cost ten times more than a keyword search, which could yield billions in rewards
Google engineers created ChatGPT-like AI years ago, but executives have blocked it for security reasons. Your work now underpins Bard
Google opens access to its competitor Microsoft’s ChatGPT and announces the public launch of its chatbot Bard
Bill Gates says AI boom threatens Google’s search engine profits, but Bing Chat and Bard Search are costing billions of dollars