Decision making and intensive knowledge search are two essential skills for agents of large natural language systems in unfamiliar contexts. OpenAI’s GPT-3 and Google’s PaLM are just two examples of LLMs that have performed impressively in various benchmark tests. The ability of these models to understand tasks in specific environments, comparable to those of humans, represents a major advance in natural language processing.
Agents can overcome high syntactical barriers that could lead to false negative errors in complex tasks when based on natural language. However, due to their large and often unbounded state spaces, natural language RL agents pose a significant challenge for learning optimal policies.
Various decision-making approaches have been proposed to help natural language agents make decisions in a textual environment without the benefit of a learned strategy. However, with longer sequences, the model becomes more susceptible to hallucinations, which reduces the accuracy of these methods as the number of subtasks increases.
Natural language agents can solve tasks more intuitively thanks to the advanced human-like properties of large-scale LLMs. Human-in-the-loop (HITL) methods have been widely used to improve performance by redirecting agent reasoning tracking after errors. Although this method improves performance with little human involvement, it is not self-contained as it requires trainers to monitor the trajectory at each time interval.
The researchers from Northeastern University and the Massachusetts Institute of Technology believe that if LLMs were able to autonomously close the trial-and-error loop, they would make good use of natural language-based self-optimization.
To test their hypothesis, the team implemented an autoreflective LLM and simple heuristics to identify hallucinations and ineffective execution of actions within an LLM-based agent, using an approach called reflection. Then they challenge the agent against two different learn-by-error benchmarks – AlfWorld, which is text-based, and HotPotQA, which answers questions. The result is increased efficiency in decision making and other knowledge-based tasks.
This article shows that it is possible to improve GPT-4 performance by 30% by asking it to think about the following question: “Why did you make a mistake?” and generate a new message taking this into account basic until it is correct.
ReAct’s problem-solving technique is enhanced by the agent’s reflection ability to reflect its performance, resulting in a 97% detection rate on the AlfWorld benchmark in just 12 autonomous trials. This is a significant improvement over the 75% accuracy achieved with the ReAct base agent. 100 questions were extracted from HotPotQA and a reflection-based ReAct agent was tested. Compared to a basic ReAct agent, the agent outperformed it by 17% by iteratively refining its search and retrieving content based on clues from its memory. It is important to note that Reflection is not designed to achieve near perfect accuracy values. rather, it aims to show how trial and error learning can facilitate the discovery of tasks and environments previously thought impossible.
The team suggests that their reflection approach can be applied to more complex problems, such as when the agent needs to learn how to generate new ideas, explore new state spaces, and develop more precise plans of action based on its knowledge of the history of its experience.
Reflection: an autonomous actor endowed with dynamic memory and the ability for self-reflection
Recent advances in Large Language Model (LLM) decision makers have shown impressive performance on various benchmarks. However, these state-of-the-art approaches typically require inner model tuning, outer model tuning, or policy optimization over a defined state space. Implementing these methods can be difficult due to the lack of high-quality training data or the lack of a well-defined state space. In addition, these agents lack certain qualities inherent in human decision-making, most notably the ability to learn from mistakes. Self-reflection empowers people to effectively solve new problems through a process of trial and error. Building on recent research, we propose reflection, an approach that provides an agent with dynamic memory and self-reflection skills to enhance their existing thought trajectory and task-specific action decision-making skills. To achieve full automation, we introduce a simple but effective heuristic that allows the agent to detect cases of hallucinations, avoid repetition in action sequences, and in some environments create an internal memory map of the given environment. To evaluate our approach, we assess the agent’s ability to perform decision-making tasks in AlfWorld environments and knowledge-intensive, research-based question-and-answer tasks in HotPotQA environments. We observe success rates of 97% and 51% respectively and discuss the emerging quality of self-reflection.
Sources: Article “Reflection: an autonomous agent with dynamic memory and self-reflection”, GitHub
And you ?
What is your opinion on the topic?
Do you think that in more or less near future a general artificial intelligence could see the light of day?
See also
Microsoft claims GPT-4 shows sparks of general artificial intelligence, we believe the intelligence of GPT-4 signals a real paradigm shift
AI researchers say OpenAI’s GPT-4 language model could pass the bar exam, sparking debate about replacing lawyers and judges with AI systems
GPT-4 successfully tricked a human into solving a CAPTCHA after pretending to be blind. OpenAI has touted this version of ChatGPT as better and more accurate