Researchers at the University of Texas at Austin have developed a new artificial intelligence system called the “semantic decoder” that can do this translate brain activity of a person in a continuous stream of text.
This innovative technology has the potential to help people who are mentally alert but unable to develop language, such as those who have been debilitated by stroke.
When listening to a story or imagining the narration silently, the system decodes the brain signals and converts them into text so that people’s ideas and thoughts can be expressed.
This promising success could open new perspectives for communication and quality of life for people who face language problems due to illness or injury.
The study that led to the development of the “semantic decoder” was conducted by Jerry Tang, a graduate student in computer science, and Alex Huth, an assistant professor of neuroscience and computer science at the university.
The results of this research were published in the journal Nature Neuroscience, one of the most respected scientific publications in the field.
Tang and Huth’s joint leadership of the research underscores the collaboration between computer science and neuroscience aimed at significantly advancing the interface between the human brain and artificial intelligence.
The work carried out by the researchers uses a transformer model similar to those used in systems such as Bard by Google and ChatGPT by OpenAI.
However, the system developed by the researchers differs in that it does not require surgical implants in the subjects and is therefore a noninvasive method. Furthermore, unlike other speech decoding systems under development, participants are not limited to a prescribed list of words to be communicated.
How does the mind reading method work?
After extensive decoder training, during which the patient listens to podcasts on the scanner for hours, brain activity is measured using an fMRI scanner.
When the participant is later ready to have their thoughts decoded, the machine can generate the appropriate text from just brain activity, whether it’s listening to a new story or imagining telling a story.
Researchers designed the decoding system to capture the essence of what is being said or thought, rather than providing an exact wordforword transcription.
Although the system is imperfect, it has proven capable of producing texts that are close and partially accurate to the intended meaning of the original words.
The decoder developed by the researchers enables speech to be continuously decoded over long periods of time and includes complex ideas.
About half the time the decoder was trained to monitor a participant’s brain activity, the machine was generating text that reflected the desired meaning of the words, thereby contributing to more effective and understandable communication.
According to Huth, this approach represents a significant improvement over previous methods, which were often limited to single words or short sentences.
The system does not aim for literal wordforword transcription but rather to capture the essence of what is said or thought, albeit imperfectly.
Although the current system is based on the use of a functional magnetic resonance imaging (fMRI) scanner, which limits its applicability outside of the laboratory setting, the researchers believe their work can be adapted for more portable brain imaging systems such as functional nearinfrared spectroscopy (fNIRS).
According to Huth, fNIRS measures blood flow in the brain at different points in time, which is essentially the same type of signal that fMRI detects.
Therefore, the approach used in the study could be applied to fNIRS. Despite this limitation, it is believed that the core of the method developed by the researchers can be adapted for fNIRS, paving the way for a more portable and accessible system for decoding brain activity.