Jim Fan, a senior researcher at Nvidia, believes AI has the potential to radically transform video games. He believes that rapid advances in AI are opening up new frontiers for video games: infinite open worlds, unique content, dynamic dialogue with non-player characters (NPCs), and the ability to accelerate game development. He added that AI and games were born for each other and their happy marriage is just beginning. Nvidia introduced an AI platform called Avatar Cloud Engine for Games earlier this year that allows players to converse with non-playable characters.
Jim Fan is an AI researcher whose research focuses on fundamental multimodal models, reinforcement learning, computer vision, and large-scale systems. He is a senior researcher at Nvidia and is responsible for the company's AI agent initiative. He explained that his mission is to develop skilled and competent agents in the physical (robotics) and virtual (games, simulation) worlds. He received his doctorate from the Stanford Vision Lab and completed an internship at OpenAI (with Ilya Sutskever and Andrej Karpathy), but also at Baidu AI Labs and at the Institut québécois d'IA (MILA), founded by Yoshua Bengio.
Before we have a million robots in the physical world, we will first see a billion embodied agents in virtual worlds. Gaming is the second major area I will devote myself to in 2024. AI and gaming were born for each other and their happy marriage is just beginning.
On the one hand,…pic.twitter.com/60ofChrbov
– Jim Fan (@DrJimFan) December 28, 2023
Fan believes that recent advances in AI, including large language models (LLM) and generative AI, will usher video games into a new era. He led Voyager (the first AI agent to play Minecraft efficiently and continually develop its skills), MineDojo (an open-ended agent that learns by watching hundreds of thousands of Minecraft videos on YouTube), and Eureka (a five-fingered robotic hand that is extremely acted cleverly). tasks such as turning a pen) and VIMA (one of the first fundamental multimodal models for robotic manipulation).
In a post on X on Thursday, Fan said 2024 will be a turning point for the video game industry. He claims that ongoing efforts to integrate MLLs into video games will change the industry once and for all, making non-playable characters “alive” and interacting with people much more fluidly and cohesively. Below are fan predictions for 2024 and the future of video games:
Sent by Jim Fan
Before we have a million robots in the physical world, we will first see a billion agents embodied in virtual worlds. Gaming is the second major area I will focus on in 2024. AI and games were born for each other, and their happy union is just beginning.
On the one hand, open games represent a “primal soup” for the emergence of general AI. An agent's capabilities are limited by the complexity of the world in which he lives. Minecraft is a great example of this. In 2023 we saw an explosion of new algorithms made possible by Minecraft. To name just a few:
– MineCLIP (from our team's open source MineDojo framework): Learn reward features by watching 100,000 gameplay videos on YouTube.
– VPT (OpenAI): imitates behavior through pseudo-labeling actions on YouTube.
– Travel: lifelong learning in the context of an explicit code skills library.
– STEVE-1: Guides the actor to follow commands in MineCLIP's latent space, inspired by DALLE-2.
– And many others: DEPS, Jarvis-1, DreamerV3…
In addition to Minecraft, there are many other games that require extremely advanced perception, agility, exploration, logical thinking and planning. We're just scratching the surface.
I believe that games (and simulations in general) will provide the next trillions of high-value tokens for training our base models. The great thing is that these tokens are actively selected by the agent itself through exploration. He may choose to experiment with things that minimize his inner insecurities – similar to human curiosity.
On the other hand, AI will bring about a paradigm shift in the gaming sector. This year we're seeing renewed community interest in Stanford Smallville, where 25 AI agents live in a digital city. They go to work, gossip, organize social activities, make new friends and even fall in love.
It's an exciting experience, but we haven't felt any impact on actual games yet. This is because our LLMs are too boring and too expensive. If you take a look at the Smallville chat log, you'll see that the conversations aren't fun at all. No parent speaks so politely to their children. The unit economics of deploying such a large number of agents also makes no sense at scale.
Still, I think 2024 is a turning point. The digital world of Westworld is coming and will change the industry once and for all. The games will be truly lively. The characters will interact with people and each other, forming relationships, acting consistently throughout their lives, and reacting in human ways. Every game will have infinite replay value and every player will have a unique and personalized experience.
NPCs generally have a fixed pattern of behavior and their behavior and speech are stilted and unnatural. However, Fan believes that generative AI will help video game developers create a greater variety of original dialogue for NPCs. Some publishers are already using generative AI in NPCs to make conversations more realistic. Chinese gaming company NetEase is using ChatGPT to generate NPC dialogue in its mobile game Justice, while Replica Studios, an Australian AI company, recently introduced “smart AI-driven NPCs” for giant game engines Unreal Engine .
This allows game developers to use AI to read out NPC dialogue instead of having to hire a human voice actor. However, Julian Togelius, an associate professor at New York University, said that using AI-generated NPCs carries a risk: game publishers could lose control of the game's narrative. Togelius researches AI and games at New York University. He explains: “NPCs can say things that are game-changing, rude, or otherwise disruptive to the gaming experience.” He also said that creating AI-controlled NPCs could prove to be more complicated than one might think.
In his opinion, it's also much harder to create “good” AI-generated NPCs to help the player than to create enemies to fight you. “We haven't seen much progress in AI that animates other characters in games, that tries to model the player or create the world. So we will see great progress in these directions,” he said. Additionally, some critics claim that the integration of generative AI into video games could negatively impact a large portion of multiplayer games by making levels unplayable or diverging from the game's storyline.
Open-world games like “Grand Theft Auto,” “Skyrim,” and “Elden Ring” approach gameplay with non-linear quests and stories. Generative AI therefore has the potential to reshape the gaming experience. Togelius imagines a “vast open game” with endless possibilities, new cities, new people and new landscapes, each with its own story and interactive elements. In his opinion, generative AI could potentially create unique scenarios and tailored quests for players of the right level using data collected about the player from previous playthroughs.
However, if developers retain control over procedurally generated content, generative AI has the potential to develop unplayable levels or inadvertently deviate from the game's narrative. “Games have functional limitations: levels must be lockable and NPCs must not be in the game world,” he tells Togelius. According to other critics, hallucinating AI models could also cause a real headache for video game developers. This could cause NPCs to deviate from the game's storyline and thus contribute to a deterioration in the gaming experience.
Source : Jim Fan
And you ?
What is your opinion on this topic?
What do you think about integrating generative AI into video games?
What risks do you think are associated with integrating generative AI into video games?
See also
Nvidia is introducing an AI platform called “Avatar Cloud Engine for Games” that allows players to converse with non-playable characters
With AI, you can now have real conversations with NPCs in video games: in the case of Matrix Awakens. This development could bring with it a variety of possible scenarios
The generative AI revolution in the gaming industry: a path to limitless creativity, with 20% of professionals considering cost reductions