Nvidia researchers have developed an approach to reconstruct a 3D scene almost instantly from a handful of 2D images. They do this by leveraging a popular new technology called Neural Radiance Fields (NeRF), which the company says is sometimes up to 1,000 times faster than competing implementations. NeRF is able to train a small neural network from a lot of photos in a few seconds, provided you know the angles of the shots well. The resulting model is then used to generate all the data needed to create the 3D environment of the scene in question. A calculation that, according to Nvidia, only takes a few tens of microseconds.
A new technology called Neural Radiance Field, or NeRF, involves training AI algorithms to enable the creation of 3D objects from two-dimensional photos. NeRF has the ability to fill in the gaps, so to speak, by interpolating what 2D photos didn’t capture. It’s a trick that could lead to advances in various fields such as video games and autonomous driving. Now NVIDIA has developed a new NeRF technique – which the company says is the fastest yet – that trains and generates a 3D scene in just a few seconds.
It takes just seconds to train the pattern, called Instant NeRF, using dozens of still images and the camera angles from which they were taken. After that, it is able to generate a 3D scene in just ten milliseconds. Like other NeRF techniques, it requires images taken from multiple positions. And for multi-subject photos, preference is given to photos taken without too much movement, otherwise the result would be blurry.
Nvidia provides context for its demo, explaining that previous NeRF techniques could take hours to train a scene and then minutes to render the target scenes. Although results from previous slower implementations have been good, Nvidia researchers using the AI technology have put the pedal to the metal on performance.
When the first instant photo was taken with a Polaroid camera 75 years ago, it was revolutionary to be able to quickly capture the 3D world in a realistic 2D image. Today, artificial intelligence researchers are working on the opposite: turning a collection of still images into a 3D digital scene in a matter of seconds.
The process, known as reverse rendering, uses AI to approximate how light behaves in the real world, allowing researchers to reconstruct a 3D scene from a handful of 2D images taken from different angles. The NVIDIA research team has developed an approach that accomplishes this task almost instantly, making it one of the first models of its kind to combine lightning-fast neural network training and fast rendering.
NVIDIA has applied this approach to a popular new technology called Neural Radiance Fields, or NeRF. The result, dubbed Instant NeRF, is the fastest NeRF technique to date, reaching over 1,000 accelerations in some cases. The model takes just a few seconds to train a few dozen still images – plus data on the camera angles from which they were taken – and can then render the resulting 3D scene in tens of milliseconds.
While traditional 3D representations like meshes resemble vector images, NeRFs are like bitmaps*: they densely capture how light emanates from an object or within a scene, explains David Luebke, vice president of graphics research at NVIDIA. In that sense, Instant NeRF could be as important to 3D as digital cameras and JPEG compression are to 2D photography, dramatically increasing the speed, ease, and scope of 3D capture and sharing.
Presented in a session at NVIDIA GTC this week, Instant NeRF could be used to create avatars or scenes for virtual worlds, capture video conference participants and their surroundings in 3D, or reconstruct scenes for digital maps. As a tribute to the early days of Polaroid images, NVIDIA Research has recreated an iconic photo of Andy Warhol taking an instant photo and turning it into a 3D scene using Instant NeRF.
What is NeRF?
NeRF uses neural networks to represent and render realistic 3D scenes based on an input collection of 2D images.
Gathering data to feed a NeRF is a bit like being a red carpet photographer trying to capture a celebrity’s outfit from all angles – the neural network needs a few dozen images, those of several Positions around it were recorded, as well as the position of the scene the camera shot from each of them.
In a scene with people or other moving objects, the faster these shots are, the better. If there is too much movement during the 2D image capture process, the 3D scene generated by the AI will be blurred.
From there, a NeRF essentially fills in the gaps and trains a small neural network to reconstruct the scene by predicting the color of light emanating in each direction from each point in 3D space. The technique can even bypass occlusions – when objects seen in some images are blocked by obstacles such as pillars in other images.
1000x acceleration with Instant NeRF
Although estimating the depth and appearance of an object based on a partial view is a natural skill for humans, it is a challenging task for AI.
Creating a 3D scene using traditional methods takes hours or more, depending on the complexity and resolution of the visualization. Introducing AI to the picture speeds things up. Early NeRF models rendered crisp, artifact-free scenes in minutes, but it still took hours to practice.
Instant NeRF, however, reduces render time by several orders of magnitude. It is based on a technique developed by NVIDIA called multi-resolution hash grid encoding, which is optimized to run efficiently on NVIDIA GPUs. By using a new method to encode input, researchers can produce high-quality results using a tiny neural network that operates quickly.
The model was developed using the NVIDIA CUDA toolkit and the Tiny CUDA Neural Networks library. Because it’s a lightweight neural network, it can be trained and run on a single NVIDIA GPU—and runs faster on boards with NVIDIA Tensor Cores.
The technology could be used to train robots and self-driving cars to understand the size and shape of real-world objects by taking 2D images or video recordings of them. It could also be used in architecture and entertainment to quickly create digital representations of real-world environments that can be modified and augmented by the developers.
Beyond NeRF, NVIDIA researchers are investigating how this input encoding technique could be used to accelerate various AI challenges, including reinforcement learning, language translation, and the general use of deep learning algorithms.
Source: NVIDIA
And you ?
Have you heard of NeRF before? What do you think ?
What do you think of Nvidia’s technological prowess?
What possible applications do you see?
See also:
Facebook falls behind Nvidia in market cap and is now the eighth most valuable American company, its value falling below $600 billion
NVIDIA blocks cryptocurrency mining on RTX 3080 and 3070 cards to make them more accessible for PC gamers