NVIDIA’s Neuralangelo is an AI model that can generate 3D objects from 2D videos

NVIDIA has introduced a new AI model called Neuralangelo that can create 3D replicas of objects from 2D videos, whether they’re classic sculptures or run-of-the-mill trucks and buildings. Neuralangelo works by selecting several frames showing the subject from different angles in a 2D video, so it can a get a clear picture of its depth, size and shape. It then creates a rough 3D representation of the object before optimizing it to mimic the details of the real thing. 

According to the company, the new model has adopted the technology from its old one, the Instant NeRF, to be able to accurately capture the finer details of whatever the user wants to recreate in 3D. Those include its texture, patterns and color variations. NVIDIA says Neuralangelo’s ability to capture tricky textures, such as the roughness of roof shingles and the smoothness of marble, “significantly surpasses prior methods.” It’s apparently even good enough to create usable virtual objects from videos taken by smartphones, which could give creators an easy way to generate 3D recreations for their projects. It’s a quick solution, too: NVIDIA Research told us the tool can create 3D scenes within two hours. 

Ming-Yu Liu, co-author of the paper on Neuralangelo, said it will “eventually enable developers to import detailed objects — whether small statues or massive buildings — into virtual environments for video games or industrial digital twins.” In the video below showing what the technology can do, NVIDIA said it can create large-scale vistas from drone footage, making it easier for developers to recreate real locations in their games. It has a lot of other potential applications, as well, including in robotics, virtual reality experiences, architecture and, of course, art.

The company’s research arm will present Neuralangelo at the Conference on Computer Vision and Pattern Recognition event that’s taking place on June 18th through the 22nd in Vancouver.