NVIDIA unveils Neuralangelo: AI model that transforms 2D videos into detailed 3D structures

NVIDIA Research has introduced Neuralangelo, an AI model that converts 2D video clips into intricate 3D structures. This technology generates lifelike virtual replicas of real-world objects, enabling creative professionals to import them into design applications for various purposes. Neuralangelo surpasses previous methods by accurately translating textures, such as roof shingles and marble, from 2D videos to 3D assets. It facilitates the rapid creation of usable virtual objects using smartphone footage.

Ming-Yu Liu, senior director of research, believes Neuralangelo will greatly benefit creators by allowing them to recreate the real world in digital environments for video games and industrial digital twins. The model has successfully reconstructed objects ranging from Michelangelo's David to a flatbed truck, along with building interiors and exteriors.

Neuralangelo incorporates instant neural graphics primitives, enhancing its ability to capture repetitive texture patterns and strong color variations. The AI analyzes 2D videos from different angles to create a rough 3D representation, which is then optimized to sharpen details. The result is a highly detailed 3D object or scene suitable for virtual reality applications, digital twins, and robotics development.

Neuralangelo is one of NVIDIA Research's nearly 30 projects to be showcased at the Conference on Computer Vision and Pattern Recognition (CVPR), covering diverse topics such as pose estimation and video generation. Another notable project, DiffCollage, utilizes diffusion methods to create large-scale content without requiring training on images of the same scale.