GSOC 2024: NeRF and Differentiable GNS for evaluating physical properties from videos

[kks32]

Evaluating physical properties from videos using NeRF and differentiable GNS

Abstract

This project proposes an innovative approach to evaluate the physical properties of dynamic events, such as fluid dynamics and granular flow, from video data using Neural Radiance Fields (NeRF) using Nvidia Instant NGP and a Differentiable Graph Network Simulator (GNS). By converting video footage into high-fidelity 3D models with NeRF and simulating physical interactions with Differentiable GNS, this framework aims to predict and analyze the complex behaviors of natural phenomena with unprecedented accuracy. The project aligns with cutting-edge research in computational physics and machine learning, aiming to integrate with existing systems to offer a powerful tool for the simulation and understanding of granular and fluid flows and material properties.

Intensity	Priority	Involves	Mentors
Moderate (350 hours - duration)	High	Integrating NeRF with Differentiable GNS for advanced simulations of physical phenomena. Developing and validating models for granular and fluid flows and material property evaluation with high fidelity.	Krishna Kumar and Yongjin Choi

Benefits of Working on This Project

Participants will significantly broaden their expertise in:

• Advanced machine learning models and their application in physics simulations.
• Handling and processing video data for 3D model generation.
• Simulation of physical properties and behaviors using graph-based neural networks.
• Interdisciplinary skills bridging computational physics, computer vision, and machine learning.

Motivation

Traditional methods for analyzing physical properties from video data often involve labor-intensive processes with limited accuracy and scalability. Integrating NeRF and Differentiable GNS presents a transformative solution, leveraging the power of neural radiance fields for detailed 3D reconstructions and graph network simulators for simulating physical interactions based on these reconstructions. This methodology opens new avenues for the real-time analysis of complex phenomena, enhancing the predictive capabilities and understanding of material behaviors in natural and engineered systems.

Technical Details

• Neural Radiance Fields (NeRF): A deep learning technique for creating detailed 3D representations of scenes from a collection of 2D images, offering a powerful tool for accurate volume rendering and scene reconstruction.
• Differentiable Graph Network Simulator (GNS): A framework for simulating the dynamics of complex systems through differentiable physics, enabling the prediction of physical properties and behaviors in a computationally efficient manner.
• The project aims to seamlessly integrate NeRF-generated 3D models with Differentiable GNS to simulate and analyze physical phenomena directly from video data, bridging the gap between observation and simulation.

Benefits to Project/Community

By advancing the capabilities for evaluating physical properties from videos, this project offers significant contributions to:

• The development of more accurate and dynamic models for understanding natural phenomena and material properties.
• The enhancement of digital twin technologies for real-world applications, from design to optimizing engineering processes.
• The research community provides a novel methodology for data-driven simulations and analysis.

Helpful Experience

Ideal candidates should have:

• Proficiency in Python and familiarity with deep learning frameworks such as TensorFlow or PyTorch.
• Experience with computer vision techniques and 3D reconstruction.
• An interest or background in computational physics and simulations.

First Steps

Interested participants are encouraged to:

• Explore the foundational concepts behind NeRF and its applications in 3D scene reconstruction. Read foundation paper
• Familiarize themselves with the Graph Network Simulator framework and its potential for simulating physical phenomena.
• Investigate existing datasets and tools for NeRF and GNS simulations to understand the workflow and integration process between 3D reconstruction and physical simulation.