Bathymetric 3D reconstruction with Neural Radiance Fields

• Applicant: Professor Dr.-Ing. Boris Jutzi
• Subject Area: Geodesy, Photogrammetry, Remote Sensing, Geoinformatics, Cartography
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 538522540

1) Motivation and research context. Underwater vegetation is an important indicator for climate change. To date, no cost-efficient methods for monitoring littoral vegetation exist. Bathymetric laser scanners mounted on Uncrewed Aerial Vehicles (UAV) deliver dense and accurate 3D point clouds of both the water bottom surface and submersed vegetation, but data acquisition is expensive. In contrast, high-quality UAV images are increasingly available, but complex underwater scene reconstruction often fails when applying multimedia photogrammetry based on established Structure-from-Motion and Multi-View Stereo (MVS). 2) Research questions and objectives. Recent developments in Neural Radiance Fields (NeRFs) showed a potential for purely image-based 3D reconstruction of complex objects. Promising results have been achieved for above-water vegetation, but it is still unclear whether NeRFs can deliver an accurate point cloud representation of complex underwater scenes. Thus, the main research objectives are (i) to improve 3D underwater object reconstruction for both submerged bottom surfaces and vegetation from images acquired from the air by extending NeRF-based deep learning techniques and to (ii) evaluate the results quantitatively. 3) Methods. We propose three NeRF-variants: In the first approach, we split the object space in two parts above and below a planar water surface and use a separate NeRF for each part. The conventional above-water NeRF is combined with an underwater NeRF, which takes the air-water intersection as starting point and considers the refracted underwater image ray direction rather than the original image ray direction. For the underwater NeRFs, the image rays are treated separately and each ray acts as a training sample. In the second approach, applicable to running water with standing waves, we also consider a planar water surface, but use locally determined surface normal vectors for refraction correction. The local surface normal vectors are learned in the optimization process. Finally, the third approach attempts indirect estimation of refraction by extending the NeRF with an optical density parameter. The ray-tracing is thereby modified so that refraction occurs in differential steps along the ray. The resulting point clouds are evaluated against (i) conventional MVS and (ii) bathymetric LiDAR as independent reference. In addition, we also try to derive underwater vegetation metrics directly from the NeRFs or 3D point clouds.

• DFG Programme: Research Grants
• International Connection: Austria
• Partner Organisation: Fonds zur Förderung der wissenschaftlichen Forschung (FWF)
• Cooperation Partner: Professor Dr.-Ing. Gottfried Mandlburger