Semantic segmentation of laser scanning point clouds for digital building modeling using Bayesian neural networks for uncertainty quantification (PointSemSeg+)

• Applicant: Professor Dr.-Ing. Jörg Blankenbach
• Subject Area: Geodesy, Photogrammetry, Remote Sensing, Geoinformatics, Cartography ; Structural Engineering, Building Informatics and Construction Operation
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 501834640

The automatic derivation of semantic-rich digital building inventory models from 3D point clouds (scan-to-BIM) follows a progressive multi-step process (data acquisition, data pre-processing, semantic segmentation, geometry fitting and object creation).The progress in the field of machine learning (ML) and in particular deep neural networks (deep learning, DL) opens up new potentials to further automate the scan-to-BIM workflow and replace time-consuming manual work steps. However, a key challenge that has not been solved yet is the uncertainty quantification of the result – especially for the semantic segmentation as core step of the scan-to-BIM process. Existing ML/DL approaches are not able to quantify the quality of semantic point cloud segmentation. This is due to object- and use case-specific variations, for example caused by (random) measurement deviations, gaps resulting from occlusions or inaccessible areas, deviations due to unfavorable acquisition conditions, extrinsic perturbations, or environmental effects. The result is therefore an incomplete or in parts erroneously segmented and classified point cloud, which would inevitably also influence the subsequent automated model generation as well.However, especially for the derivation of digital twins for engineering applications (e.g. SHM, AM), the knowledge of the quality or fuzziness of the underlying digital model is of immense importance, which requires manual checking and post-processing of the automatic segmentation result by a human operator before being used during geometry fitting.In this project investigations into the uncertainty of semantic point cloud segmentation will be carried out using novel approaches from the field of deep learning. Therefore it is investigated at the input data level whether the uncertainty of semantic segmentation can be reduced by supplementary feature information and at the model level if it can be quantified by using Bayesian neural networks (BNN). For this purpose 3D point clouds are captured with geodetic laser scanners - also mounted on unmanned aerial vehicles (UAV) - which also contain additional supplementary radiometric information besides the 3D point information.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2388: Hundred plus - Extending the Lifetime of Complex Engineering Structures through Intelligent Digitalization