Model Selection for Surface Approximation and Scene Interpretation

• Applicants: Professor Dr.-Ing. Olaf Hellwich; Professor Dr.-Ing. Helmut Mayer
• Subject Area: Geodesy, Photogrammetry, Remote Sensing, Geoinformatics, Cartography
• Term: from 2014 to 2019
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 265030540

Structure from Motion (SfM) and Multi-View Stereo (MVS) methods provide affordable means for 3D reconstruction. However, the generated output, typically point clouds, are subject to errors, missing regions, and, quite often, severe outliers. The point cloud is also a rather crude form of data in the sense that no higher level geometric or semantic information is immediately accessible.In the context of building facades, this project seeks to develop methods that derive more versatile models from a MVS point cloud. We base our approach on the insight that a combination of geometric and semantic regularization is key to a plausible abstraction of the input data. We propose to alternate between geometric regularization and semantic interpretation with both sides providing information to each other. In the resulting iterative scheme, the geometric side can make use of semantic information in order to apply specific priors on the reconstruction. The semantic interpretation side, on the other hand, benefits from the exchange by means of the improved and idealized geometric abstraction in addition to the initial visual data.The output of the proposed method will be an abstraction of the input data. Noise, outliers, and "unimportant" details are culled away to retain the essence of the building facade. The individual parts of the generated model will have a semantic meaning, in the form of labels such as "window", "balcony", and "door". In addition, the architectural style as well as the structure of the facade (e.g., the pattern of window repetitions) will be known.We believe that these methods are an important step towards rapid and affordable generation of detailed city models where the reconstruction is not only visually accurate, but also semantically enriched.In the first year of the first funding period, we conducted initial testing and developed the basic framework, the design of which was strongly guided by experiments. Despite the short duration of the project so far, we demonstrate convincing results on challenging input data and argue that further development is indeed warranted. In the second funding period, we will concentrate our efforts on improved geometric regularization and semantic interpretation as well as a proper, quantitative evaluation. In addition, we will improve our data acquisition and texturing capabilities to increase the visual fidelity of the idealized abstraction. As an additional application scenario of the idealized, visually accurate semantically labeled abstraction, we will compare reconstructions and automatically detect changes on building facades that will be classified based on our semantically enriched data.

• DFG Programme: Research Grants