Point Cloud Upsampling Using Sparse Frequency Models

• Applicant: Professor Dr.-Ing. André Kaup
• Subject Area: Communication Technology and Networks, High-Frequency Technology and Photonic Systems, Signal Processing and Machine Learning for Information Technology ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 532402151

Due to the increasing demand of capturing our environment for virtual and augmented reality applications, in autonomous driving, in archaeology and architecture, the use of point cloud data significantly emerges. Point clouds are a three-dimensional data type. For every point in a point cloud, the three-dimensional coordinates and eventually an associated attribute is stored. Attributes can be anything from color to normal vectors or intensity values. Thus, point clouds are predestined for a versatile use in various application scenarios. However, one drawback of point clouds is the complex capturing process, as each point has to be captured individually. Thus, interpolation techniques that artificially generate a high-resolution point cloud from a low-resolution input point cloud are necessary. This problem is referred to as point cloud upsampling. Current state-of-the-art techniques hold various drawbacks. Some require the points to be located on a rasterized grid and others require normal vectors as attributes. These are not necessarily available, and the computation of normal vectors might hold inaccuracies as point cloud data is usually noisy. Data-driven approaches overcome this drawback but are mainly trained for single use cases. Thus, a generalization to new data sets and other scaling factors of the point cloud is not possible without further training. Hence, the goal of this project is to improve the upsampling of point clouds. Therefore, sparse frequency models will be incorporated. These are generated in an iterative and block-based manner and could already show promising results for upsampling of the geometry and color attributes of point clouds. The upsampling of the geometry will be adapted to the underlying point cloud in order to further improve the upsampling quality. Moreover, the geometry information will be induced into the attribute upsampling step in order to improve the results of the attribute upsampling. This part will also be extended to other attributes than color and it will be specifically optimized to cope with noisy point cloud data such as LiDAR. Combining these steps, a sequential upsampling procedure for point clouds is established. The final goal of this proposal is to transfer the knowledge from attribute and geometry upsampling into a joint cross-modal upsampling scheme for any attribute. Such an approach has not yet been reported and is expected to significantly improve the quality of upsampled point clouds.

• DFG Programme: Research Grants