VAMPIR – Virtualized Non-Functional Memory Properties for Data-Pipeline Scheduling

• Applicants: Professor Dr.-Ing. Wolfgang Lehner; Professor Dr.-Ing. Horst Schirmeier
• Subject Area: Data Management, Data-Intensive Systems, Computer Science Methods in Business Informatics ; Security and Dependability, Operating-, Communication- and Distributed Systems
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 502444078

Recent advances in computer architecture and modern hardware can be considered an inflection point for HW/SW co-design. Although all areas of hardware components are affected by this development, especially memory and storage has seen an unprecedented change: Novel hardware has blurred the traditional mental picture of a memory/storage hierarchy. Very large caches, High-Bandwidth-Memory (HBM), Non-Uniform Memory Access (NUMA), or even remote-memory designs (like RDMA or CXL) as well as extremely fast SSDs add to the heterogeneous portfolio of available memory/storage techniques. Thus, the traditional hierarchy has to be replaced by the mental picture of a pool of memory/storage techniques positioned in a multidimensional design space. Unfortunately, this degree of freedom makes it difficult for applications to efficiently and effectively use these memories. Within our envisioned VAMPIR project, we scrutinize the research hypothesis that only an adequate HW/SW co-design (more specifically an innovative OS/App co-design) is able to fully embrace and exploit the individual properties of heterogeneous memories. We underpin our hypothesis by observations from two directions: On the one hand, we see a clear demand for an operating system to offer more than just access to memory/storage devices, but to provide a virtualized memory layer with features beyond virtual address spaces. For example, compression, encryption, increase of fault-tolerance, replication for a higher degree of persistency, etc. should be provided by the operating system and leveraged by individual applications. On the other hand, we envision future applications to “talk” to an underlying operating system and to “negotiate” individual properties required by the application. For this aspect, we focus on the class of data-intensive applications (like database query engines), which deploy individual data streams consisting of (a) a network of dependent data pipelines with non-blocking operators along a pipeline and (b) blocking operators as connection points between pipelines. Within the context of the SPP 2377, our VAMPIR project targets exactly the interplay of optimizing the execution of data-intensive applications based on data pipelines with respect to virtualized non-functional memory properties on disruptive memory technologies. The core VAMPIR idea is to combine (a) the capability of an operating system providing memory allocations with respect to non-functional requirements – in terms of capabilities as well as time-constraints – with (b) the opportunity of data-pipeline processing with a pre-defined schedule of pipeline executions and dependent memory requirements. We envision to investigate the combination in a bi-directional manner to allow negotiation of memory requirements and provisioning, thus pushing the envelope in OS/App co-design.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2377: Disruptive Main-Memory Technologies
• Co-Investigator: Professor Dr.-Ing. Dirk Habich