ADA-FS - Advanced Data Placement via Ad-hoc File Systems at Extreme Scales

• Applicants: Professor Dr.-Ing. André Brinkmann; Professor Dr. Wolfgang E. Nagel; Professor Dr. Achim Streit
• Subject Area: Security and Dependability, Operating-, Communication- and Distributed Systems ; Computer Architecture, Embedded and Massively Parallel Systems
• Term: from 2015 to 2020
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 279334509

The proposed ADA-FS project aims at improving the I/O performance for extreme-scale parallel applications. It will introduce three novel components to the system software. First, it proposes the use of an ad-hoc overlay file system concept. Second, a central I/O planner component is envisioned. The third component integrates a concept for application monitoring, with emphasis on its resource usage and I/O behavior. The ad-hoc overlay file system will be deployed as a separate instance for every parallel application on the same compute nodes that the application is running on. This file system will use traditional and novel system components such as RAM, local SSDs, or NVRAM as data caches for the I/O subsystem provided that these are not in use by the particular application. This approach will increase the effective I/O bandwidth and reduce the latency of I/O operations. Parallel applications will require no modifications to make use of this ad-hoc file system. The central I/O planner will coordinate all data transfers to and from all instances of the overlay file system. The planner will prevent a random mix of I/O requests in which no application can anticipate how much of the total I/O bandwidth is available at any given point in time in the globally shared I/O infrastructure. Instead, the central I/O planning will allow appropriate predictions about transfer times to and from the ad-hoc overlay file systems. As long as an application reads and writes only within its private ad-hoc overlay file system, it will use dedicated I/O resources in an exclusive manner. In addition, information about an application I/O behavior will be extracted. Such information is useful for planning the staging-in and staging-out of data as well as to optimize the data distribution within the ad-hoc overlay file system. This information will be provided by integrated monitoring components at the application level. The performance advantage by the proposed solution increases with the parallelism, provided that the combined bandwidths of local storage caches will always surpass the total bandwidth of central storage systems. The solutions proposed by ADA-FS will incorporate novel node-local storage locations forecasted for upcoming HPC installations.

• DFG Programme: Priority Programmes
• Subproject of: SPP 1648: Software for Exascale Computing