NoSQL Schema Evolution and Big Data Migration at Scale II

• Applicants: Professorin Dr.-Ing. Meike Klettke; Professorin Dr. Stefanie Scherzinger; Professorin Dr. Uta Störl
• Subject Area: Security and Dependability, Operating-, Communication- and Distributed Systems
• Term: from 2017 to 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 385808805

This proposal is a follow-up on DFG research grant number (sign removed), where we have been targeting database schema evolution in applications backed by NoSQL data stores. While many NoSQL data stores do not enforce a fixed schema, the application code nevertheless makes assumptions about the structure of the data stored. Thus, the application code imposes a schema. Along with the application code, this schema evolves. We have been successfully collaborating on DFG grant (sign removed) since December 2017, addressing the dire need for well-principled tools managing NoSQL database schemas and their evolution. While our work is still ongoing, we have already made a range of genuine contributions leading to well-received publications, and in particular, working prototype systems. So far, our research has had a strong focus on the evolution of the logical NoSQL schema in single-model data stores. With this follow-up proposal, we expand to the more general family of polystores. While polystores are often recommended for continuously evolving and data intensive applications, research on schema evolution in polystores is still in its infancy:- As of today, there are no systematic approaches for schema evolution management in polystores, involving several interlinked store-specific logical schemas.- As of today, there are no automatable best practices for logical schema design in NoSQL data stores. Moreover, the schema design must be adaptive w.r.t. schema evolution.- As of today, there are no well-principled strategies for physical schema optimisation in NoSQL data stores that are adaptive w.r.t. schema evolution. In this follow-up proposal, we address these pressing issues holistically, on the level of the global polystore schema, the logical schemas of the individual data stores backing the polystore, as well as their physical schemas. We survey the research state-of-the-art, point out the current pain points and limitations, and present a sequence of novel and interlocking work packages in which we research solutions. One sustainable contribution to the research community is the development of the first schema evolution benchmark dedicated to polystores.

• DFG Programme: Research Grants