19th Workshop on high performance computing in meteorology

We will introduce the Maestro middleware framework, a data- and memory-aware abstraction for workflow coupling, inter- and intra-application data exchange and redistribution. A central design goal was to enable modelling of memory and storage hierarchies to allow for reasoning about data movement and placement based on costs of moving data objects. At the same time, data objects can carry user-defined metadata as it is typical in meteorological production workflows, allowing workflow management software to reason about the data without inspecting it, at scale.
We will also present an outlook on future requirements for such frameworks in the context of federated data handling and cross-site workflow coupling.