#NEXTGenIO2019
Increased complexity and resolution of modeling systems and increased demand for environmental input to downstream products combine to pose significant challenges for increased volume and velocity of I/O as part of a research or operational workflows, particularly when seeking to exploit distributed computing architectures. While many of the challenges faced are common in the earth system modeling community, I plan to highlight tradeoffs between data locality and task granularity in workflows (and the implications of on-demand storage), particularly when faced with workflows of tasks with highly dissimilar computational resource requirements. This will include a discussion of asynchronous I/O in forecast models, and the possibility of data-driven downstream workflows that are able to fully exploit the use of high-speed persistent storage as a method to increase task parallelism.
Keywords | challenges |
---|