ESMValCore

ESMValTool Core: core functionalities and driver for ESMValTool, a community diagnostic and performance metrics tool for routine evaluation of Earth System Models in CMIP.

35
mentions
25
contributors
7692 commits | Last update: September 24, 2021

Cite this software

Choose a version:
DOI:
[[ releases.length > 0 ? releases[selectedIndex].doi : conceptDOI ]]
Copy to clipboard
Choose a reference manager file format:
Download file

What ESMValCore can do for you

  • Finding data in a directory structure typically used for CMIP data
  • ESMValTool preprocessor functions based on iris for e.g. regridding, vertical interpolation, statistics, correcting (meta)data errors, extracting a time range, etcetera.
  • High flexibility: new diagnostics and more observational data can be easily added.
  • Multi-language support: Python, NCL, R, Julia... other open-source languages are possible.
  • CF/CMOR compliant: data from many different projects can be handled (CMIP, obs4mips, ana4mips, CCMI, CCMVal, AEROCOM, etc.). Routines are provided to CMOR-ize non-compliant data.
  • Integration in modeling workflows: for EMAC, NOAA-GFDL and NEMO, can be easily extended.

The ESMValCore software package provides the core functionality for ESMValTool, the Earth System Model eValuation Tool. It provides a configurable framework for finding CMIP files using a “data reference syntax”, applying commonly used pre-processing functions to them, running analysis scripts, and recording provenance. Numerous pre-processing functions, e.g. for data selection, regridding, and statistics are readily available and the modular design makes it easy to add more. The ESMValCore package is easy to install with relatively few dependencies, written in Python 3, based on state-of-the-art open-source libraries such as Iris and Dask, and widely used standards such as YAML, NetCDF, CF-Conventions, and W3C PROV. An extensive set of automated tests and code quality checks ensure the reliability of the package.

The ESMValCore package uses human-readable recipes to define which variables and datasets to use, how to pre-process that data, and what scientific analysis scripts to run. The package provides convenient interfaces, based on the YAML and NetCDF/CF-convention file formats, for running diagnostic scripts written in any programming language. Because the ESMValCore framework takes care of running the workflow defined in the recipe in parallel, most analyses run much faster, with no additional programming effort required from the authors of the analysis scripts. For example, benchmarks show a factor of 30 speedup with respect to version 1 of the tool for a representative recipe on a 24 core machine. A large collection of standard recipes and associated analysis scripts is available in the ESMValTool package for reproducing selected peer-reviewed analyses.

Read more
Tags
  • Big data
  • Workflow technologies
  • Visualization
  • Optimized data handling
Programming Language
  • Python
License
  • Apache-2.0
Source code

Participating organizations

ETH Zurich

Mentions

21 Journal articles

Contributors

  • Fakhereh Alidoost
    Netherlands eScience Center
  • Bouwe Andela
    Netherlands eScience Center
  • Björn Brötz
    DLR
  • Jaro Camphuijsen
    Netherlands eScience Center
  • Lee de Mora
    Plymouth Marine Laboratory
  • Niels Drost
    Netherlands eScience Center
  • Faruk Diblen
    Netherlands eScience Center
  • Veronika Eyring
    DLR
  • Bettina Gier
    University of Bremen
  • Birgit Hassler
    DLR
  • Axel Lauer
    DLR
  • Peter Kalverla
    Netherlands eScience Center
  • Rémi Kazeroni
    DLR
  • Saskia Loosveldt-Tomas
    Barcelona Supercomputing Center
  • Nikolay Koldunov
    Alfred Wegener Institute
  • Ruth Lorenz
    ETH Zurich
  • Benjamin Müller
    Ludwig Maximilian University of Munich
  • Inti Pelupessy
    Netherlands eScience Center
  • Valeriu Predoi
    University of Reading
  • Mattia Righi
    DLR
  • Manuel Schlund
    DLR
  • Alistair Sellar
    Met Office
  • Stef Smeets
    Netherlands eScience Center
  • Javier Vegas-Regidor
    Barcelona Supercomputing Center
  • Klaus Zimmermann
    Swedish Meteorological and Hydrological Institute
Show all contributors
Contact person
Bouwe Andela
Netherlands eScience Center

Information for page maintainers

OAI-PMH metadata:
OK
citation metadata:
OK
Show