Regridding E3SM Data with ncremap

This page is devoted to instruction in NCO’s regridding operator, ncremap. It describes steps necessary to create grids, and to regrid datasets between different grids with ncremap. Some of the simpler regridding options supported by ncclimo are also described at Generate, Regrid, and Split Climatologies (climo files) with ncclimo. This page describes those features in more detail, and other, more boutique features often useful for custom regridding solutions.

The Zen of Regridding

Most modern climate/weather-related research requires a regridding step in its workflow. The plethora of geometric and spectral grids on which model and observational data are stored ensures that regridding is usually necessary to scientific insight, especially the focused and variable resolution studies that E3SM models conduct. Why does such a common procedure seem so complex? Because a mind-boggling number of options are required to support advanced regridding features that many users never need. To defer that complexity, this HOWTO begins with solutions to the prototypical regridding problem, without mentioning any other options. It demonstrates how to solve this problem simply, including the minimal software installation required. Once the basic regridding vocabulary has been introduced, we solve the prototype problem when one or more inputs are "missing", or need to be created. The HOWTO ends with descriptions of different regridding modes and workflows that use features customized to particular models, observational datasets, and formats. The overall organization, including TBD sections (suggest others, or vote for prioritizing, below), is:

At a minimum, install a recent version NCO on your executable $PATH with the corresponding library on your $LD_LIBRARY_PATH. NCO installation instructions are here. Unless you have reason to do otherwise, we recommend installing NCO through the Conda package (conda install -c conda-forge nco) or via activating the E3SM-Unified environment. The Conda NCO package automagically installs two other important regridding tools, the ESMF_RegridWeightGen (aka ERWG) executable and the TempestRemap (aka TR) executables. Execute ncremap --config to verify you have a working installation:

zender@aerosol:~$ ncremap --config
ncremap, the NCO regridder and grid, map, and weight-generator, version 4.9.3-alpha02 "Fuji Rolls"
...[Legal Stuff]...
Config: ncremap script located in directory /Users/zender/bin
Config: NCO binaries located in directory /Users/zender/bin, linked to netCDF library version 4.7.3
Config: No hardcoded machine-dependent path/module overrides. (If desired, turn-on NCO hardcoded paths at supported national labs with "export NCO_PATH_OVERRIDE=Yes").
Config: External (non-NCO) program availability:
Config: ESMF weight-generation command ESMF_RegridWeightGen version 7.1.0r found as /opt/local/bin/ESMF_RegridWeightGen
Config: MOAB-Tempest weight-generation command mbtempest not found
Config: MPAS depth coordinate addition command add_depth.py found as /Users/zender/bin/add_depth.py
Config: TempestRemap weight-generation command GenerateOfflineMap found as /usr/local/bin/GenerateOfflineMap

Only NCO is required for many operations including applying existing regridding weights (aka, regridding) and/or generating grids, maps, or conservative weights with the NCO algorithms. Generating new weights (and map-files) with ERWG or TR requires that you install those packages (both of which come with the NCO Conda package). MOAB-Tempest (MBTR) is only required to generate TR weights on the largest meshes. It is also available as a Conda package, and comes with the E3SM-Unified environment. Make sure ncremap reports a sufficiently working status as above before proceeding further.

The horizontal regridding problem most commonly faced is converting output from a standard resolution model simulation to equivalent data on a different horizontal grid for visualization or intercomparison with other data. The EAM v1 model low-resolution simulations are performed and output on the ne30np4 SE (spectral element) grid, aka the "source grid". The corresponding EAM v2 simulations were conducted on the ne30pg2 FV (finite volume) grid. E3SM source grids like these are called “unstructured” because they have only one horizontal dimension (i.e., 1D) which makes them difficult to visualize. The recommended 2D (latitude-longitude) grids for analysis (aka the "destination grid") of v1 simulations was the 129x256 Cap grid (the gridcells at the poles look like yarmulke caps), and since v2 is the 180x360 equi-angular grid used by CMIP. The single most important capability of a regridder is the intelligent application of weights that transform data on the input grid to the desired output grid. These weights are stored in a "map-file", a single file which contains all the necessary weights and grid information necessary. While most regridding problems revolve around creating the appropriate map-file, this prototype problem assumes that has already been done, so the appropriate map-file (map.nc) already exists and ncremap can immediately transform the input dataset (dat_src.nc) to the output (regridded) dataset (dat_rgr.nc):

This solution is deceptively simple because it conceals the choices, paths, and options required to create the appropriate map.nc for all situations. We will discuss creating map.nc later after showing more powerful and parallel ways to solve the prototype problem. The solution above works for users savvy enough to know how to find appropriate pre-built map-files. Map-files used by the E3SM model are available at https://web.lcrc.anl.gov/public/e3sm/inputdata/cpl/gridmaps/ . Additional map-files useful in post-processing are available at https://web.lcrc.anl.gov/public/e3sm/diagnostics/maps/. Many commonly used maps and grids can also be found in my (@czender's) directories as ~zender/data/[maps,grids] at most DOE High Performance Computing (HPC) centers. Take a minute now to look at these locations.

Pre-built map-files use the (nearly) standardized naming convention map_srcgrd_to_dstgrd_algtyp.YYYYMMDD.nc, where srcgrd and dstgrd are the source and destination grid names, algtyp is a shorthand for the numerical regridding algorithm, and YYYYMMDD is the date assigned to the map (i.e., the date the map was created). The source grid in the example above is called ne30np4, the destination is called fv129x256. A pre-built map for the v1 combination is map.nc = map_ne30np4_to_fv129x256_aave.20150901.nc, and for v2 is map_ne30pg2_to_cmip6_180x360_aave.20200201.nc. What is aave? Weight generators can use about a dozen interpolation algorithms for regridding, and each has a shorthand name. For now, it is enough to know that the two most common algorithms are (first-order) conservative area-average regridding (aave) and bilinear interpolation (bilin or blin). Hence this conservatively regrids dat_src.nc to dat_rgr.nc with first order accuracy:

Before looking into map-file generation in the next section, try a few ncremap features. For speed's sake, regrid only selected variables:

To regrid multiple files with a single command, supply ncremap with the source and regridded directory names (drc_src, drc_rgr). It regrids every file in drc_src and places the output in drc_rgr:

Or supply specific input filenames on the command-line, piped through standard input, or redirected from a file:

When an output directory is not specified, ncremap writes to the current working directory. When the output and input directories are the same, ncremap appends a string (based on the destination grid resolution) to each input filename to avoid name collisions. Finally, be aware that multiple-file invocations of ncremap execute in parallel by default. Power users will want to tune this as described in the section on "Intermediate Regridding".

The simplest regridding procedure applies an existing map-file to your data, as in the above example (public servers of pre-existing map-files are also linked to above). At most DOE High Performance Computing (HPC) centers these and others can also be found in my (Charlie Zender 's) directory, ~zender/data/maps. If the desired map-file cannot be found, then you must create it. Creating a map-file requires a complete specification of both source and destination grids (meshes). The files that contain these grid specifications are called "grid-files". Many E3SM grid-files are publicly available within model-specific directories of the previous location, e.g., https://web.lcrc.anl.gov/public/e3sm/inputdata/ocn/mpas-o/oEC60to30v3/ . Many grids useful for post-processing are publicly served from https://web.lcrc.anl.gov/public/e3sm/diagnostics/grids/. At most DOE High Performance Computing (HPC) centers these can also be found in my (@czender's) directory, ~zender/data/grids. Take a minute now to look there for the prototype problem grid-files, e.g., for FV 129x256, cmip6_180x360, and ne30pg2 grid-files.

You might find multiple grid-files that contain the string 129x256. Grid-file names are often ambiguous. The grid-file global metadata (ncks -M grid.nc) often displays the source of the grid. These metadata, and sometimes the actual data (fxm: link), are usually more complete and/or accurate in files with a YYYYMMDD-format date-stamp. For example, the metadata in file 129x256_SCRIP.20150901.nc clearly state it is an FV grid and not some other type of grid with 129x256 resolution. The metadata in 129x256_SCRIP.130510 tell the user nothing about the grid boundaries, and some of the data are flawed. When grids seem identical except for their date-stamp, use the grid with the later date-stamp. The curious can examine a grid-file (ncks -M -m grid.nc) and easily see it looks completely different from a typical model or observational data file. Grid-files and data-files are not interchangeable.

Multiple grid-files also contain the string ne30. These are either slightly different grids, or the same grids store in different formats meant for different post-processing tools. The different spectral element (SE) and Finite Volume (FV) grid types are described with figures and described here (https://acme-climate.atlassian.net/wiki/spaces/Docs/pages/34113147/Atmosphere+Grids). As explained there, for E3SMv1 data many people will want the "dual-grid" with pentagons. The correct grid-file for this is ne30np4_pentagons.091226.nc. Do not be tempted by SE grid-files named with latlon. Datasets from E3SM v2 and v3 simulations are all on FV grids. EAM v2 and v3 grids are named in the format neXXXpg2. ELM and MPAS names take a wider variety of forms, many of which appear below.

All grid-files discussed so far are in SCRIP-format, named for the Spherical Coordinate Remapping and Interpolation Package (authored by @pjones). Other formats exist and are increasingly important, especially for SE grids. For now just know that these other formats are also usually stored as netCDF, and that some tools allow non-SCRIP formats to be used interchangeably with SCRIP.

Once armed with source and destination grid-files, one can generate their map-file with

Regrid a datafile at the same time as generating the map-file for archival:

Regrid a datafile without archiving the (internally generated) map-file:

For the prototype problem, the map-file generation procedure becomes

The map-files above are named with alg_typ=nco because the ncremap default interpolation algorithm is the first-order conservative NCO algorithm (NB: before NCO 4.9.1 the default algorithm was ESMF bilin). To re-create the aave map in the first example, invoke ncremap with -a esmfaave (the newest v3 naming convention) or -a conserve (same algorithm, different name in v1, v2):

This takes a few minutes, so save custom-generated map-files for future use. Computing weights to generate map-files is much more computationally expensive and time-consuming than regridding, i.e., than applying the weights in the map-file to the data. We will gloss over most options that weight-generators can take into consideration, because their default values often work well. One option worth knowing now is -a. The invocation synonyms for -a are --alg_typ, --algorithm, and --regrid_algorithm. These are listed in the square brackets in the self-help message that ncremap prints when it is invoked without argument, or with --help:

-a alg_typ Algorithm for weight generation (default ncoaave) [alg_typ, algorithm, regrid_algorithm]
ESMF algorithms: esmfbilin,bilinear|esmfaave,aave,conserve|conserve2nd|nearestdtos|neareststod|patch
NCO algorithms: ncoaave,nco,nco_con|ncoidw,nco_dwe (inverse-distance-weighted interpolation/extrapolation)
Tempest (and MOAB-Tempest) algorithms: traave,fv2fv_flx|trbilin|trfv2|trintbilin|tempest|fv2fv|fv2fv_stt

At least one valid option argument for each supported interpolation type is shown separated by vertical bars. The arguments shown have multiple synonyms, separated by commas, that are equivalent. For example, -a esmfaave is equivalent to -a aave and --alg_typ=conserve. Use the longer option form for clarity and precision, and the shorter form for conciseness. The full list of synonyms, and the complete documentation, is at http://nco.sf.net/nco.html#alg_typ. The NCO algorithm ncoaave is the default (because it is always available). Commonly-used algorithms that invoke ERWG are esmfbilin and esmfaave. TR options are discussed below. As of E3SM v3, TR algorithms are preferred for all mappings because they are the most accurate. Peruse the list of options now, though defer a thorough investigation until you reach the "Intermediate Regridding" section.

Thus far we have explained how to apply a map-file to data, and how, if necessary, to generate a map-file from known grids. What if there is no map-file and the source or the destination grid-files (or both) are unavailable? Often, one knows the basic information about a grid (e.g., resolution) but lacks the grid-file that contains the complete information for that grid geometry. In such cases, one must create the grid-file via one of two methods. First, one can let ncremap attempt to infer the grid-file from a data file known to be on the desired grid. This procedure is called "inferral" and is fairly painless. Second, one can feed ncremap all the required parameters (for rectangular grids only) and it will generate a grid-file. This requires a precise specification of the grid geometry, and will be covered the sub-section on "Manual Grid-file Generation".

Before we describe what the inferral procedure does, here is an example that demonstrates how easy it is. You can regrid an SE (or FV) dataset from our prototype example to the same grid as an evaluation dataset. Pick any 2D (e.g., MxN latitude-by-longitude) dataset to compare the E3SM simulations to. Inferral uses the grid information in the evaluation dataset, which is already on the desired destination grid, to create the (internally generated) destination grid-file. Supply ncremap with any dataset on the desired destination grid (dat_dst.nc) with -d (for "dataset") instead of -g (for "grid"):

This tells ncremap to infer the destination grid-file and to use it to generate the desired map-file, named with the supposed destination resolution (here, MxN degrees or gridcells). To archive the inferred destination grid for later use, supply a name for the grid with -g:

One can infer a grid without having to regrid anything. Supply ncremap with a data-template file (dat.nc) and a grid-file name (grd.nc). Since there are no input files to regrid, ncremap exits after inferring the grid-file:

Grid-inferral is easier to try than manual grid-generation, and will work if the data file contains the necessary information. The only data needed to construct a SCRIP grid-file are the vertices of each gridcell. The gridcell vertices define the gridcell edges and these in turn define the gridcell area which is equivalent to the all-important weight parameter necessary to regrid data. Of course the gridcell vertices must be stored with recognizable names and/or metadata indicators. The Climate & Forecast (CF) metadata convention calls the gridcell vertices the "cell-bounds". Coordinates (like latitude and longitude) usually store cell-center values, and should, according to CF, have bounds attributes whose values point to variables (e.g., lat_bounds or lon_vertices) that contain the actual vertices. Relatively few datasets "in the wild" contain gridcell vertices, though the practice is, happily, on the rise. Formally SE models have nodal points with weights without any fixed perimeter or vertices assigned to the nodes, so the lack of vertices in SE model output is a feature, not an oversight. The dual-grid (referenced above) addresses this by defining "pretend" gridcell vertices for each nodal point so that an SE dataset can be treated like an FV dataset. However, dual-grids are difficult to generate, and may not exist or be accurate for many SE grids. In that case, ncremap cannot infer the grid (because the vertices are unknown) and one needs to use a different package (such as TempestRemap, below) to construct the grid-file and the mapping weights.

Inferral works well on important categories of grids for which ncremap can guess the missing grid information. In the absence of gridcell vertice information, ncremap examines the location of and spacing between gridcell centers and can often determine from these what type of grid a data-file (not a grid-file!) is stored on. A data-file simply means the filetype preferred for creation/distribution of modeled/observed data. Hence ncremap has the (original and unique, so far as we know) ability to infer all useful rectangular grid-types from data-files that employ the grid. The key constraint here is "rectangular", meaning the grid must be orthogonal (though not necessarily regularly spaced) in latitude and longitude. This includes all uniform angle grids, FV grids, and Gaussian grids. For curvilinear grids (including most swath data), ncremap infers the underlying grid to be the set of curves that bisect the curves created by joining the gridcell centers. This often works well for curvilinear grids that do not cross a pole. Inferral works for unstructured (i.e., 1D) grids only when the cell-bounds are stored in the datafile as described above. Hence inferral will not work on raw output from SE models.

A few more examples will flesh-out how inferral can be used. First, ncremap can infer both source and destination grids in one command:

Here the user provides only data files (no grid- or map-files) yet still obtains regridded output! The first positional (i.e., not immediately preceded by an option indicator) argument (dat_src.nc) is interpreted as the data to regrid, and the second positional argument (dat_rgr.nc) is the output name. The -d argument is the name of any dataset (dat_dst.nc) on the desired destination grid. ncremap infers the source grid from dat_src.nc, then infers the destination grid from dat_dst.nc, then generates weights (with the default algorithm since none is specified) and creates a map-file that it uses to regrid dat_src.nc to dat_rgr.nc. No grid-file or map-file names were specified (with -g or -m) so both grid-files and the map-file are generated internally in temporary locations and erased after use.