Abstract | Simulating the Earth system is an important and complex problem, thus Earth system models are similarly complex, comprised of millions of lines of code. To appropriately utilize the latest computational and software infrastructure advancements in Earth system models running on modern hybrid computing architectures for the improvement of their performance, precision, accuracy, or all three; it is important to ensure that model simulations are repeatable and robust. This introduces the need for establishing statistical or non bit-for-bit reproducibility, since bit-for-bit reproducibility may not always be achievable. Here, we demonstrate a short-simulation ensemble-based test for the E3SM atmosphere model (EAM) to evaluate the null hypothesis that modified model results are statistically equivalent to that of the unmodified model. This test marks an improvement to the multi-testing Kolmogorov-Smirnov (MVK) test previously used in E3SM, which had a too high a false positive rate. This updated test evaluates a standard set of output variables across the two simulation ensembles and uses a false discovery rate correction to account for multiple testing. The false positive rates of the test are examined using re-sampling techniques on large simulation ensembles and are found to be lower than the previously implemented bootstrapping-based testing approach in E3SM. We also evaluate the statistical power of the test using perturbed simulation ensemble suites, each with a progressively larger magnitude of change to a tuning parameter. The new test is generally found to exhibit more statistical power than the current approach, being able to detect smaller changes in parameter values with higher confidence. |
|---|