Testing indicator performance

By Saskia | October 16, 2018

Indicators are useful and versatile tools applied in disciplines such as engineering, chemistry, medicine, economy or sociology. In ecosystem-based management a key role of an indicator is to inform on the current status of the system component as well as the effectiveness of specific management measures to move the component into a different state. In European Union (EU) marine policy, indicator development has recently progressed as part of the implementation of the Marine Strategy Framework Directive (MSFD)1 to aid the achievement of Good Environmental Status (GES) of the EU’s marine waters by 2020. GES is defined with respect to 11 qualitative descriptors (D1-D11), which describe the environment when GES has been achieved, such as the maintenance of biodiversity (D1) or the stability of the food web (D4).

To achieve GES requires ideally operational indicators that not only describe the state of the descriptor but also have defined target and threshold values for triggering management measures as well as clearly defined and well-understood relationships to pressures that can be managed such as fishing mortality. The latter is particularly important for quantifying and predicting the success of individual measures. But at the same time, establishing such relationships is challenging for highly stochastic and complex adaptive systems such as food webs. Consequently, greater focus has been given hitherto to criteria such as data availability, scientific basis, or measurability than pressure sensitivity or indicator robustness.


One approach for a complete indicator testing could involve the following 3 steps:


Figure 1: Conceptual framework of indicator testing (modified from Samhouri et al. 2012). 2


Step A: Direct effect of single pressure

In the first step, functional relationships of individual indicators to single pressures need to be established. If an indicator is mainly directly affected by a single pressure, a target (represented by the point on the line) could be set depending on the functional relationship.

To apply a more flexible modeling approach that allows also non-linear relationships, suitable candidates would be

  • Non-Least Squares (NLS) models (using e.g. the base nls() function or the R package nlmrt), which require you to specify the function beforehand
  • Smoothing Splines (using the R function splines2)
  • Generalized Additive Models (GAM) (using e.g. the mgcv R package), which require no prior function setting
  • Tree-based methods (e.g. Bagging, Random Forest, Boosting using the packages randomForest, gbm, or caret)

Step B: Direct effects of multiple pressures (≥2)

The analysis of thresholds is not only complicated by non-linear dynamics but also by multiple pressures that can interact. Because of these interactions a given GES target reference value might be achieved by two or more sets of different pressure conditions (e.g. indicated by the two points in each 3-dimensional plot).

In this second step, indicator responses to multiple pressures including potential interactions need to be tested and GES targets in this two-dimensional context evaluated using the same modeling approaches as in step A. To test within the GAM framework for interactions that show threshold dynamics one could use an extension, the so-called threshold-GAM (TGAM) 3 4, which is partly implemented in INDperform.

Step C: Direct and indirect effects within the food web context

Most often food web dynamics are the result of direct and indirect effects through species interactions that act simultaneously, which can lead to congruence or conflicts between indicator targets. A thorough robustness testing requires, hence, also a food web model approach.

There are numerous numerical modeling approaches such as Ecopath with Ecosim, OSMOSE, ATLANTIS, etc. that could be used - as long as indicators can be derived from the model output.

Statistical modeling frameworks that allow the model setup based on the indicators itself could be

  • Multivariate AutoRegressive State-Space Models (MARSS) (using the marss package)
  • Structural Equation Models (SEM) (using e.g. the sem package)
  • Coupling of individual GAMs/TGAMs

Baltic Sea Case study

Colleagues and me adopted these 3 testing steps for food web indicators in the Baltic Sea within the Swedish project (“Ecosystem-based approach for developing and testing pelagic food web indicators”) I coordinated.

Step 1 and partly 2 (based on GAMS/TAMs) lead to the paper Otto, S.A., Kadin, M., Casini, M., Torres, M.A. & Blenckner, T. (2018). A quantitative framework for selecting and validating food web indicators. Ecological Indicators, 84, 619-631 and the R package INDperform I developed with PhD- and Master students.

Step 3 was adopted in the paper Torres, M.A., Casini, M., Huss , M., Otto, S.A., Kadin, M. & Gårdmark, A. (2017). Food-web indicators accounting for species interactions respond to multiple pressures. Ecological Indicators, 77, 67–79. using the MARSS approach.

Another paper based on coupled GAMs/TGAMs to model trade-offs and evaluate management strategies under climate change lead by Martina Kadin is in preparation.



The final project report is available under: https://www.naturvardsverket.se/Om-Naturvardsverket/Publikationer/ISBN/6700/978-91-620-6788-5/ (albeit in Swedish; there will be hopefully soon an English version)


  1. EU Directive 2008/56/EC; download here

  2. Samhouri, J.F., Lester, S.E., Selig, E.R., Halpern, B.S., Fogarty, M.J., Longo, C. et al. (2012). Sea sick? Setting targets to assess ocean health and ecosystem services. Ecosphere, 3, 1-18

  3. Ciannelli, L., Chan, K.-S., Bailey, K.M. & Stenseth, N.C. (2004). Nonadditive effects of the environment on the survival of a large marine fish population. Ecology, 85, 3418-3427

  4. Otto, S.A., Kornilovs, G., Llope, M. & Möllmann, C. (2014). Interactions among density, climate, and food web effects determine long-term life cycle dynamics of a key copepod. Marine Ecology Progress Series, 498, 73-84, doi: 10.3354/meps10613

comments powered by Disqus