Why monitoring and evaluation?

The purpose of evaluation is to learn from past experience and improve future performance, thereby supporting decision makers to steer actions and policies towards climate resilience. Learning requires information, which can be provided by monitoring indicators. Examples of specific objectives of monitoring include:

• analysing climate parameters and climate change impacts;
• tracking progress in implementing adaptation strategies;
• assessing the impact of adaptation actions;
• benchmarking the current status with other cities.

Assuming that monitoring the development of the climate threats will usually be done by national authorities we focus here on the monitoring and evaluation of the city’s adaptation actions. For doing this we can distinguish indicators at area-level and at project level. Indicators at area-level provide insight in the current status of the area under study with regards to certain objectives. The first assessment of these indicators provides a baseline. When regularly updated, the indicators can show progress (or lack thereof) towards the objectives. If needed, policies can be adjusted and improved based on these indicator results.

Project indicators can be used ex-ante and ex-post. Ex-ante assessments can estimate the contribution of projects to the goals that are defined for an area. The impact of various projects or measures can thus be compared with each other and prioritized. As such, the indicator outcomes can play a role in tender procedures. Finally, after the completion of a project, indicators can evaluate the actual contribution of a project to the objectives.

Why is it a key challenge?

It is always possible to find some simple indicators to measure the amount of effort spend on adaptation and the immediate outputs, such as number of trees planted. However, it is far more difficult to define impact indicators that show actual progress towards increased resilience. Often changes are not regularly monitored, and common metrics for success are lacking. Also the time and context specific nature of some adaptation options makes it difficult to attribute the impact to a specific adaptation measure. For example, planting (young) trees will not immediately lead to improved thermal comfort on a square; less complaints about water nuisance may be related to changes in the population of the neighbourhood instead of to improvements in drainage. For finding practical solutions to the monitoring and evaluation issue, it is often necessary to focus on a plausible contribution of the adaptation options to the achievement of outcomes. This FEC-page addresses some core aspects to take into account when developing a monitor and evaluation plan.

How to monitor and evaluate?

Important elements in establishing a good monitoring and evaluation strategy are a monitoring framework, suitable indicators, a thorough organisation and an approach for closing the learning cycle. These elements will be discussed in the following paragraphs.

Monitoring framework

A monitoring framework is built on the main goals and sub goals for a certain area. Clear and specific objectives are, therefore, crucial for a good evaluation and monitoring procedure and indicators should reflect these objectives as closely as possible. When the goals are not defined well or when the indicators don’t represent them well, the indicators might not measure the right things and steer policy in the wrong direction. One should also be aware not to define the means to achieve a goal as the goal itself. Therefore, it’s important to define the main objectives of the adaptation strategy: What do you really want to achieve (see also Goal definition)? These objectives can then be translated to indicators at area and project level. Examples of indicators can be found in Identify indicators.

Selection of indicators

When developing and defining indicators, two aspects are important to keep in mind: the type of indicator that suits the purpose of monitoring, and whether the indicator complies with certain criteria. Examples of adaptation indicators are gathered through a literature study and shown at the bottom of this page (classified according to the elements of risk assessment).

Types of indicators

Various types of indicators exist which assess different elements of a strategy or project: input, process, output, outcome and impact indicators (see the table below an explanation).

Criteria for indicators

Good indicators comply with a set of criteria. The criteria mentioned in the table below are based on the criteria used in the Civitas framework ¹⁾Rooijen, T., Nesterova, N. & Guikink, D., 2013. Applied framework for evaluation in CIVITAS PLUS II. Deliverable 4.10 of CIVITAS WIKI of CIVITAS initiative. Cleaner and better transport in cities (CIVITAS WIKI). Especially with regards to the criteria ‘familiarity’ and ‘data availability’ it is valuable to consult users and stakeholders, for example project leaders, city departments concerned with data collection and analysis, politicians, contractors and other parties that could provide information that is required to calculate the indicators. Involving a larger group in this ensures a check of whether the indicators are feasible and appealing. In addition, the consultation increases awareness with the goals of the area and creates support for the framework.

Table 2 Criteria for indicators. Adapted from: (Rooijen and Nesterova, 2013).

RELEVANCE	The indicator should have a strong link to the adaptation goals
FAMILIARITY	The indicators should be easy to understand by the users
DATA AVAILABILITY	Data for the indicators should be easily available and be gathered at reasonable costs
MEASURABILITY	The identified indicators should be capable of being measured, preferably as objectively as possible
RELIABILITY	The results of the indicators should have a limited degree of uncertainty and margin of error. Factors that increase reliability are; good quality of the underlying data, clear and specific definition of the indicator and a transparent and direct calculation methodology.
NON-REDUNDANCY	Indicators within a framework should not measure the same aspect
COMPLETENESS	The total set of indicators should consider all aspects that affect the adaptation goals

Quantitative /qualitative

Preferably, indicators are expressed in objective, quantifiable parameters. It’s not always possible, however, to collect hard data and measurements. In these instances, one could also resolve to alternatives and use semi-quantitative and qualitative parameters, such as the Likert scale. ²⁾A Likert scale is a five (or seven) point scale which is used to express the analyst or independent expert estimate on the indicator. For example, assuming that an indicator on the climate resiliency of local policies is highly desirable, but it is difficult to measure this exactly. In that case a semiquantitative indicator can be constructed assessing the extent to which the city has developed and implemented a climate resilient strategy between 1) not at all, and 5) a climate resilience strategy has been developed and approved and climate adaptation is integrated in every municipal department and plan.

Organisation

Regular assessment of the indicators at area and project level requires a firm basis of the monitoring and evaluation framework in the organization. This starts with the ratification of the framework, the indicators, and the assessment and reporting frequency by management. In addition, it requires broad communication within the organisation to familiarise everyone with the framework and approach. Furthermore, it is recommended to appoint a lead monitoring and evaluation manager who coordinates the process and assign responsible persons for (a set of) indicators (‘ambassadors’) who organize the data collection and assessment of indicators.

At project level, project managers can be made responsible for assessing project indicators after completion. Ex-ante assessments of projects can be made standard procedure to provide information to compare projects and alternatives and make them more effective.

Closing the learning cycle

Indicator results can provide valuable information for policy and decision makers. To make use of this information, it would be good to regularly gather the outcomes of all indicators in a comprehensive report and communicate this to politicians, citizens and other stakeholders and discuss the results with them. Especially if the results are lagging behind, it is important to further investigate the causes for this. Having defined possible solutions, policies and plans can be adjusted accordingly.  In addition, this is might be a good phase in the process to reconsider the previously defined adaptation goals and see whether roles and responsibilities should be reassigned.

Most encountered in decision framework steps:

• Goal definition
• Baseline
• Monitoring and Evaluation plan

Examples of urban adaptation indicators

Impact

Indicators that address the (direct) effects/consequences on natural and human systems (lives, livelihoods, health, ecosystems, economies, societies, cultures, services and infrastructure).

Sensitivity

Indicators that target the degree to which a system or species is affected, either adversely or beneficially, by climate variability or change. Often addressing intrinsic properties of an object resulting in susceptibility to a risk source.

Adaptive Capacity

Indicators that assess the ability of systems, institutions, humans and other organisms to adjust to potential damage, to take advantage of opportunities, or to respond to consequences.

Stressors

Indicators that signify a change or trend unrelated to climate that can exacerbate the impact of climate hazards.

Exposure

Indicators that the presence of people, livelihoods, species or ecosystems, environmental services and resources, infrastructure, or economic, social, or cultural assets in places that could be adversely affected.

Hazard

Indicators that assess the potential occurrence of a natural or human-induced physical event or trend or their physical impact (i.e. flooding, heat stress, drought).

Drivers

Indicators that assess meteorological parameters that drive climate change.