The purpose of evaluation is to learn from past experience and improve future performance, thereby supporting decision makers to steer actions and policies towards climateresilience. Learning requires information, which can be provided by monitoring indicators. Examples of specific objectives of monitoring include:
benchmarking the current status with other cities.
Assuming that monitoring the development of the climate threats will usually be done by national authorities we focus here on the monitoring and evaluation of the city’s adaptation actions. For doing this we can distinguish indicators at area-level and at project level. Indicators at area-level provide insight in the current status of the area under study with regards to certain objectives. The first assessment of these indicators provides a baseline. When regularly updated, the indicators can show progress (or lack thereof) towards the objectives. If needed, policies can be adjusted and improved based on these indicator results.
Project indicators can be used ex-ante and ex-post. Ex-ante assessments can estimate the contribution of projects to the goals that are defined for an area. The impact of various projects or measures can thus be compared with each other and prioritized. As such, the indicator outcomes can play a role in tender procedures. Finally, after the completion of a project, indicators can evaluate the actual contribution of a project to the objectives.
Why is it a key challenge?
It is always possible to find some simple indicators to measure the amount of effort spend on adaptation and the immediate outputs, such as number of trees planted. However, it is far more difficult to define impact indicators that show actual progress towards increased resilience. Often changes are not regularly monitored, and common metrics for success are lacking. Also the time and context specific nature of some adaptation options makes it difficult to attribute the impact to a specific adaptation measure. For example, planting (young) trees will not immediately lead to improved thermal comfort on a square; less complaints about water nuisance may be related to changes in the population of the neighbourhood instead of to improvements in drainage. For finding practical solutions to the monitoring and evaluation issue, it is often necessary to focus on a plausible contribution of the adaptation options to the achievement of outcomes. This FEC-page addresses some core aspects to take into account when developing a monitor and evaluation plan.
How to monitor and evaluate?
Important elements in establishing a good monitoring and evaluation strategy are a monitoring framework, suitable indicators, a thorough organisation and an approach for closing the learning cycle. These elements will be discussed in the following paragraphs.
Monitoring framework
A monitoring framework is built on the main goals and sub goals for a certain area. Clear and specific objectives are, therefore, crucial for a good evaluation and monitoring procedure and indicators should reflect these objectives as closely as possible. When the goals are not defined well or when the indicators don’t represent them well, the indicators might not measure the right things and steer policy in the wrong direction. One should also be aware not to define the means to achieve a goal as the goal itself. Therefore, it’s important to define the main objectives of the adaptation strategy: What do you really want to achieve (see also Goal definition)? These objectives can then be translated to indicators at area and project level. Examples of indicators can be found in Identify indicators.
Selection of indicators
When developing and defining indicators, two aspects are important to keep in mind: the type of indicator that suits the purpose of monitoring, and whether the indicator complies with certain criteria. Examples of adaptation indicators are gathered through a literature study and shown at the bottom of this page (classified according to the elements of risk assessment).
Types of indicators
Various types of indicators exist which assess different elements of a strategy or project: input, process, output, outcome and impact indicators (see the table below an explanation).
When monitoring progress towards climateresilience or evaluating projects, one is interested in the actual impact that has been achieved in relation to the goals. Impact indicators are, therefore, the main types of indicators for monitoring and evaluating climateadaptation.
As impact indicators are defined on a high level, they are applicable to all kinds of projects in all contexts, whether they assess flooding or heat related projects. For the same reason, impact indicators leave room for the cities and projects to find their own solutions to achieve a certain performance. By focusing the indicators on impacts instead of sectors, also cross-sectoral solutions can be easily evaluated. The indicator framework will not implicitly put a focus on isolated, sector specific solutions.
A disadvantage of impact indicators, however, is that impacts can be hard to perceive and it might take years before the results are apparent. Estimating the impact of a project in advance can be a difficult exercise taking into account various uncertainties. In addition, numerous contextual factors can influence the final impact reached. As an alternative, outcome indicators can be used to approach the impact reached as closely as possible. Nevertheless the impact is the only measure that counts for reaching policy goals.
Output indicators are often used for regular reporting purposes. When selecting indicators one should be aware that this type of indicator prescribes the way targets should be reached. They limit the measures that can be implemented, leave little room for innovative solutions to achieve the same goal, and risk being outdated within a few years.
Input or process indicators are easier to define and to measure than the more complex impact indicators. It is simple a question of counting persons, money, activities, connection, downloads, etc.. They give an impression of the scale of the effort needed for a given impact and can be used as background information to investigate whether something is lacking behind and can be used to demonstrate the efficiency of a project.
Table 1 Types of indicators
Input indicators
These indicators refer to the resources needed for the implementation of an activity or intervention, measuring the quantity, quality, and timeliness of resources. Policies, human resources, materials, financial resources are examples of input indicators.
Process /activity indicators
Process indicators refer to indicators to measure whether planned activities took place. Examples include the number of meetings and training courses.
Output indicators
Output indicators measure the product (“output”) of the activity, e.g. the number of homes made dry-proof.
Outcome indicators
Outcome indicators measure the intermediate results generated by project outputs. Outcome indicators refer more specifically to the objectives of an intervention, that is its ‘results’. They relate to both the “quantity” (“how many”) and quality (“how well”) of the activities implemented. Often they are ‘coverage indicators’ measuring the extent to which the target population has been reached by the project, e.g. share of dry-proof homes in the total population of homes needing water protection.
Measuring the quality and quantity of long-term results generated by programme outputs in relation the main goals, e.g. damage due to flooding, number of traffic disturbances caused by extreme weather.
Criteria for indicators
Good indicators comply with a set of criteria. The criteria mentioned in the table below are based on the criteria used in the Civitas framework 1)Rooijen, T., Nesterova, N. & Guikink, D., 2013. Applied framework for evaluation in CIVITAS PLUS II. Deliverable 4.10 of CIVITAS WIKI of CIVITAS initiative. Cleaner and better transport in cities (CIVITAS WIKI). Especially with regards to the criteria ‘familiarity’ and ‘data availability’ it is valuable to consult users and stakeholders, for example project leaders, city departments concerned with data collection and analysis, politicians, contractors and other parties that could provide information that is required to calculate the indicators. Involving a larger group in this ensures a check of whether the indicators are feasible and appealing. In addition, the consultation increases awareness with the goals of the area and creates support for the framework.
Table 2 Criteria for indicators. Adapted from: (Rooijen and Nesterova, 2013).
The results of the indicators should have a limited degree of uncertainty and margin of error. Factors that increase reliability are; good quality of the underlying data, clear and specific definition of the indicator and a transparent and direct calculation methodology.
NON-REDUNDANCY
Indicators within a framework should not measure the same aspect
COMPLETENESS
The total set of indicators should consider all aspects that affect the adaptation goals
Quantitative /qualitative
Preferably, indicators are expressed in objective, quantifiable parameters. It’s not always possible, however, to collect hard data and measurements. In these instances, one could also resolve to alternatives and use semi-quantitative and qualitative parameters, such as the Likert scale. 2)A Likert scale is a five (or seven) point scale which is used to express the analyst or independent expert estimate on the indicator. For example, assuming that an indicator on the climate resiliency of local policies is highly desirable, but it is difficult to measure this exactly. In that case a semiquantitative indicator can be constructed assessing the extent to which the city has developed and implemented a climate resilient strategy between 1) not at all, and 5) a climateresilience strategy has been developed and approved and climateadaptation is integrated in every municipal department and plan.
Organisation
Regular assessment of the indicators at area and project level requires a firm basis of the monitoring and evaluation framework in the organization. This starts with the ratification of the framework, the indicators, and the assessment and reporting frequency by management. In addition, it requires broad communication within the organisation to familiarise everyone with the framework and approach. Furthermore, it is recommended to appoint a lead monitoring and evaluation manager who coordinates the process and assign responsible persons for (a set of) indicators (‘ambassadors’) who organize the data collection and assessment of indicators.
At project level, project managers can be made responsible for assessing project indicators after completion. Ex-ante assessments of projects can be made standard procedure to provide information to compare projects and alternatives and make them more effective.
Closing the learning cycle
Indicator results can provide valuable information for policy and decision makers. To make use of this information, it would be good to regularly gather the outcomes of all indicators in a comprehensive report and communicate this to politicians, citizens and other stakeholders and discuss the results with them. Especially if the results are lagging behind, it is important to further investigate the causes for this. Having defined possible solutions, policies and plans can be adjusted accordingly. In addition, this is might be a good phase in the process to reconsider the previously defined adaptation goals and see whether roles and responsibilities should be reassigned.
Indicators that address the (direct) effects/consequences on natural and human systems (lives, livelihoods, health, ecosystems, economies, societies, cultures, services and infrastructure).
CCIS5)Stadelmann M., Michaelowa A., Butzengeiger-Geyer S., Kohler M. (2011), Universal metrics to compare the effectiveness of climate changeadaptation projects. Centre for Comparative and International Studies. University of Zurich. Available at: http://www.oecd.org/env/cc/48351229.pdf
% of grey/blue/green areas affected by extreme weather conditions/events (e.g. Heat Island Effect, Flood, Rockfalls and/or Landslides, Forest/Land Fire)
Average length (in hours) of the public service interruptions (e.g. energy/water supply, public transport traffic, health/civil protection/emergency services)
Indicators that target the degree to which a system or species is affected, either adversely or beneficially, by climate variability or change. Often addressing intrinsic properties of an object resulting in susceptibility to a risk source.
% share of sensitive population groups (e.g. elderly (65+)/young (25-) people, lonely pensioner households, low-income/unemployed households) – compared to national average in year X in country X
%
output
Covenant of Mayors
% of (public/residential/tertiary) buildings retrofitted for adaptive resilience
Type and extent of assets strengthened and/or better managed to withstand the
effects of climate change
#
output
GEF7)GEF (2014). Updated results-based management framework for adaptation to climate change under the least developed countries fund and the special climate change fund. GEF/LDCF.SCCF.17/05/Rev.01.
Physical infrastructure improved to withstand climate
change and variability-induced stress
#
output
CCIS
Ecosystem services and natural assets maintained or improved under climate change and variability-induced stress
#
output
CCIS
Percentage green and blue surface area in the neighbourhood in relation to paved and built surface area
%
output
City of Almere8)Bosch, P.R., Rovers, V. (2016). Growing Green indicatoren AIB Almere – Nulmeting stadsindicatoren 2016. TNO confidential report.
Adaptive Capacity
Indicators that assess the ability of systems, institutions, humans and other organisms to adjust to potential damage, to take advantage of opportunities, or to respond to consequences.
Percentage of large businesses (500+ employees) within the city that have developed business continuity plans in accordance with ISO 22301
%
output
ISO
Percentage of households and businesses with insurance coverage for hazard risks
%
output
ISO
Percentage of total insured value to total value at risk of hazard threats within the city
%
output
ISO
City electricity supply capacity as a percentage of total demand
%
output
ISO
Imported energy as a percentage of final energy consumption
%
output
ISO
Number of days that city fuel supplies could maintain essential household functions
(through alternative sources)
#
output
ISO
Percentage of natural areas within the city that have undergone ecological
evaluation for their protective services
%
output
ISO
Percentage of emergency responders able to communicate during a disaster event via professional mode radio (PMR) (e.g., TETRA), satellite telephony, or privilegedaccess mobile communication networks (e.g., MTPAS)
%
output
ISO
Percentage of local severe weather warnings issued by national metrological
agency that are received in timely fashion by city emergency responders
%
output
ISO
Number of people covered by multi-hazard early warning system per 100 000
population
%
output
ISO
Percentage of population with access to primary health care facilities
%
output
ISO
Percentage of population with health insurance
%
output
ISO
Capacity of emergency shelters per 100 000 population
#
output
ISO
Percentage of new buildings completed within the city in the past 5 years that
complies with current building codes and standards
%
output
ISO
Percentage of building stock that complies with current building codes and
standards
%
output
ISO
Percentage of population with access to social assistance programs
%
output
ISO
Number of different supply sources providing at least 5 percent of water supply
capacity
#
output
ISO
Percentage of population that can be supplied water by alternative methods for 72 hours during disruption
%
output
ISO
Percentage of population that could be served by intra-city food reserves for 72
hours
%
output
ISO
Number of media types used to alert people in an emergency
#
output
ISO
Percentage of city government data with secure back-up remote storage
%
output
ISO
Number of climate responsive tools developed and tested
#
output
GIZ
Number of vulnerable stakeholders using climate responsive tools to respond to climate variability or climate change
Percentage of population living in flood and/or drought-prone areas with access to rainfall forecasts
%
output
GIZ
Number of people supported to cope with the effects of climate change through the availability of a service or facility
#
output
GIZ
Number of wave recorders installed along coastal areas
#
output
GIZ
Number of existing meteorological stations per territorial unit
#
output
GIZ
Number of properties with retrofitted flood resilience measures; water meters; water efficiency measures; cooling measures
#
output
GIZ
Uptake of early warning systems (UV and air/water quality)
#
output
GIZ
Number of businesses that have changed their working hours
#
output
GIZ
Number of water efficiency measures used in energy generation/extraction
#
output
GIZ
Number of water companies rationing water during droughts
#
output
GIZ
Area of land under ‘landscape scale’ conservation
hectare
output
GIZ
Uptake of riparian tree planting
#
output
GIZ
Number of businesses with insurance for extreme weather events
#
output
GIZ
Percentage of companies assessing risks and opportunities from extreme weather and reduced water availability to their supply chains
%
output
GIZ
Priority areas for precautionary flood protection
#
output
GIZ
Energy Storage Capacity
%
output
GIZ
Percentage of climate resilient roads in the country
%
output
GIZ
Number of cubic metres of water conserved
m3
output
GIZ
Percentage of water demand being met by existing supply
%
output
GIZ
protection and improved flood emergency preparedness
output
GIZ
Number of people with diversified income
#
output
GIZ
Extent of adoption of climate-resilient technologies/ practices
%
output
GEF
Number of people/ geographical area with access to improved climate information services
#
output
GEF
Number of people/ geographical area with access to improved, climate-related
early-warning information
#
output
GEF
Population benefiting from the adoption of diversified, climate-resilient livelihood options
#, % of targeted population
output
GEF
Number of targeted institutions with increased capacity to minimize exposure to climate variability risks
#
output
CCIS
Percentage of targeted population aware of predicted adverse impacts of climate change, and of appropriate responses
%
output
CCIS
Percentage of households and communities
having more secure (increased) access to
livelihood assets
%
output
CCIS
The extent to which adaptation options have been considered in the project
Likert
output
CITYkeys9)Bosch et al. (2017). CITYkeys indicators for smart city projects and smart cities. H2020 CITYkeys D1.4. Available at: http://www.citykeys-project.eu/citykeys/resources/general/download/CITYkeys-D1-4-Indicators-for-smart-city-projects-and-smart-cities-WSWE-AJENUD
Number of awareness-raising events targeting citizens and local stakeholders
#
activity
Covenant of Mayors
Number of training sessions targeting staff
#
activity
Covenant of Mayors
Number of public awareness campaigns on water efficiency
#
activity
GIZ
Percentage of trade and industry chambers using and distributing climate information
%
activity
GIZ
Number of urban adaptation best practices disseminated
#
activity
GIZ
Number of government staff that have received training on adaptation
#
activity
GIZ
Degree of integration of climate change into development planning
Likert
activity
GIZ
Number of policies and coordination mechanisms explicitly addressing climate change and resilience
#
activity
GIZ
Number of policies, plans or programmes introduced or adjusted that mainstream climate risks
#
activity
GIZ
Percentage of municipalities with local regulations considering adaptation and vulnerability assessment results
%
activity
GIZ
Percentage of new hydroelectric projects that consider future climate risks
%
activity
GIZ
Existence of interministerial/ intersectoral commissions working on adaptation
Number of inventories of climate change impacts on biodiversity
#
activity
GIZ
Number of methodological guides produced to assess impacts of extreme weather events on transport systems
#
activity
GIZ
Percentage of schools in the city that teach emergency preparedness within their
curriculum
%
activity
ISO
Percentage of the police force that has undertaken disaster response training in the last 5 years
%
activity
ISO
Percentage of the population that has been engaged with emergency preparedness information by responsible authorities
%
activity
ISO
Value of disaster reserve funds as a percentage of total municipal reserves
€
input
ISO
Number of building inspectors per 100 000 population
#
input
ISO
Percentage of city population reached by risk communication activities
%
activity
ISO
Number of years since city hazard maps have been updated
#
activity
ISO
Number of years since city flood-risk assessment was undertaken
#
activity
ISO
Number of years since last city-wide critical asset assessment
#
activity
ISO
Number of years since the city evacuation plan was updated
#
activity
ISO
Number of reviews of city-wide emergency protocols undertaken in the past 5
years
#
activity
ISO
Percentage of city area covered by publicly available, up-to-date hazard maps
%
activity
ISO
Number of times that city multi-stakeholderrisk assessments happen per year
#
activity
ISO
Percentage of city departments with access to risk assessments
%
activity
ISO
Percentage of city departments that consider the results of the risk assessment
process in their planning activities
%
activity
ISO
Public awareness activities carried out and population reached
#
activity
GEF
Number of people trained to identify, prioritize, implement, monitor and
evaluate adaptation strategies and measures
#
activity
GEF
Capacities of regional, national and sub-national institutions to identify,
prioritize, implement, monitor and evaluate adaptation strategies and measures
#
activity
GEF
Institutional arrangements to lead, coordinate and support the integration of climate changeadaptation into relevant policies, plans and associated processes
#
activity
GEF
Regional, national and sector-wide policies, plans and processes developed and
strengthened to identify, prioritize and integrate adaptation strategies and
measures
#
activity
GEF
Sub-national plans and processes developed and strengthened to identify,
prioritize and integrate adaptation strategies and measures
#
activity
GEF
Countries with systems and frameworks for the continuous monitoring, reporting
and review of adaptation
#
activity
GEF
Risk and vulnerability assessments, and other relevant scientific and technical
assessments carried out and updated
#
activity
GEF
Relevant threat and hazard information generated and disseminated to stakeholders on a timely basis
activity
CCIS
Climate change priorities are integrated into national development strategy
activity
CCIS
The extent to which the city has developed and implemented a climate resilient strategy
Likert
activity
Citykeys
€ investment in adaptation research (e.g. soil conservation, water/energy efficiency…) by the city / by other stakeholders
Euro
input
Covenant of Mayors
€ investment in education / in health & emergency systems by the city
Euro
input
Covenant of Mayors
Funding for climate-adapted construction and refurbishment
Euro
input
GIZ
Total sum of investments in programmes for the protection of livestock
Euro
input
GIZ
Emergency planning budget as a percentage of total city budget
%
input
ISO
Average percentage of the city’s transport budget spent on maintenance and upgrades over the past 5 years
%
input
ISO
Percentage of annual budget for storm water infrastructure spent on upgrades
%
input
ISO
Stressors
Indicators that signify a change or trend unrelated to climate that can exacerbate the impact of climate hazards.
Population density (compared to national/regional average in year X in country/region X)
#/hectares
na
Covenant of Mayors
Current energy consumption per capita vs. projections 2020/2030/2050
GJ
na
Covenant of Mayors
Current water consumption per capita vs. projections 2020/2030/2050
m3
na
Covenant of Mayors
% change in water loss (e.g. due to leakage in the water distribution system)
%
na
Covenant of Mayors
% change in solid waste collected / recycled / disposed of / burned
%
na
Covenant of Mayors
% change in water consumption for agriculture/irrigation
%
na
Covenant of Mayors
% change in tourist flows
%
na
Covenant of Mayors
% change in tourism activities
%
na
Covenant of Mayors
% of treated wastewater
%
na
GIZ
Exposure
Indicators that the presence of people, livelihoods, species or ecosystems, environmental services and resources, infrastructure, or economic, social, or cultural assets in places that could be adversely affected.
Percentage of city land area located in a flood zone
%
na
ISO
Percentage of health facilities situated in high-risk zones
%
na
ISO
Percentage of educational facilities situated in high-risk zones
%
na
ISO
Percentage of properties located in high-risk zones
%
na
ISO
Percentage of population at high risk to natural hazards
%
na
ISO
Hazard
Indicators that assess the potential occurrence of a natural or human-induced physical event or trend or their physical impact (i.e. flooding, heat stress, drought).
ESPON10)ESPON (2011). ClimateClimate Change and Territorial Effects on Regions and Local Economies. Applied Research Project 2013/1/4. Final Report Annex 9 – Indicators
Rooijen, T., Nesterova, N. & Guikink, D., 2013. Applied framework for evaluation in CIVITAS PLUS II. Deliverable 4.10 of CIVITAS WIKI of CIVITAS initiative. Cleaner and better transport in cities (CIVITAS WIKI)
2.
↑
A Likert scale is a five (or seven) point scale which is used to express the analyst or independent expert estimate on the indicator
3.
↑
ISO/TC268/WG2 N100 – ISO/WD 37123 Resilient Cities Standard March 2017. Working Draft 2017-03-28
Stadelmann M., Michaelowa A., Butzengeiger-Geyer S., Kohler M. (2011), Universal metrics to compare the effectiveness of climate changeadaptation projects. Centre for Comparative and International Studies. University of Zurich. Available at: http://www.oecd.org/env/cc/48351229.pdf
6.
↑
Neves A; Blondel L; Brand K; Hendel Blackford S; Rivas Calvete S; Iancu A; Melica G; Koffi Lefeivre B; Zancanella P; Kona A. The Covenant of Mayors for Climate and Energy Reporting Guidelines; EUR 28160 EN; doi:10.2790/586693
7.
↑
GEF (2014). Updated results-based management framework for adaptation to climate change under the least developed countries fund and the special climate change fund. GEF/LDCF.SCCF.17/05/Rev.01.
8.
↑
Bosch, P.R., Rovers, V. (2016). Growing Green indicatoren AIB Almere – Nulmeting stadsindicatoren 2016. TNO confidential report.
9.
↑
Bosch et al. (2017). CITYkeys indicators for smart city projects and smart cities. H2020 CITYkeys D1.4. Available at: http://www.citykeys-project.eu/citykeys/resources/general/download/CITYkeys-D1-4-Indicators-for-smart-city-projects-and-smart-cities-WSWE-AJENUD
10.
↑
ESPON (2011). ClimateClimate Change and Territorial Effects on Regions and Local Economies. Applied Research Project 2013/1/4. Final Report Annex 9 – Indicators