Evaluation guide

CONTENTS

I. Introduction

Purpose of evaluation, users of this guide

Accountability for monitoring and evaluation

Glossary of terms

II. Standardised components of monitoring and evaluation in project proposals

Introduction to evaluation approach at proposal stage

Monitoring and project preparation

Frameworks for project evaluation

Project evaluability (baseline data) - monitoring and evaluation indicators

Content of evaluation section in project proposals: standard templates

Sample evaluation section in investment project proposals

Sample evaluation section in non-investment project proposals

III. Evaluation and monitoring of project implementation

Introduction to monitoring of the implementation of investment projects

Indicators of progress in the implementation of investment projects (definition; rationale; optimum number)

List of key indicators for investment projects

Implementation performance and decision-making (When are decisions required? Who needs to know/decide?)

Monitoring of non-investment projects

Relationship of monitoring to evaluation

Mid-term evaluation

IV. Project completion reporting

Rationale for project completion reporting

Content of investment project completion report

Standard templates

Sample of investment project completion report

Content of non-investment project completion report

Sample of non-investment project completion report

V. Conducting evaluations under the Multilateral Fund

A. Background and rationale for evaluation

B. Timing, scope and focus of multilateral fund evaluations

1. Timing

2. Scope

3. Focus

C. Evaluation management and procedures

1. Initiating a specific evaluation

2. Evaluation work plan

3. Roles and responsibilities

D. Procedures for implementing work plans

1. Selecting projects for evaluation

2. Evaluation framework matrix

3. Activity/effort analysis

4. Data collection plan

5. Budget

6. Collecting and analysing data (see later section for general aspects)

7. Reporting

E. Data collection and analysis

1. Types of data

2. Data sources

3. Methods of data collection

4. Instrumentation

5. Indicators

Appendices

Appendix I: Sectoral evaluation matrix

Appendix II Non-investment project evaluation matrix – Training projects

Appendix III: Non-investment project evaluation matrix – Institutional strengthening projects

Glossary of Terms

For the purposes of this Guide, the following definitions will be assumed:

Activity	Action taken or work performed within a project in order to transform inputs into outputs.
Assumption	External factors, influences, situations or conditions which are necessary for project success, worded in terms of positive conditions. Assumptions are external factors which are quite likely but not certain to occur and which are important for the success of the project, but which are largely or completely beyond the control of project management.
Baseline benchmarks	Data that describe the situation before any project intervention.
Effectiveness	A measure of the extent to which a project is successful in achieving its planned objectives or results.
Efficiency	A measure of the extent to which inputs were supplied and managed and activities organised in the most appropriate manner at the least cost to produce the required outputs.
Evaluability	The extent to which a project has been defined in such a way as to enable evaluation later on.
Ex-post evaluation	An evaluation conducted after project completion.
Findings vs. conclusions	A finding is a factual statement (e.g. 405 tonnes of ODS were phased out). A conclusion is a synthesis of findings incorporating the evaluator’s analysis (e.g. the project was not efficient since it cost twice as much to phase out 3 tonnes of ODS compared to the costs in other similar projects).
Impact/effect	An expression of the ultimate planned and unplanned changes brought about as a result of a project; the planned and unplanned consequences of the project. In projects that follow logical frameworks, effects are generally related to the purpose, impacts to the goal.
Indicator	An explicit statistic or benchmark that defines how performance is to be measured.
Input	Resources such as human resources, materials, services, etc., which are required for achieving the stated results by producing the intended outputs through relevant activities.
Objective	Expresses the particular effect which the project is expected to achieve if completed successfully and on time.
Output	The physical products, institutional and operational changes or improved skills and knowledge to be achieved by the project as a result of good management of the inputs and activities.
Project	A planned undertaking designed to achieve certain specific objectives/results within a given budget and specified time period through various activities.
Stakeholders	Interested and committed parties; a group of people with a vested interest in the phenomena under study.

V. Conducting evaluations under the Multilateral Fund

A. Background and rationale for Evaluation

In the context of the Multilateral Fund, an evaluation may be defined as "an assessment, as systematic and independent as possible, of projects or clusters of projects, their design, implementation and results. The aim of evaluation is to assess the continued relevance of Fund support to various types of projects in various regions, the efficiency of project implementation, and the effectiveness of such projects in achieving the Fund’s/project’s objectives, as well as any lessons that can help guide future policy and practice".

The purpose of Multilateral Fund evaluations is to provide information on:

overall Fund performance in reducing ODS according to established targets;
the effectiveness of projects in particular sectors, and of non-investment projects;
the strengths and limitations of various types of projects;
the major causes of observed failures to reach targets;
possible actions that might improve performance of the Fund.

The Executive Committee and all other stakeholders, such as Article 5 countries and Implementing Agencies, are intended to benefit from evaluation information and lessons learned that will help them improve their efforts in achieving the goals of the Montreal Protocol. The Executive Committee acknowledges evaluation priorities through a budget for evaluations approved annually.

The Executive Committee considered the Multilateral Fund’s work programme and work plan for monitoring and evaluation at its Twenty-second Meeting and adopted deliverables 1, 2, 4, and 5 in the work programme and outputs 1 to 4 in the work plan.

Output 1 mandates the preparation of an Evaluation Guide covering both investment and non-investment projects. This guide incorporates and builds on the guidelines and procedures already developed by the Implementing Agencies, including, inter alia:

project baseline data;
data from Progress and Completion reports;
evaluation data collected by the Implementing Agencies;
established guidelines for evaluation data collection.

Timing, scope and focus of Multilateral Fund evaluations

Evaluations can be classified according to their timing, their scope and their focus.

1. Timing

Evaluations may be undertaken during project implementation or after projects have been completed as characterised below.

Evaluation timing	Description	Rationale
Mid-term evaluation	An evaluation of a specific project, done at any time during project implementation.	Projects that may require mid-term evaluations include those that are very large, that have high risks associated with their design, that are using novel technology, or that are experiencing problems, such as implementation delays.
Ex-post evaluation	Evaluation of one or more projects that takes place at some point after operational project completion.	Such evaluations are intended to confirm that projects performed as reported, and to facilitate future decision-making by learning about strengths, weaknesses and unplanned effects of projects of various types.

2. Scope

The scope of Fund evaluations will respond to particular needs which will be identified by the Executive Committee’s evaluation work programme. Evaluations may examine a collection of projects in a sector or region, or may focus on a single project.

Type of evaluation	Scope
Evaluation of a single investment project	Such an evaluation would focus on a single project, but would examine the context in which it is situated. The project may be in the process of being implemented, or it may be completed.
Evaluation of projects within a sector (sectoral or thematic)	Such evaluations would normally deal with a group of projects within the sector. They could include both investment and non-investment projects, and both completed and non-completed projects. Specific evaluation studies may relate to a designated geographic area or theme, or be limited in other ways.
Evaluation of non-investment projects	Such evaluations would normally deal with a group of completed projects and may be designed to focus on one or more of a combination of particular issues, sectors, Implementing Agencies, or geographic areas.

3. Focus

The focus of an evaluation refers to the types of issue it is to address. These are described by the major questions an evaluation is expected to answer. The Executive Committee has considered the following as illustrative of key potential questions for sectoral and thematic evaluations (training and institutional strengthening) supported by the Fund. The following tables provide possible evaluation questions for sectoral, training, and institutional strengthening projects. (Appendices I-III provide additional examples).

Sectoral evaluations	Training	Institutional strengthening
Effectiveness and effects	Effectiveness and effects	Effectiveness and effects
In general, how effective have the various types of investment projects been in achieving ODP targets and reducing ODS within the sector?	To what extent is training supported by the Fund effective?	To what extent is institutional strengthening supported by the Fund effective?
Was the old technology successfully discontinued?	Is training impacting the enabling environment in ways that support achievement of the Fund’s objectives?	Is institutional strengthening impacting the enabling environment in other ways that support achievement of the Fund’s objectives?
What have been the effects of the new technology on operating costs? On market demand? On safety and environment?	Is technical training leading to more effective technical conversions?
How sustainable are the project results?
Efficiency	Efficiency	Efficiency
What were the major implementation challenges and how were they overcome? How efficient are the various approaches to project implementation (e.g.: financial intermediary; local executing agency; ozone unit)?	Are training activities planned and implemented in the most cost-effective way? How could cost-effectiveness be improved?	Are institutional strengthening activities planned and implemented in the most cost-effective way? How could cost-effectiveness be improved?
Which aspects of investment projects in this sector (equipment, technical assistance, training) worked very well?	Do Implementing Agencies include suitable monitoring and evaluation of training activities that enable such activities to benefit from participant feedback?	Have expenditures been allocated appropriately among the allowable categories?
How effective was transfer of technology in the various projects and regions?		Have regional network activities been implemented in a cost effective way?
Project design	Project design	Project design
What were the critical factors in the enabling environment that have affected project success? How have they contributed to or hindered project efficiency and effectiveness?	Are Implementing Agencies addressing the most pressing training needs?	Was the chosen mechanism appropriate for the institutional strengthening tasks?
Did the design of various types of projects change prior to implementation?	To what extent are training activities suitably targeted to reach people and institutions with a need for such support?	Did the original provisions reflect the needs?
Was the level of funding provided by the Fund understood by the enterprise and appropriate to the need and incremental cost requirements?	Are training programmes designed in conformity with contemporary international standards for training?	Did original project documents contain adequate information for subsequent evaluation?
Did original project documents contain adequate information for subsequent evaluation?	Did original project documents contain adequate information for subsequent evaluation?
Lessons learned	Lessons learned	Lessons learned
What lessons have been learned that may be useful in guiding future project preparation, approval, or implementation?	What lessons have been learned that may be useful in guiding future project preparation, approval, or implementation?	What lessons have been learned that may be useful in guiding future project preparation, approval, or implementation?
What lessons have been learned about monitoring and evaluation under the Fund?	What lessons have been learned about monitoring and evaluation under the Fund?	What lessons have been learned about monitoring and evaluation under the Fund?

C. Evaluation management and procedures

The general process for approving and conducting evaluations under the Fund is depicted below.

The Sub-Committee on Monitoring, Evaluation and Finance recommends the annual evaluation work programme and work plan of the Multilateral Fund for approval by the Executive Committee. The approved work programme and plan of the Fund on monitoring and evaluation is the normal basis on which specific evaluations are carried out; however, the Executive Committee may decide to conduct special evaluations at any time. The annual work programme provides, in the form of proposed outputs, a summary description of specific evaluations to be undertaken. The management of these evaluations is the responsibility of the Secretariat as described below.

1. Initiating a specific evaluation

The Monitoring and Evaluation Officer within the Secretariat has overall responsibility for managing evaluations approved by the Executive Committee. For each evaluation, it is the responsibility of the Monitoring and Evaluation Officer to prepare terms of reference (TOR) leading to the contracting of external consultants. The content of the TOR is as follows:

Terms of reference

Background

Reasons for Evaluation

Scope and Focus

Specific Evaluation Requirements

Estimated Level of Effort

Description of Required Evaluators

Schedule for the Evaluation

Indicative Costs

Using established contracting procedures, the Secretariat will contract a firm or consultant to conduct the evaluation. The Secretariat typically issues a letter of invitation to qualified consulting firms to submit the qualifications of personnel proposed for the assignment and professional fees for the assignment. The TOR are normally included with this invitation to bid.

2. Evaluation work plan

Once evaluators have been contracted, the first deliverable in the contract is normally a work plan for the assignment, with the details worked out in consultation with the Secretariat. The suggested outline for such an evaluation work plan is shown below.

Evaluation work plan outline

Overview

Evaluation team

Project selection

Evaluation matrix

Methodology

Activity/effort analysis

Data collection plan

Budget

The evaluation work plan is an important control document as it supplements the contract and enables the Monitoring and Evaluation Officer to exercise control over the quality of the evaluation. The evaluation work plan will conform to the general requirements of this guide and will continue to evolve in matters of operational detail.

3. Roles and responsibilities

a) Evaluation Team

In order to benefit from a range of perspectives, and to ensure a balance of independent views and a mix of expertise, evaluations are normally conducted by teams of independent experts who are not directly linked to the preparation and/or implementation of projects and activities approved under the Multilateral Fund. These teams are contracted under the normal procedures for contracting of consultants. The specific composition of each evaluation team will vary according to the evaluation needs and cost effectiveness considerations. Evaluation teams for a simple project evaluation may include as few as one or two external consultants.

Each evaluation conducted by a team will involve an Evaluation Team Leader with expertise related to the work of the Multilateral Fund, and/or ODS technology, and/or evaluation methodology, experienced in leading evaluation teams in international contexts. Evaluation teams will be contracted by the Fund Secretariat. The Team Leader’s role is to:

Lead the evaluation team in all aspects of the work, so as to produce all required outputs according to agreed standards and time frames;
Be responsible for co-ordinating the implementation of the required evaluations;
Liase with the Evaluation Officer within the Secretariat;
Participate with the team in data collection and analysis;
Be responsible for drafting the evaluation report;
Submit reports that respond to the TOR to the Secretariat.

b) Multilateral Fund Secretariat

The Fund Secretariat ensures that evaluations relate to the evaluation needs of the Fund, the decisions of the Executive Committee and the requirements of the Executive Committee’s work programme on monitoring and evaluation. The role of the Secretariat is to:

Manage the evaluation process;
Provide an ongoing link between the evaluation and the Secretariat;
Approve the evaluation work plan developed by the Evaluation Team Leader;
Facilitate communication between the evaluation team and Implementing Agencies, participating Article 5 countries and bilateral agencies;
Provide technical expertise and participate in field missions as required;
Provide data from the Secretariat’s databases and archives;
Review final evaluation report to ensure it meets the requirements of the TOR and has adequate technical quality.

c) Implementing agencies

Implementing agencies are expected to support the evaluation process by:

Being responsive to the requirements of evaluation team members;
Meeting the evaluators at Headquarters and/or in field offices as required;
Facilitating meetings with financial intermediaries and enterprises as appropriate;
Advising the evaluation team on suitable approaches for data collection if requested;
Providing relevant data on projects, enterprises and their context;
Commenting on the accuracy of data in report drafts;
Contributing to the formulation of lessons learned.

d) Article 5 Countries

Involvement of Article 5 countries is the key to improving the Fund’s performance reducing ODS. Country representatives such as Ozone Officers are important contributors to the work of evaluation teams. The role of Article 5 country representatives is to:

Meet with the evaluators during field missions;
Advise the evaluation team on suitable approaches for data collection if requested;
Provide relevant data and interpretation on projects implemented within the country;
Facilitate the collection of data within government departments and on site visits to enterprises;
Advise on local product markets;
Comment on the accuracy of data in report drafts;
Contribute to the formulation of lessons learned.

D. Procedures for implementing work plans

1. Selecting projects for evaluation

Sometimes the selection of specific projects to be evaluated will be specified in the TOR. In other situations, such as with sectoral evaluations, all projects that have certain characteristics will be reviewed, but at different levels of detail as shown below:

The Evaluation Team Leader, in consultation with the Monitoring and Evaluation Officer, and within the context of the approved work programme, will make the technical decision about the particular projects which will be included in an evaluation, and at what level of examination. The selection of projects for site visits will depend on a variety of factors including the needs for coverage, cost efficiency, and the scale and type of projects (e.g.: demonstration; completed or ongoing).

2. Evaluation framework matrix

The framework for data collection and analysis is recorded in an evaluation matrix. This matrix outlines the key questions and sub-questions to be addressed, and shows the indicators and sources of data to be included in the data analysis relative to each question.

Three generic evaluation matrices (including possible evaluation questions, indicators and sources of data) are presented in Appendices I-III): Appendix I: a matrix for a sectoral evaluation, Appendix II: a matrix for an evaluation of training projects, and Appendix III: a matrix for an evaluation of institutional strengthening projects.

Using the generic evaluation matrix as a guide, the Team will refine the evaluation questions and develop the specific indicators and data sources required to address the specific TOR.

3. Activity/effort analysis

The work plan will include a table of the activities to be undertaken, who will undertake them, and the amount of time planned for each. This table will link to the personnel costs in the budget. The Team will divide responsibilities so that all aspects of data collection and analysis are efficient. In practice, this may involve different team members conducting different site and country visits.

4. Data collection plan

The Evaluation Team Leader will develop a detailed data collection plan; assign specific roles and responsibilities; schedule specific activities such as site visits; and develop the necessary data collection methods and instruments.

In developing the detailed data collection plan, the Team may review available Implementing Agency reports and project completion reports. The Evaluation Team Leader may make a preliminary request for data from Implementing Agencies and from Ozone Officers.

5. Budget

The work plan will include a budget for the costs of personnel, travel, and other expenses. This budget is indicative of the emphasis of various components of the evaluation; however, contracting may be on a fixed fee basis with payments linked to specific deliverables.

6. Collecting and analysing data (see later section for general aspects)

a) Initial analysis

The first level of analysis will be through the existing data found in Implementing Agency reports, of which the Project Completion Reports are particularly important. The initial data analysis will help the team to understand what data are not available and need to be collected elsewhere, and will help define issues that require follow-up.

b) Country field missions

Field missions are an important supplement to existing reported data. They provide an opportunity to validate available data, to supplement it, and to collect data on developments following operational completion of a project.

Once the dates of field missions are known, the Secretariat informs the concerned Article 5 countries and Implementing Agencies of the start of the evaluation field mission. The nature of their involvement and expected support will be indicated.

Country missions may begin with in-country briefings with the Ozone Officer to review and obtain input and assistance on the data collection plan.

The purpose of site visits will be to gain additional understanding by confirming and/or complementing information available from existing data sources, and situating the findings in the context. During the mission, data will be collected according to the data collection plan (through interviews and visits with government representatives, Implementing Agencies’ field offices, enterprises, and bilateral donors as applicable) with modifications made as needed and as agreed by the Team.

c) Non-investment evaluations

As in other types of evaluations, studies of non-investment projects will involve analysis of extensive existing data (e.g. internal evaluations of training workshops, country programmes and reports). These tend to be self-reported data that are collected before or at project completion. In addition, evaluations emphasising effects and impact will require follow-up or tracer study methods such as questionnaire surveys, telephone interviews, electronic communication, and, when warranted, visits to the field.

7. Reporting

The Team Leader bears overall responsibility for the final analysis and reporting. Following accepted practice for sound evaluation, the Team Leader will attempt to share drafts of relevant sections of reports with involved Implementing Agencies and Article 5 countries to give them the opportunity to correct factual errors in the drafts. While every attempt will be made to ensure factual accuracy, the substantive conclusions of the evaluation are the responsibility of the evaluators.

The Evaluation Team Leader will submit the report to the Monitoring and Evaluation Officer. The latter ensures conformity to the TOR, technical accuracy and quality, and may require revisions before submitting the report to the Sub-Committee.

a) Sectoral evaluations

The outline of each evaluation report will be tailored to the specific TOR and other requirements. A suggested outline is provided below to indicate the type of reporting desired. The emphasis is on clear reports that state what was found, the resulting conclusions and recommendations directed at specific stakeholders. Every report should contain a concise executive summary of 2-5 pages.

Sectoral evaluation report outline

Executive summary

Introduction

Background

Description of projects

Investment

Non-investment

Evaluation methodology

Organisation of report

Design and Rationale

Assumptions

Sector context

Context - enabling environment

Design

Changes

Evaluability

Alternative designs

Cost

Planned/actual

Cost sharing

Sources of extra cost

Effectiveness and effects

Achievement of results

ODS phase-out

Institutional strengthening at operational level

Differences by sector, region

Equipment rendered unusable

Effects on enterprises

Effects on safety/environment

Implementation efficiency

Conversion of inputs to outputs

Differences by component

Differences by type of project, region, agency

Project management

Sustainability

Conclusions

Recommendations and follow-up

Lessons Learned

Annex 1 - TOR

Annex 2 - Evaluation matrix

Annex 3 - Organisations visited

Annex 4 - Project list

b) Reporting on evaluations of non-investment projects

The outlines of the evaluation reports for non-investment projects will follow the key questions of the evaluation framework matrix. A sample outline for a training evaluation and for an institutional strengthening evaluation are shown below.

Training evaluation report outline

Executive summary

Introduction

Background

Description of projects

Evaluation methodology

Organisation of report

Design and rationale

Assumptions

Context - enabling environment

Design

Relevance of plan

Changes

Cost

Planned/actual

Cost sharing

Sources of extra cost

Effectiveness and Effects

Achievement of targets

Effects on enterprises

Effects on safety/environment

Implementation efficiency

Delivery of inputs

Project management

Sustainability

Conclusions

Recommendations

Lessons Learned

Annex 1 – TOR

Annex 2 – Evaluation matrix

Annex 3 – Organisations visited and interviews conducted

Annex 4 - Project list

Institutional strengthening evaluation report outline

Executive summary

Introduction

Background

Description of IS funding

Evaluation methodology

Organisation of report

Design and rationale

Assumptions

Design

Relevance of plan

Level of responsibility

Variations in different category countries

Changes in roles of units

Cost

Planned/actual

Cost sharing

Sources of extra cost

Effectiveness and effects

Achievement of objectives: data-gathering; information exchange; dissemination; monitoring; co-ordination

Fulfilment of obligations

Differences by sector, region, category of country

Regional networks

Effects on ODS phase-out

Efficiency

Time lags in implementation

Capital expenditures

Professional staff

Operational costs

Regional networks

Sustainability

Need for continuation

Government plans

Conclusions

Recommendations

Lessons Learned

Annex 1 - TOR

Annex 2 - Evaluation matrix

Annex 3 - Organisations visited and interviews conducted

Annex 4 - Project list

E. Data Collection And Analysis

1. Types of Data

Data can be hard or soft, quantitative or qualitative. Hard (quantitative) data generally include technical or financial facts such as the amount of ODS phased-out through a project or the number of trainees who participated in a course. Soft (qualitative) data reflects perceptions or judgements. It includes both non-technical judgements such as the perceptions of people about what took place, and the expert judgement of an individual who is knowledgeable and experienced in a particular field. Valid evaluations try to obtain as many types of data from as many sources as possible. One of the rules of thumb of evaluation is that the more sources that confirm a finding, the more valid the finding.

2. Data sources

Evaluation studies draw from many data sources, as it is a combination of sources that lend strength to evaluation findings. Some of the major sources include the following:

Documents:

Project proposals;

Project documents;

Project progress reports;

Project completion reports;

Country programmes.

Interviews:

Government officials;

Persons involved in any aspect of project implementation;

Persons involved in training and institutional strengthening supported by the Fund;

Bilateral donors involved in the sector;

Managers (e.g.: production; marketing) and technical personnel from involved enterprises;

Persons involved in product markets (e.g.: distributors; retailers).

Enterprises:

Equipment and production processes;

Production reports;

Product sampling.

Note that there are instances where data are missing or not available, in which case alternative sources may provide data with which to address the questions. In extreme cases, there are no data and the questions cannot be answered, at least at the time of the evaluation. This would suggest recommendations for improved data systems in future project approvals and implementation.

3. Methods of data collection

It is expected that the Evaluation Team will use a combination of methods of data collection and analysis, including:

review of project proposals and reports, especially project completion reports;

surveys and telephone interviews with project stakeholders;

country and on-site visits to enterprises, where the volume of projects warrants it;

selective sampling of products considered to be ozone-friendly may also be undertaken through market surveys.

Whatever methods are used, the evaluators will ensure the confidentiality of people who provided data by avoiding the use of interpretations and conclusions that could be traced back to the person providing them.

4. Instrumentation

Each evaluation team will also develop data collection instruments and procedures suited to the needs of particular evaluation studies and sites. The types of instruments normally used include:

Interview protocols:

Country officials;

Persons knowledgeable about project implementation;

Persons who have been supported by non-investment projects;

Other stakeholders (bilateral donors; persons involved with product markets).

Checklists:

Factors in the enabling environment;

Environmental and safety concerns.

Questionnaire surveys:

Training participant tracer surveys.

5. Indicators

Indicators are important quantifiable measures of various aspects of project performance. The amount of ODP phased-out is an example. The proportion of training participants who are successful in applying new skills is another. The time taken to reach agreed targets is a third. Each of the evaluation questions will be judged using one or more indicators of this type. The use of indicators helps make the rules of judgement transparent, and it provides a sound and rational basis for data analysis.

Sectoral evaluation report outline

Executive Summary

Introduction

Background

Description of projects

Evaluation methodology

Organisation of report

Design and rationale

Assumptions

Sector context

Regulation/legislation

Context - enabling environment

Design

Relevance of plan

Changes

Cost

Planned/actual

Cost sharing

Sources of extra cost

Effectiveness and effects

Achievement of targets

Differences by sector, region, etc.

Effects on enterprises

Effects on safety/environment

Sustainability

Implementation efficiency

Delivery of inputs

Project management

Conclusions

Recommendations

Lessons Learned

Annex 1 - TOR

Annex 2 - Evaluation matrix

Annex 3 - Organisations visited and interviews conducted

Annex 4 - Project list

Appendix I: Sectoral Evaluation Matrix

The following matrix includes generic questions, indicators and data sources. It is included to suggest the types of questions and approaches that may be useful; however, it is not intended to be prescriptive – each evaluation will need to develop a matrix that addresses its TOR.

Possible evaluation questions	Possible sub-questions	Possible indicators	Possible sources of data
Effectiveness and Effects
In general, how effective have the various types of investment projects been in achieving ODP targets and reducing ODS within the sector?	Were there differences by region or Implementing Agency? Were there differences by sub-sector? Were there differences by type of technology?	Baseline + ODS reduction Change in ODP Planned/actual target achievement	Project documents Enterprise data Country representatives Project implementation agencies
Was the old technology successfully discontinued?	For how long was the old technology in use after implementation of the project? How was the de-commissioned equipment rendered unusable?	% old technology destruction % of various means of disposal months for phase-out	Project documents Enterprise Country representatives Project implementation agencies
What have been the effects of the new technology on operating costs? On market demand? On safety and environment?	What were the effects on production following conversion? What were the effects of conversion on product quality, price, market acceptance? What were the effects on safety and the environment?	% change in products % change in costs % market penetration Changes in accident rates; safety guidelines	Project documents Enterprise Product testing Market sampling
How sustainable are the project results?	Has the project led to plans for additional conversions? What are the risks of re-conversion?	Number of inquiries about adopting technology Instances of re-conversion	Project documents Enterprise Country representatives Project implementation agencies Bilateral agencies
Efficiency
What were the major implementation challenges and how were they overcome? How efficient are the various approaches to project implementation (e.g.: financial intermediary; local executing agency; ozone unit)?	How has the capacity of local Implementing Agencies affected project efficiency and effectiveness? Have conversions complied with environmental/safety standards? Have new equipment or processes introduced new safety or environmental risks?	Time to various project milestones Frequency of specific contextual constraints Frequency of specific environmental or safety concerns	Project documents Enterprises Country representatives Project implementation agencies and associates
Which aspects of investment projects in this sector (equipment, technical assistance, training) worked very well?	Were there contextual factors that affected the implementation of certain components?	Frequency of specific contextual constraints	Project documents and IAs Enterprises Country representatives
How effective was transfer of technology in the various projects and regions?	What types of difficulty were encountered in obtaining non-ODS technology? Is there any evidence of conversion back to ODS? Have other producers demonstrated interest in adopting this technology?	Frequency of specific difficulties Instances of re-conversion Number of inquiries about adopting technology	Project documents Enterprises Country representatives Project implementation agencies Bilateral agencies
Project design
What were the critical factors in the enabling environment that have affected project success? How have they contributed to or hindered project efficiency and effectiveness?	Have there been effective changes in regulation and policy during project implementation? Are there constraints in the enabling environment that the Fund or country should attempt to address? Have training and institutional strengthening activities supported the success of investment projects? Were assumptions valid? Are there any contextual factors that should be a concern for future project approvals?	Checklist of critical factors in the enabling environment List of changes in legislation/regulation	Country representatives, IAs, project implementation agencies, enterprises, bilateral agencies Legislation, regulations
Did the design of various types of project change prior to implementation?	Did the technology implemented differ from the technology approved? Why and with what effects?	% of each alternative technology changed % popularity of alternative technologies	Project documents Enterprise Country representatives Project implementation agencies
Was the level of funding provided by the Fund understood by the enterprise and appropriate to the need and incremental cost requirements?	Did the cost change appreciably during implementation? If so, who paid the additional cost?	% change in project cost % cost borne by different stakeholders	Project documents Enterprise Country representatives Project implementation agencies
Did original project documents contain adequate information for subsequent evaluation?		Sufficient material available to complete evaluability checklist (e.g.: baseline data, training needs assessments include skill levels prior to training)	Project documents
Lessons Learned
What lessons have been learned that may be useful in guiding future project preparation, approval, or implementation?	What are the implications of the findings for additional and/or alternative information in future project proposals?		All stakeholders

Appendix II: Non-Investment Project Evaluation Matrix – Training Projects

The following matrix includes generic questions, indicators and data sources. It is included to suggest the types of question and approach that may be useful; however, it is not intended to be prescriptive – each evaluation will need to develop a matrix that addresses its TOR.

Possible Evaluation Questions	Possible Sub-Questions	Possible Indicators	Possible Sources of Data
Design
Are Implementing Agencies addressing the most pressing training needs?	Are training needs assessments conducted in conformity with contemporary international standards? Do programming priorities reflect priorities of key stakeholders?	Expert judgement Congruence of training demand and supply	Training experts Stakeholders: IAs, countries
To what extent are training activities suitably targeted to reach people and institutions with a need for such support?	Are policies and procedures for identification of training participants suitable for addressing identified needs?	Expert judgement	Training experts Stakeholders: IAs, countries
Are training programmes designed in conformity with contemporary international standards for training?	Do training workshops incorporate key principles for effective adult learning? Are training materials effective in supporting training outcomes?	Expert judgement Participant ratings of satisfaction; effectiveness of materials	Training experts Training participants Training manuals and materials
Did original project documents contain adequate information for subsequent evaluation?		Sufficient material available to complete evaluability checklist (e.g.: baseline data, training needs assessments include skill levels prior to training)	Project documents
Effectiveness and Effects
To what extent is training supported by the Fund effective?	Are participants learning the intended knowledge and skills? Is training being applied on the job? If not, what are the constraints?	Skill performance; Knowledge acquisition % participants reporting successful transfer Frequency of constraints	Tests and records Training participants Ozone Units Enterprises
Is training impacting the enabling environment in ways that support achievement of the Fund’s objectives?	What policies, regulations, procedures have been initiated by countries as a result of training programmes?	Frequency of targeted changes to regulations, etc. (e.g.: customs and import, licensing, re-export, non-compliance measures)) Degree of implementation of Article 4 of the Montreal Protocol Extent of financial support of ODS phase-out activities	Training participants Ozone Units Enterprises IAs
Is technical training leading to more effective technical conversions?		Reduced time for introduction of new technology	Enterprises Project completion reports
Efficiency
Are training activities planned and implemented in the most cost-effective way? How could cost-effectiveness be improved?	What are unit training costs, and how do they compare with costs of other international training of this type? What is the breakdown of training costs and are there ways to reduce cost components without negatively affecting quality?	Cost comparisons Expert judgement	Budgets financial reports Training experts Other UN agencies
Do Implementing Agencies include suitable monitoring and evaluation of training activities that enable such activities to benefit from participant feedback?	Does M&E address all the steps in the training cycle: attitudes? learning? transfer? impact? How might monitoring and evaluation systems be improved?	Expert judgement	Training experts
Lessons Learned
What lessons have been learned that may be useful in guiding future project preparation, approval, or implementation?			All stakeholders

Appendix III: Non-Investment Project Evaluation Matrix – Institutional Strengthening Projects

Possible evaluation questions	Possible sub-questions	Possible indicators	Possible sources of data
Design
Was the chosen mechanism appropriate for the institutional strengthening tasks?	Is the designated mechanism a central national facility?	Degree of confidence in the mechanism	Ozone/Institutional strengthening experts Stakeholders: IAs; enterprises
Did the original provisions reflect the needs?	Was funding adequate for country requirements?	Amount of supplementary funding required	Government representatives Ozone unit
Did original project documents contain adequate information for subsequent evaluation?	Did the proposal conform to the requirements of the TOR and qualifying areas of expenditure? Did documents identify indicators?	Number of instances of non-congruence	Project documents
Effectiveness and effects
To what extent is institutional strengthening supported by the Fund effective?	Are ozone units collecting and processing data to fulfil national obligations as parties to the Protocol? Have units exchanged relevant information with other countries, etc. and disseminated information to end-users? Are capacities to co-ordinate phase-out activities being enhanced? Are capacities to monitor phase-out activities being enhanced? Have units served as a focal point for the Fund Secretariat and IAs, including reporting?	Extent of obligations for data collection and reporting to Meeting of Parties met Amount of information exchange and public awareness activities Improved co-ordination Improved monitoring Contributions to country programmes Adoption/Changes/ harmonisation of legislation and/or regulations	Ozone units Ozone Secretariat Enterprises Implementing agencies Fund Secretariat
Is institutional strengthening impacting the enabling environment in other ways that support achievement of the Fund’s objectives?	Have regional networks been effective in supporting institutional strengthening? What actions have been initiated by countries as a result of the institutional strengthening programme?	Ratings of the extent to which regional networks effective Frequency of various actions	Ozone Units Enterprises IAs Participants in regional networks
Efficiency
Are institutional strengthening activities planned and implemented in the most cost-effective way? How could cost-effectiveness be improved?	What has been the time lag in implementation and what are the reasons?	Planned/actual time variance	Reports of ozone units Ozone units
Have expenditures been allocated appropriately among the allowable categories?	What proportions have been allocated between capital and recurrent expenditures in various categories of country?	Proportions of budget	Proposals Reports Ozone Units
Have regional network activities been implemented in a cost effective way?	Have network meetings conformed to standards of similar international gatherings of this type?	Cost comparisons	UNEP reports and budgets
Lessons Learned
What lessons have been learned that may be useful in guiding future project preparation, approval, or implementation?			All stakeholders

(UNEP/OzL.Pro/ExCom/23/68, Decision 23/5, para. 17).

(Supporting document: UNEP/OzL.Pro/ExCom/23/4)).