Work programme and work plan on monitoring and evaluation of the Multilateral Fund for a twelve-month period between 1997 - 1998

A. DELIVERABLES

Deliverable 1:               Action oriented indicators for monitoring project progress

Justification:                  UNEP/OzL.Pro/ExCom/21/36, para. 51(d), decision 21/36

Date of completion:        4 months upon approval of the work programme

Brief description of the final product:

This emanates from one of the recommendations of the Consultant’s report. Since a project monitoring system provides continuous supervision over the entire process of project implementation, it should have a number of signposts, or milestones, identified along the way which can assist management to easily track the movement of the project. If these signposts could be carefully chosen to be associated with a responsible party involved in the process that will facilitate identification of any hold-up and adoption of remedial action to be taken.

Currently, the existing reporting system has a number of indicators which focus the tracking on the commencement and the completion dates of a project, while what happens in between the two end points is hard to track.

The proposed modifications are to reduce the existing number of indicators, but add a few action-oriented signposts as discussed earlier. This will enhance the effective in-process supervision over the project implementation and facilitate identification of strategic remedial actions by the Executive Committee.

Deliverable 2:               A set of performance indicators for non-investment projects

Justification:                  Recommendation from the Consultant (UNEP/OzL.Pro/ExCom/21/30).

Date of completion:        4 months upon approval of the work programme

Brief description of the final product:

The existing progress reporting system does not adequately monitor non-investment projects. This is mainly due to the absence of good indicators which can effectively track the movement of these projects. These projects cover country programmes, institutional strengthening, training, networking, information clearing-house and others. Since most of these projects do not result in direct ODS phase-out, the indicators to measure their performance have to respond to the specific nature of these activities and, at the same time, render them accountable to management supervision. The deliverable should define these indicators and suggest how best they may be applied.

Deliverable 3:               Reports of a select number of evaluations, as proposed in the annual work plan of evaluations

Justification:                  UNEP/OzL.Pro/ExCom/21/36, para. 51(b), decision 21/36

Date of Completion:        12 months upon approval of the work programme

Brief description of the final product:

In recognition of the desire of the Executive Committee to start off with project evaluations, a draft work plan on evaluations is proposed. The work plan of evaluations is prepared on the basis of the various options, as proposed in the Consultant’s report (UNEP/OzL.Pro/ExCom/21/30).

The work plan is intended as a pilot scheme to carry out a select number of evaluations. The deliverables will be the reports on the evaluations carried out by consultants according to this work plan. Those reports should provide the Executive Committee with the opportunity to assess the usefulness of the evaluations both in terms of the modality and the substance of such an exercise for future benefits.

Deliverable 4:               Formats for project completion reports for investment and non-investment projects

Justification:                  UNEP/OzL.Pro/ExCom/21/36, para. 51(e), decision 21/36).

Date of Completion:        3 months upon approval of the work programme

Brief description of the final product:

Project completion reports will be the building blocks for project/programme evaluations. In view of the cost involved, project level evaluations will be carried out only on a very selective basis or with a very specific mandate, and the focus of evaluations will be at programmatic level (sector, substance, thematic, etc.). For the majority of projects, project completion reports would serve as the project evaluation reports. Therefore, the content of the project completion report will be crucial for management oversight.

The final product should include:

Key reporting criteria defined, in lieu of the baseline data;

Reporting formats for investment and non-investment projects;

Recording of lessons learned;

A success rating system;

A draft directive to be adopted by the Executive Committee to implement the formats.

Deliverable 5:               A proposal for standardized component on monitoring and evaluation in project proposals

Justification:                  UNEP/OzL.Pro/ExCom/21/36, para. 51(e), decision 21/36

Date of completion:        3 months upon approval of the work programme

Brief description of the final product:

The effectiveness of project monitoring and evaluation depends on availability and accuracy of baseline data (e.g., ODS consumption, existing equipment and their capacity, etc.), because the latter provides a point of reference for measuring success of the project in achieving its objectives, and providing management with lessons learned for future benefit. This baseline data should be included in the project proposal for future evaluation. Looking from the end point, what is going to be evaluated in the project completion report should be present in the project proposal as the baseline data.

The final deliverable should include:

Categories of baseline data;

Forms of presentation;

A schedule of monitoring and evaluation, with dates for completion and major milestones for monitoring.

Inputs needed

In order to produce the above deliverables, estimates of inputs needed are made on the basis of tasks to be accomplished under each deliverable. The input requirements are shown in person/months in two categories, internal and external, because, depending on the nature of the task and the expertise required, the inputs may be provided by the Secretariat[8] (internal) or by the consultant (external).

INPUTS NEEDED

B. WORK PLAN OF EVALUATIONS

OUTPUT 1: EVALUATION GUIDE FOR INVESTMENT PROJECTS AND NON-INVESTMENT PROJECTS

This guide will incorporate and build on guidelines and procedures already developed by the Implementing Agencies (UNIDO/DG/B.106: In-depth evaluation of technical co-operation projects, 1989; IBRD: Monitoring and Evaluation Guidelines for ODS Phase-out Investment Projects, 1995; UNDP: Policy and Procedures Manual, Monitoring, Evaluation and Reporting, 1987). This will help develop standardized procedures for these and subsequent evaluations specifically related to the requirements of the Fund.

The guide will include frameworks, key evaluation questions, sources of data, instruments and approaches for data collection, evaluation teams and divisions of responsibility of various stakeholders, as well as a sample table of contents for evaluation reports of investment projects. It would incorporate suggestions from the Executive Committee over time so that evaluations respond increasingly to concerns and questions of the committee.

Proposed Development Team: The proposed development team would include:

·         Coordinator contracted by the Secretariat and Technical Assistant

·         Representative of the Fund Secretariat

·         Representatives of the Implementing Agencies: IBRD, UNDP, UNEP, UNIDO

Methodology

The team would exchange views and documentation and contribute to the drafting and critique of proposed drafts. A workshop with the development team would be used to share materials, develop outlines and key questions. This would be followed by circulation of drafts and contributions by all members of the team. The draft guide would be added to and improved through the results of the evaluations to be conducted.

Timeframe:  June 1997 - September 1997 for draft guide

Cost:           US$ 43,500

Definition of the evaluations during 1997/98

As well as building evaluation capacity in the Secretariat, the work plan proposes conducting three evaluations in 1997/98. The selection criteria are to choose evaluations that:

·         Are representative of the range of projects and activities supported by the Fund (i.e. include investment and non-investment projects).

·         Focus on sectors where the Fund has been most involved

·         Include projects of all Implementing Agencies

·         Provide for examination of projects in all regions

·         Allocate effort to the most involved countries

·         Allocate effort to the individual projects with the greatest ODP reductions

·         Focus on investment projects that have been completed rather than those still in progress

·         Avoid the earliest projects, approved before all procedures were fully operational

Review of the data base for completed investment projects indicates that the most cost effective evaluation option would be to focus on the Foam and Refrigeration sectors. These comprise 56 and 34 completed projects respectively, with total funding of US$ 60.8 million. Furthermore, these two sectors contain 31% and 56% of targeted investment project approvals in 1997, so lessons that are learned will be of continuing potential value. The 90 projects to be included as the target population in the evaluation have proposed phase-out of 7,062 tonnes and an actual phase-out that is even larger.

The two proposed evaluations relate to interrelated sectors, so it is proposed to conduct them with a common evaluation team that will divide data collection and analysis among appropriate team members. This will enable cost-effective missions to many of the involved countries (The exact selection of countries can be made after approval and in keeping with the available travel budget.) The definition of the targeted projects and proposed methodology are described with each deliverable.

OUTPUT 2: EVALUATION OF COMPLETED PROJECTS IN THE FOAM SECTOR

This component of the evaluation work plan will involve the evaluation of a sample of completed investment projects in the foam sector. This evaluation will demonstrate the feasibility and cost effectiveness of sector investment project evaluations in contributing to the performance of the Fund in ODS phase-out and future decision-making by the Executive Committee.

Description of Completed Projects: Completed projects in the foam sector are characterized as follows:

Table 1: Completed foam projects by Implementing Agency

Table 2: Completed foam projects by region

Table 3: Partial listing of completed foam projects by sub-sector

The majority of these projects were approved by the Executive Committee at its 8th-18th meetings, so they represent projects that benefited from the early experience of the Fund.

Participating Article 5 countries: The following countries are the locations of completed investment projects in the foam sector (number of projects in brackets):

                        Egypt (10)                                                         China (9).

                        Indonesia                                                           India (4).

                        Malaysia (18)                                                     Philippines (2).

                        Thailand (4)                                                       Argentina (3).

                        Chile (2)                                                            Ecuador

      Mexico (2)  Uruguay

Focus of Evaluation

Possible Evaluation Questions: The following questions apply to evaluations of both sectors:

Project Design and Rationale

1.   What were the critical factors in the enabling environment that have affected project success? How have they contributed to or hindered project efficiency and effectiveness? Are there any contextual factors that should be a concern for future project approvals? Are there constraints in the enabling environment that the Fund should attempt to address?

2.   Did the design of various types of project change prior to implementation? Was the technology implemented different than the technology approved? Why and with what effects?

3.   Was the level of funding provided by the Fund understood by the enterprise and appropriate to the need and incremental cost requirements?

Effectiveness and Effects

4.   In general, how effective have the various types of investment projects been in achieving ODP targets and reducing ODS within the sector? Were there differences by region or Implementing Agency?

5.   Was the old technology successfully discontinued? For how long was the old technology in use after implementation of the project? How was the de-commissioned equipment disposed of?

6.   What have been the effects of the new technology on operating costs? On market demand?

Implementation Efficiency

7.   Given the recent findings on speed of implementation of investment projects (UNEP/OzL.Pro/ExCom/22/6 para. 4 indicates 20 - 37 months), what were the major implementation challenges and how were they overcome?

8.   Which aspects of investment projects in this sector (equipment, technical assistance, training) worked very well?

9.   How effective was transfer of technology in the various projects and regions?

Lessons Learned

10.  What lessons have been learned that may be useful in guiding future project preparation, approval, or implementation?

11.  What are the implications of the findings for additional and/or alternative information in future project proposals?

Proposed Evaluation Team

The proposed evaluation team would include:

·         Evaluation Team Coordinator contracted by the Secretariat

·         Two external technical experts in foam/refrigeration technology

·         Representative of the Fund Secretariat

·         One expert representative of UNDP

·         One expert representative of UNIDO

·         One expert representative of the World Bank

·         Representatives of each country in which evaluation activities take place would be involved for evaluation data collection and analysis related to that country

Methodology

The evaluation team will begin with a collective work planning phase that will develop standard data collection instruments and procedures well understood by all members of the evaluation team. Planning will include detailed allocation of responsibilities and scheduling of country missions. It is expected that the team will use a combination of methods including review of project proposals and reports, surveys and telephone interviews to project stakeholders, and country and on-site visits where the volume of projects warrants it. Since the proposed team would cover both foam and refrigeration sectors, actual data collection will relate to both sectors and may involve different team members visiting different countries. Everyone would contribute to data analysis and the coordinator would ensure that all aspects come together into an integrated report.

Timeframe:  September 1997 - May 1998

Cost:           US$ 129,000

OUTPUT 3: EVALUATION OF COMPLETED PROJECTS IN THE REFRIGERATION SECTOR

This component of the evaluation work plan will involve the evaluation of a sample of completed investment projects in the refrigeration sector and will be conducted in tandem with the evaluation within the foam sector.

Description of Completed Projects: Completed projects in the refrigeration sector are characterized as follows:

Table 4: Completed refrigeration projects by Implementing Agency

Table 5: Completed refrigeration projects by region

The majority of these projects were approved by the Executive Committee at its 8th-18th meetings.

Participating Article 5 countries: The following countries are the locations of completed investment projects in this sector (number of projects in brackets):

                        Algeria                                                              Cameroon

                        Egypt (3)                                                           Malaysia

                        Philippines (3)                                                    Syria (2).

                        Thailand (7)                                                       Vietnam

                        Romania                                                            Turkey (3).

                        Brazil                                                                Chile (3).

                        Guatemala                                                         Mexico (4).

                        Venezuela (2)                                                    

Possible Evaluation Questions:              See output 2

Proposed Evaluation Team:                   See output 2

Methodology:                                      See output 2

Timeframe: September 1997 - May 1998

Cost:           US$ 129,000

OUTPUT 4: NON-INVESTMENT PROJECT EVALUATION

To complement the evaluation of investment projects in the two noted sectors, the work plan includes a collaborative evaluation of some of the major activities of UNEP’s OzonAction Programme. It will enable UNEP to develop useful qualitative performance indicators and evaluate the extent to which its strategy in support of crucial enabling environments is being achieved. Because most country programmes have been approved, it is not considered cost effective to evaluate this aspect of the programme. The optimal configuration would be determined in collaboration with UNEP, but is expected to focus on information exchange, training, and networking.

Description of Ongoing Projects

Since inception of the Fund, UNEP has received $11.4 million for technical co-operation. Decision 21/14 outlined the dimensions of recurring programme activities and capped information-exchange activities (US$ 1.05 million) and networking (US$ 1.1 million). The proposed evaluation will assist future decision-making by assessing the extent to which previously funded activities have been cost effective and impacted the enabling environment.

Focus of Evaluation

Possible Evaluation Questions

UNEP/OzL.Pro/ExCom/22/7/33 proposes some of the qualitative performance indicators for UNEP’s recurring core clearinghouse, networking and information dissemination activities. The evaluation will consider these and add to them in an attempt to develop a more complete understanding of UNEP’s effectiveness in affecting the enabling environment for the work of the Fund. This will contribute to the aspect of the evaluation work programme that proposes to develop indicators for non-investment projects.

Some of the specific questions that could be addressed are:

Design and Rationale

1.   To what extent are UNEP’s activities suitably targeted to reach people and institutions with a need for such support? How has the configuration of activities evolved?

2.   Does UNEP include suitable monitoring and evaluation of activities that enable programme activities to benefit from participant feedback? How might monitoring and evaluation systems be improved?

Effectiveness and Effects

3.   To what extent have UNEP’s information exchange activities been relevant to ODS phase-out in Article 5 countries?

4.   To what extent was the training effective? Is it being applied on the job? If not, what are the constraints? How could training be improved?

5.   What have been the effects of networking, training, information exchange activities on initiation of awareness-raising or other activities supported by countries?

6.   What policies have been initiated by countries as a result of UNEP’s programme?

7.   What improvements in data reporting and enacted legislation and policies for networking countries can be attributed to UNEP’s programme?

Efficiency

8.   Are UNEP’s activities planned and implemented in the most cost effective way? How could cost-effectiveness be improved?

9.   Is the allocation of resources optimal given related evaluation findings on the various aspects of programming? Should UNEP re-allocate resources for greater impact?

Lessons Learned

10.  How can UNEP’s programme better serve the needs of involved countries?

Proposed Evaluation Team

·         Coordinator of Evaluation Team

·         Representative of the Secretariat

·         External expert on organizational and programme self-assessment

·         UNEP Representatives

·         Country representatives will be involved in various aspects linked to ongoing programme activities and missions of the investment project evaluation team

Methodology

The recommended methodology would incorporate contemporary empowerment approaches to organizational evaluation by combining participatory evaluation with some independent data collection. The process would engage UNEP and its clients in collaborative processes to refine the key questions, develop and refine indicators, collect and analyze relevant data in a collaborative way, and develop a report coordinated by suitable external experts. Ideally, this evaluation will capitalize on the data collection activities of the investment project evaluation team which would collect independent data in the various countries visited. As well, it is anticipated that a questionnaire survey will be distributed to users and potential users of UNEP’s programme activities. In this way the evaluation report will have objectivity while leading directly to programming improvements at UNEP.

Timeframe: June 1997 - May 1998

Cost:           US$ 78,500

C. SUMMARY BUDGET

(UNEP/OzL.Pro/ExCom/22/79/Rev.1, Decision 22/19, para. 34).

(Supporting document: UNEP/OzL.Pro/ExCom/22/SC-MEF/2, Corr.1, Add.1).