PART TWO
CHAPTER 24. EVALUATION OF A SUBREGIONAL PROGRAMME

Chapter 24 contains the following:

1. questions and answers to guide the formulation of the TOR for a subregional programme evaluation;
   
   
2. a suggested outline for the TOR for the evaluation of the subregional programme; and
   
   
3. a sample outline for the corresponding evaluation report.

Information from the Fifth ASEAN/UNDP Sub-Programme (ASP-5) (presented in italics) is used for illustration purposes.

DEVELOPING THE TOR FOR THE EVALUATION OF A SUBREGIONAL PROGRAMME
What background information is essential to formulate the TOR for the evaluation?

• Period covered by the programme
  
  
• Main objectives of the programme
  
  
• Areas of thematic focus
  
  
• Main components of the programme

Planning for ASP-5 began with a series of joint studies and meetings in 1990-1991. This process was interrupted by decisions taken by the heads of Government of the Association of South-East Asian Nations (ASEAN) at the Singapore Summit in January 1992 to restructure the ASEAN Secretariat in order to strengthen its role in ASEAN cooperation activities.

With the restructuring of the Secretariat and the appointment of a new ASEAN Secretary General in 1993, a programme framework for ASP-5 was formulated, emphasizing mutual priorities in economic reform, human development, environmental protection and capacity development. ASP-5 was approved in July 1994 and became operational six months later. Executed by the ASEAN Secretariat, it has a three-year time frame.

Based on selected common goals and priorities, ASP-5 has five programme components:

• capacity-building, which seeks to improve the institutional arrangements for ASEAN cooperation and to strengthen the capacity of the Secretariat to plan and manage ASEAN cooperation programmes effectively;
• human development, which is expected to strengthen ASEAN capacity to initiate and sustain programmes in social development, control of drug abuse, AIDS prevention, the advancement of women, and the establishment of an ASEAN university network;
• liberalization of trade and investment, which supports the development of the policy and institutional framework for attaining the objectives of the ASEAN Free Trade Area (AFTA);
• Trade and environment, which aims to increase the understanding among government policy-makers and the business community of the complex relationships between trade and environment in order to lay the foundation for reconciling trade and environmental policies;
• Science and technology, which seeks to strengthen ASEAN cooperation in science, technology, research, information-sharing and technology transfer.

Why is an evaluation of ASP-5 needed?

A joint mid-term review of ASP-5 was undertaken in mid-1996. A joint issues paper (JIP) was prepared and presented at the first Joint Management Committee Meeting of ASEAN and UNDP in Kuala Lumpur in October 1996. The findings and recommendations articulated in the JIP were endorsed. One of the major recommendations was to evaluate ASP-5, considered to be one of the success stories among all technical cooperation programmes in ASEAN countries. The evaluation will help to determine the validity of this conclusion. It will systematically take stock of experience (both negative and positive) and draw lessons from it.

Furthermore, preparations for programming ASP-6 are about to start. An external evaluation of ASP-5 is needed to: (a) examine the use of the UNDP investment in the development process of the ASEAN member countries, both individually and collectively; (b) assess the value added by UNDP interventions to the development efforts of ASEAN member countries in terms of results and management; and (c) examine external factors, such as the emerging political, economic and social environment of the ASEAN region, which might influence the achievement of desired results.

In addition to comparing the outcomes with the indicators established to measure the relevance, performance and success of ASP-5, the evaluation will provide a further opportunity to advocate better management practices and technical approaches.

Who are the main stakeholders in the evaluation?

The Governments of the ASEAN member countries, the ASEAN Secretariat, and UNDP are the main stakeholders.

What issues will the evaluation address?

The evaluation will focus on (a) assessing the continued relevance of ASP-5 to the development priorities of the member countries that were confirmed through the last mid-term review and endorsed at the first ASEAN-UNDP dialogue; (b) the comparative advantage of UNDP vis-à-vis other donors; (c) the flexibility of the programme in responding to emerging needs; (d) the ownership of the design and implementation process and the various outputs of ASP-5; and (e) the capacity of the ASEAN countries to manage change within the context of the rapidly developing ASEAN region. The role of specific networking arrangements that have been established within the framework of ASP-5 will be evaluated in terms of their contribution to sustainability.

A World Bank report stated that ASEAN is viewed as a success story after 30 years of existence, giving due recognition to its political stability and economic growth. Yet within the current plan for the next five to ten years, the social dimension of the development of ASEAN member countries deserves more attention. Unless human development is made the top priority of those countries, the achievements will be in jeopardy, especially when the three new least developed countries (LDCs) (Cambodia, Lao People's Democratic Republic and Myanmar) are integrated into ASEAN in the next few years. In this context, the evaluation will recommend ways to put a human face on present successes and to give greater priority to gender issues.

The evaluation will also assess the capacity of the ASEAN Secretariat as the executing agency, including its monitoring and evaluation capacity. It will examine the effectiveness of United Nations agencies and civil society organizations as implementing agents.

In addition, the linkages between and among the subprogramme components of ASP-5 and how they complement related initiatives at the national level of the ASEAN member countries will be evaluated.

With a view to programming for ASP-6, which covers the period 1997 to 2001, the evaluation team will address the question of whether the design and implementation processes were owned by the Governments of ASEAN member countries. It will identify improvements that can be incorporated into the design process of ASP-6. It will also determine the additional costs that would be incurred if the processes were more participatory.

What products are expected from the evaluation?

The evaluation mission will determine the extent to which ASP-5 has met its objectives and draw lessons (both conceptual and practical) based on the implementation of the subprogramme. The evaluators will also make recommendations on the best way to conclude ASP-5 and on the design of the future programme (ASP-6), i.e., its main thrusts, execution modality, implementation and management arrangements.

What is the evaluation methodology?

A desk review of relevant documents will be undertaken. Questionnaires, complemented by selected field visits and interviews, will be used to obtain the views of key persons in the ASEAN Secretariat and the Governments of ASEAN member countries.

The UNDP Resident Representatives in the ASEAN member countries will provide their assessments of the linkages between ASP-5 and the country-level interventions.

Who are the members of the evaluation team?

An independent team of consultants will undertake the evaluation. The team will be composed of an economist with experience pertaining to ASEAN, a sociologist and a specialist in development management with experience in capacity-building. At least one of the consultants will have worked extensively with the United Nations system and be thoroughly familiar with gender issues.

Who is responsible for organizing and managing the evaluation?

As part of the decentralization process, the UNDP Resident Representative in Jakarta will bear primary responsibility for organizing and managing the evaluation. However, consultations will take place with UNDP resident representatives in the other ASEAN member countries, UNDP headquarters (Regional Bureau for Asia and the Pacific and OESP), and the ASEAN Secretariat.

UNDP country offices in the ASEAN member countries and the ASEAN Secretariat will designate their respective focal points to facilitate the work of the evaluators.

How much time will the evaluation require?

Given the coverage of the subregional programme and the number of stakeholders involved, the evaluation will require about 15 weeks.

What are the resource requirements for the evaluation?*

National consultants

International consultants - 3

Travel for consultants visiting ASEAN member countries

*NOTE: The UNDP country office in Indonesia will prepare a detailed budget to

cover these and other costs related to the evaluation. However, the budget itself and the source(s) of funding are not to be included in the TOR.

What are the funding arrangements for the evaluation?

The cost of the evaluation will be charged against the budget of ASP-5.

TOR FOR THE EVALUATION OF ASP-5: SUGGESTED OUTLINE

1. Introduction
   

Brief description of ASP-5

• Main objectives
• Present status
• Highlights of achievements

2. Objectives of the Evaluation
   • Brief description of how the need for the evaluation was identified
   • Main stakeholders of the evaluation
   • Objectives of the evaluation

3. Scope of the Evaluation
   
   
4. Issues to be Addressed
   • Relevance
     • Level of ownership in design and implementation processes
     • Relationship of the programme to the development priorities of the ASEAN member countries
     • Use of the UNDP comparative advantage
   • Performance
     • Capacity of the ASEAN Secretariat as executing agency
     • Effectiveness of the implementing agency
     • Adequacy of monitoring and evaluation mechanisms
     • Achievement of the immediate objectives of ASP-5
   • Success
     • Contribution of ASP-5 to the development of the ASEAN region and the individual ASEAN member countries
     • Sustainability of results
     • Capacities developed at the national and subregional levels

5. Products Expected from the Evaluation
   • Strategies for concluding ASP-5
   • Recommendations for formulating ASP-6
   • Lessons learned concerning best and worst practices

6. Methodology
   • Desk review
   • Questionnaires
   • Interviews
   • Field visits
   • Participation of stakeholders

7. Evaluation Team
   • Composition and profile of independent team of consultants
   • Responsibilities of the evaluators

8. Implementation Arrangements
   • Responsibilities and mechanisms for managing the evaluation
   • Timetable
     • Desk review
     • Briefings of evaluators
     • Visits to the field, interviews and questionnaires
     • Debriefings
     • Preparation of report
   • Resources required

ASP-5 EVALUATION REPORT: SAMPLE OUTLINE
Executive Summary

• Brief description of ASP-5
  
  
• Context and purpose of the evaluation
  
  
• Main conclusions, recommendations and lessons learned

Introduction

• Purpose of the evaluation
  
  
• Key issues addressed
  
  
• Methodology of the evaluation
  
  
• Structure of the evaluation report

ASEAN Subregion and ASP-5

• Social, economic and political situation of the subregion
  
  
• Role of ASEAN
  
  
• ASP-5 and the development of the ASEAN Subregion
  • Main objectives of ASP-5
  • Major components of ASP-5
  • Present status of ASP-5

Findings and Conclusions

• ASP-5 formulation
  • Process or approach used
  • Identification of priority areas: their relevance at the national, subregional and regional levels
  • UNDP comparative advantage
  • Linkages among the programme components
  • Linkages with other development initiatives (in the region, with other regions)
  • Indicators
  • Management arrangements, including monitoring and evaluation

• Implementation
  • Execution and implementation modalities
  • Management by the UNDP country office
  • Coordination and operational issues
  • Adequacy of monitoring mechanisms

• Results
  • Immediate results at the national and subregional levels
  • Sustainability
  • Contribution to capacity-building at the national and subregional levels

Recommendations

• Concluding ASP-5
  
  
• Formulating ASP-6
  • Main programme thrusts
  • Implementation and management arrangements

Lessons Learned

• Best and worst practices in addressing issues relating to relevance, performance and success

Annexes

• TOR
  
  
• Itinerary
  
  
• List of persons interviewed
  
  
• Summary of field visits
  
  
• List of documents reviewed
  
  
• Questionnaire used and summary of results
  
  
• Other relevant material