Evaluation

Introduction

1.       This paper presents the policy of UNDP for evaluation. The purpose of the policy is to establish a common institutional basis for the UNDP evaluation function. The policy seeks to increase transparency, coherence and efficiency in generating and using evaluative knowledge for organizational learning and effective management for results, and to support accountability. The policy applies to UNDP and its associated funds and programmes – the United Nations Development Fund for Women (UNIFEM), the United Nations Volunteers (UNV) programme, and the United Nations Capital Development Fund (UNCDF).  The policy also applies to all UNDP-managed programmes, irrespective of funding source.

2.       The policy responds to General Assembly resolution 59/250 of 2004 which requires the systematic evaluation of United Nations system operational activities by assessing their impact on poverty eradication, economic growth and sustainable development of programme countries. It further mandates the United Nations system to promote national ownership and capacity development and to make system-wide progress in collaboration in evaluation. The policy draws from and is aligned with the norms and standards for evaluation in the United Nations system approved by the United Nations Evaluation Group (UNEG) in April 2005.

3.       UNDP subscribes to the overarching United Nations goal of reducing extreme poverty and hunger by half by 2015. It supports the efforts of partner countries to achieve the Millennium Development Goals (MDGs) and other development results primarily through its role as a global development network, advocating for change and connecting countries to knowledge, experience and resources to help people build better lives. While emphasizing national ownership and capacity development, UNDP helps countries develop and share solutions to challenges in key strategic development areas. As resident coordinator of United Nations operational activities at the country level, UNDP promotes effective coordination of United Nations system support to national priorities.

4.       Evaluation in UNDP will provide an objective assessment of contributions to development results, through assessing its programmes and operations, including advocacy, advisory services, knowledge networks, technical assistance, coordination and partnerships. Evaluation will address what works and why, as well as what does not work and unintended outcomes. This will support accountability, inform decision-making and allow UNDP to better manage for development results.

5.       Evaluation will improve learning and knowledge for development among UNDP and its partners. Engagement of all key stakeholders will enhance capacity for evaluation as well as its utility. The development of knowledge-management systems, learning groups and communities of practice will increase access to knowledge and enhance knowledge-sharing, collaboration and innovation.

6.         This policy note establishes the guiding principles and norms; explains key evaluation concepts; outlines the main organizational roles and responsibilities; defines the types of evaluation covered; and identifies the key elements of a system for learning and knowledge management. It also outlines the capacity and resource requirements to enhance excellence in the development of an evaluation culture and a learning organization

I.          Guiding principles

7.       The following are the key principles.

a) Human development and human rights. Evaluation is guided by the people-centred approach of UNDP to development, which enhances capabilities, choices and rights for all men and women. Evaluation abides by universally shared values of equity, justice, gender equality and respect for diversity.

(b) United Nations system coordination and global partnership. Evaluation draws on and contributes to collaboration within the UN system to improve effectiveness and reduce transaction costs for development cooperation. UNDP advocates strengthened professional collaboration under the aegis of UNEG and country-level coordination in evaluation under the Resident Coordinator system. With the increasing engagement of UNDP in global initiatives and partnership programmes with other donors, non-governmental organizations and civil society, the conduct of joint evaluations enhances global partnership.

(c) National ownership. Evaluation should be guided by national priorities and concerns. It should be inclusive and take into account diverse national interests and values. Evaluation should strengthen partnerships with governments and key stakeholders. It should build the capacity of national institutions to implement, monitor and evaluate.

(d) Managing for results. Evaluation supports UNDP to manage for results by assessing the extent to which UNDP’s processes, products and services contribute effectively to development results affecting people’s lives. Through this, evaluation highlights the need for quality in the design of programmes so that results are clear, measurable and can be monitored and evaluated. Through the generation of evidence, evaluation enables more informed management and decision-making for strategic planning and programming.

II.        Norms

8.       UNDP evaluations will operate under norms based on the UNEG norms for evaluation in the United Nations system, as follows:

(a) Independence. The evaluation function should be structurally independent from the operational management and decision-making functions in the organization so that it is free from undue influence, more objective, and has full authority to submit reports directly to appropriate levels of decision-making. Management must not impose restrictions on the scope, content, comments and recommendations of evaluation reports. To avoid conflict of interest, evaluators must not be directly involved in policy-setting, design, implementation or management of the subject of the evaluation either before, during or after the evaluation.

(b) Intentionality. The rationale for an evaluation and the decisions to be based on it should be clear from the outset. The scope, design and plan of the evaluation should generate relevant, timely products that meet the needs of intended users.

(c) Transparency. Meaningful consultation with stakeholders is essential for the credibility and utility of the evaluation. Full information on evaluation design and methodology should be shared throughout the process to build confidence in the findings and understanding of their limitations in decision-making.

(d) Ethics. Evaluation should not reflect personal or sectoral interests. Evaluators must have professional integrity and respect the rights of institutions and individuals to provide information in confidence and to verify statements attributed to them. Evaluations must be sensitive to the beliefs and customs of local social and cultural environments and must be conducted legally and with due regard to the welfare of those involved in the evaluation, as well as those affected by its findings. In line with the Universal Declaration of Human Rights, evaluators must be sensitive to and address issues of discrimination and gender inequality.

(e) Impartiality. Removing bias and maximizing objectivity are critical for the credibility of the evaluation and its contribution to knowledge. Prerequisites for impartiality are: independence from management; objective design; valid measurement and analysis; and the rigorous use of appropriate benchmarks agreed upon beforehand by key stakeholders. In addition to being impartial, evaluation teams should include relevant expertise and be balanced in their gender and regional composition.

(f) Quality. All evaluations should meet minimum quality standards defined by the Evaluation Office. The key questions and areas for investigation should be clear, coherent and realistic. The plan for evaluation should be practical and cost effective. To ensure that the information generated is accurate and reliable, evaluation design, data collection and analysis should reflect professional standards, with due regard for any special circumstances or limitations reflecting the context of the evaluation. To ensure this, the professionalism of evaluators and their intellectual integrity in applying standard evaluation methods is critical. Evaluation findings and recommendations should be presented in a manner that will be readily understood by target audiences.

(g) Timeliness. Evaluations must be designed and completed in a timely fashion so as to address the specific purpose and objectives for which they were commissioned and ensure the usefulness of the findings and recommendations. Balancing technical and time requirements with practical realities while providing valid, reliable information is central to ensuring that the evaluation function supports management for results.

(h) Utility. Evaluation is a management discipline that seeks to provide information to be used for evidence-based decision-making. To enhance the usefulness of the findings and recommendations, key stakeholders should be engaged in various ways in the conduct of the evaluation. The scope, design and plan of the evaluation should generate relevant, timely products that meet the needs of intended users. The interpretation of findings should be grounded in the realities of the country and programme context, and the recommendations made should be practical and realistic.

III.       Key concepts

9.       Evaluation. Evaluation is judgment made of the relevance, appropriateness, effectiveness, efficiency, impact and sustainability of development efforts, based on agreed criteria and benchmarks among key partners and stakeholders. It involves a rigorous, systematic and objective process in the design, analysis and interpretation of information to answer specific questions. It provides assessments of what works and why, highlights intended and unintended results, and provides strategic lessons to guide decision-makers and inform stakeholders.

10.       Monitoring. Monitoring is distinct from evaluation. It is a continuous function providing managers and key stakeholders with regular feedback on the consistency

or discrepancy between planned and actual activities and programme performance and on the internal and external factors affecting results. Monitoring provides an early indication of the likelihood that expected results will be attained. It provides an opportunity to validate the programme theory and logic and to make necessary changes in programme activities and approaches. Information from systematic monitoring serves as a critical input to evaluation.

Definition

11.     Development results.

(a)      Output: Tangible product (including services) of an intervention that is directly attributable to the initiative. Outputs relate to the completion (rather than the conduct) of activities and are the type of results over which managers have most influence. An example of an output for a project for judicial reform is the number of judges trained and qualified.

(b) Outcome: Actual or intended changes in development conditions that an intervention(s) seeks to support. The contribution of several partners is usually required to achieve an outcome. Using the same example an outcome is the improvement in the judicial process as evidenced by a reduction in the backlog of cases.

(c) Impact: Actual or intended changes in human development as measured by people’s well-being. In this example, an impact is that more people have access to justice and are better able to exercise their rights.

12.     Attribution: The precise causal link to changes in development results flowing from an individual intervention. For example, the number of judges trained could be directly attributed to a specific intervention.

13.     Contribution: The changes in development results that can be credibly linked to an intervention. Contribution implies a logical cause and effect relationship which points to the meaningful input of an intervention to the development result(s). For instance, improvement in the judicial process could result from the interventions of several actors working to train judges, improve administrative processes or bring about changes in legislative policy.

14.     Organizational effectiveness: The more direct, accountable and attributable measures of performance over which the organization has relatively more control or manageable interests.

15.     Development effectiveness: The extent to which the intended development goals of a country are achieved through the agency of the government, civil society and development partners. Evaluation assesses the effectiveness of the partners’ contribution in enhancing the factors and conditions that enable countries to achieve their development goals.

IV.       Roles and responsibilities

16.     The roles and responsibilities of the key constituents of the UNDP evaluation system are identified below.

17.     The Executive Board of UNDP/UNFPA is the custodian of the evaluation policy. The Executive Board:

(a) approves the evaluation policy;

(b) ensures the independence of the evaluation function;

(c) approves the annual work programme for the centralised evaluation function;

(d) requires management response and follow-up to evaluation by UNDP; and

(e) uses evaluation and reports on compliance with evaluation policy for accountability; and

 (f) draws on the findings and recommendations of evaluation for oversight and approval of corporate policy, strategy and programmes.

18. The Evaluation Office is the custodian of the evaluation function. The Evaluation Office will:

(a) Governance and accountability
           (i) prepare and periodically review and update UNDP’s policy for evaluation;
           (ii) submit its annual plan to the Executive Board;
           (iii) report annually to the Executive Board on the function, findings and recommendations of evaluations, on compliance, quality assurance, and follow-up to evaluations conducted by UNDP and its associated funds and programmes;
           (iv) maintain a system to record management responses to all evaluations; and
           (v) alert senior management to emerging evaluation-related issues of corporate significance;

(b) Conduct of evaluations
           (i) develop an agenda for independent evaluations, based on consultations with the Executive Board, senior management, the associated funds and programmes and other stakeholders, and in response to emerging issues that the Evaluation Office may identify;
           (ii) conduct strategic and thematic evaluations, programme evaluations such as the Assessment of Development Results (ADRs) at the country level, evaluations of global, regional, and South-South programmes, and other evaluations as required; and
           (iii) ensure that the evaluations provide strategic and representative coverage of UNDP programmes and results, and that mandatory evaluations are carried out;

(c) Quality assurance
           (i) set evaluation standards for planning, conducting and using evaluations, developing and disseminating methodology and establishing the institutional mechanisms for applying the standards;
           (ii) support the setting of frameworks and standards for monitoring in the context of the UNDP results-based management system to facilitate the evaluation of programmes and activities; and
           (iii) assure mandatory decentralized evaluations and support quality assurance of the evaluations conducted by the associated funds and programmes;

(d) Capacity development
           (i) build knowledge of good practice standards and approaches for evaluation management in UNDP; and
-         (ii) strengthen programme country evaluation capacity and involvement in evaluations through country-led evaluations, joint evaluations and use of partner country professional resources;

(e) Knowledge management 
           (i) maintain a publicly accessible repository of evaluations;
           (ii) distil evaluation findings and lessons for dissemination in appropriate formats for targeted audiences; and
           (iii) support the development of learning groups and communities of practice in evaluation by establishing and supporting knowledge networks;

(f) United Nations reform
           (i) ensure that evaluation in UNDP contributes to and remains consistent with UN policy and reforms including supporting and participating in joint evaluations;
           (ii) support the harmonization of the evaluation function of UNDP and its associated funds and programmes;
           (iii) contribute evaluative evidence to system-wide evaluations, such as the Triennial Comprehensive Policy Review; and
           (iv) use the experience of UNDP and its partners, including through working with UNEG, to advance the science, practice, quality and usefulness of evaluation.

19. The Administrator of UNDP is accountable for UNDP results, and:

(a) ensures compliance with the evaluation policy as integral to effective accountability across the organization;

(b)  safeguards the integrity of the evaluation function and its independence from operational management;

(c) appoints the Director of the Evaluation Office, in consultation with the Executive Board, and ensures that there is no conflict of interest in employment, including limiting the term of appointment to four years, renewable once, and barring reentry into the organization;

(d) safeguards the independence of the Evaluation Office by ensuring that the Director has the final say on the contents of all evaluation reports issued by the Evaluation Office;
(e) provides sufficient resources and capacity for evaluation in the organization;

(f) ensures that UNDP prepares a management response to evaluations that are submitted to the Executive Board; and

(g) ensures that senior management responds to and utilizes evaluation in their operational, strategic, policy and oversight functions and that appropriate follow-up to the findings and recommendations of evaluation is taken by the relevant units.

20. The senior management of country offices, regional bureaux, practice and policy bureaux, and the associated funds and programmes will:

(a) ensure the evaluability of programmes by identifying clear results, developing measurable indicators, and establishing performance targets and baseline information;

(b) in collaboration with national stakeholders and partners, ensure the effective monitoring of implementation and performance of programmes to generate relevant, timely information for management for results and evaluation;

(c) identify, with partner governments and key stakeholders, priority areas for evaluation when preparing the programme, and designing and implementing a strategic evaluation plan;

(d) establish an appropriate institutional arrangement to manage evaluation;

(e) ensure adequate resources for evaluation;

(f) safeguard the independence of the evaluation process and product;

(g)      ensure the conduct of mandatory evaluations in line with established quality standards;

(h) promote joint evaluation work with the United Nations system and other partners and, in the case of country offices, contribute to the UNDAF evaluation;

(i) prepare management responses to all evaluations, and ensure and track appropriate, timely implementation of the agreed evaluation recommendations;

(j) draw on evaluation findings to improve the quality of programmes, guide strategic decision-making on future programming and positioning, and share knowledge on development experience;

(k) ensure the transparency of, and public access to, all evaluation reports; and

(l) Directors of regional bureaux, in their exercise of line oversight, ensure compliance by country offices with mandatory requirements of the evaluation policy, and support and guide country office capacity in evaluation, including establishing regional expertise and evaluation support systems.

21. Evaluation units of the associated funds and programmes are the custodians of the evaluation function in their organizations, and they will, respectively, for their fund or programme:

(a)  periodically review and revise, as necessary, the evaluation policy;

(b) contribute to the development by the Evaluation Office of common evaluation quality standards and guidelines;

(c) support the elaboration of well-defined results frameworks to facilitate the evaluation of programmes and activities;

(d) submit to their senior management a biennial plan and budget for the evaluation unit;

(e) develop, in consultation with key stakeholders, an annual agenda of evaluations to be conducted;

(f)  manage and conduct evaluations;

(g)  ensure, whenever possible, joint evaluation work with the United Nations system and other partners;

(h) quality-assure mandatory evaluations outsourced or managed by programme staff;

(i) ensure the maintenance of a publicly accessible repository of evaluations;

(j)  ensure the dissemination of evaluation findings and lessons in appropriate formats for targeted audiences, and promote their consideration in decision-making and for learning;

(k) track management response and follow-up to agreed evaluation recommendations;      

(l) alert their senior management to evaluation-related issues of corporate significance;

(m)  provide input to the annual report on evaluation to the Executive Board;

(n)  contribute to developing evaluation capacity

(o)  ensure consistency with United Nations policy and reforms, and contribute to improving evaluation collaboration, quality and usefulness, including through participation in UNEG.

V.         Types of evaluations conducted by UNDP

22. The evaluations conducted by UNDP fall into two categories: independent evaluations conducted by the Evaluation Office, and decentralized evaluations managed by country offices, regional bureaux and practice and policy bureaux, and conducted by external experts. Together they represent a coherent system of evaluation that provides the necessary coverage to manage for results and to support organizational accountability.

Evaluations conducted by the Evaluation Office

23. The Evaluation Office is mandated to conduct evaluations for corporate accountability, strategic planning, and the development of information for global knowledge use.

(a) Strategic evaluations assess UNDP performance in areas that are critical to ensuring sustained contribution to development results in the context of emerging development issues and changing priorities at the global and regional levels. To this end, strategic evaluations may cover, for example, UNDP policies, practice areas, partnerships, programmatic approaches, cooperation modalities, or business models. The Evaluation Office will also engage in evaluations conducted jointly with other United Nations organizations to assess system-wide performance.

(b)  Programmatic evaluations

(i) Global, regional and South-South programme evaluations assess the performance and intended and achieved results of those programmes. They are intended to reinforce the substantive accountability of UNDP to the Executive Board, and will be timed to contribute to the preparation and approval of the next programme.

(ii) Assessments of Development Results (ADRs) assess the attainment of intended and achieved results as well as UNDP contributions to development results at the country level. Their scope will include, but not necessarily be confined to, UNDP responsiveness and alignment to country challenges and priorities; strategic positioning; use of comparative advantage; and engagement with partners. The number and selection of countries, and the timing of these evaluations, will be determined to ensure coverage and to allow findings and recommendations to feed into the preparation of the subsequent programme. Wherever possible, these evaluations will be conducted in conjunction with other United Nations organizations.

Decentralized evaluations

24. Country offices, regional bureaux, and practice and policy bureaux commission evaluations in the programmatic frameworks for which they are responsible. All programme areas or project clusters should be subject to evaluation. The selection of what is evaluated and the number of evaluations are decided with stakeholders at the outset of the programme cycle. The focus is on information for programme improvement and the development of new programmatic frameworks. The information also provides the basis for strategic and programmatic evaluations conducted by the Evaluation Office as described above. The key decentralized evaluations are the following:

25. Outcome evaluations address the short-term, medium-term and long-term results of a programme or cluster of related UNDP projects. They include an assessment of the effectiveness, efficiency, sustainability and relevance of the programme against their own objectives, their combined contribution, and the contribution of external factors and actors. Outcome evaluations also examine non-intended effects of the programme or projects. Rather than being ad hoc, the selection of the programme or project cluster to be evaluated should be guided by strategic decisions made by the programme unit, in line with the evaluation plan. This decision should be informed by agreements with national government and key stakeholders and partnership requirements, with attention to utility and linkage with strategic and programmatic evaluations.

26. Project evaluations assess the efficiency and effectiveness of a project in achieving its intended results. They also assess the relevance and sustainability of outputs as contributions to medium-term and longer-term outcomes. Projects can be evaluated during the time of implementation, at the end of implementation (terminal evaluation), or after a period of time after the project has ended (ex-post evaluation). Project evaluation can be invaluable for managing for results, and serves to reinforce the accountability of project managers. Additionally, project evaluation provides a basis for the evaluation of outcomes and programmes, as well as for strategic and programmatic evaluations and ADRs, and for distilling lessons from experience for learning and sharing knowledge. In UNDP, project evaluations are mandatory when required by a partnership protocol, such as with the Global Environment Facility.

VI.       Types of evaluations conducted by the associated funds
            and programmes

27.  Strategic and thematic evaluations provide a basis for developing forward-looking strategies for organizational effectiveness and core thematic programme areas. They involve reviews of past experience to identify strengths and gaps in the approach and results relating to a particular aspect of the organization and its approach, or to a thematic programme area. They may also include a review of the global trends and partner practices in the organizational or thematic area to determine whether approaches and interventions are relevant. Strategic and thematicevaluations are undertaken by UNCDF, UNIFEM and UNV.

28.  Country reviews assess the comprehensive coverage of the programme contribution to projects and support provided by the organization; they are the only mechanism that measures the contribution of non-project-related volunteer assignment. Country reviews are undertaken by UNV, and are typically conducted in countries where there has been a significant UNV presence over a period of time.

29.  Outcome evaluations assess projects or clusters of related programmes in terms of their contribution to the achievement of target outcomes. Outcome evaluations are undertaken by UNCDF, UNIFEM and UNV.

30.   Programme and projects evaluations focus on evaluating performance mid-way through and at the end of the programme cycle. They assess the specific contributions, efficiency, effectiveness, relevance and sustainability of interventions, as well as strategic positioning and partnerships. Programme and projects evaluations are undertaken by UNCDF, UNIFEM and UNV.

VII.     Mandatory evaluations

31.     The table below presents mandatory evaluations and the responsible units.