Development and the HIV Epidemic: A forward-looking evaluation of the approach of the UNDP HIV and Development Programme 2. THE EVALUATION PROCESS
2.1 The nature and purpose of the evaluation The purpose of the evaluation was to review the use by the HIV and Development Programme of Special Programme Resources (SPR) at global, regional and national levels during the fifth UNDP programming cycle (1992-1996), and to consider what UNDP should be doing during the sixth programming cycle. The evaluation was intended to be forward-looking, starting with consideration of what has been learnt through the HDP to date, but aiming to generate further and deeper understanding of the practices of sustainable human development as applied to the HIV epidemic, its causes and consequences. Evaluation is a process inextricably linked to a sense of purpose, and there is no such thing as an all-purpose evaluation (Weiss, 1972). Evaluation processes should therefore relate to the needs and interests of the users of the evaluation. In this case, the people for whom the outcome may be of value and interest will not be limited to those defined in many evaluation texts as "key stakeholders". Along with UNDP, the users of this evaluation will include a range of people and organisations interested in learning more about how sustainable human development may work in their own and others' contexts: the partners, direct and indirect, of the HDP initiatives, staff of national HIV/AIDS programmes, UN agencies, NGOs, communities, consultants, academics and others affected by the epidemic. The evaluation process was therefore designed to elucidate lessons learnt through the approach of the HDP in ways which would also be of value to these users of the evaluation. The evaluation was also designed to fit within a framework outlined in a UNDP discussion paper on guiding principles for evaluation (Benbouali, 1995). In relation to programmes of this nature, the paper notes that, "The development objectives are finally the results of strategic effects of many programmes/projects and efforts" (Benbouali, 1995, p.2). A suggested means of enhancing understanding of such strategic effects is to concentrate on issues of relevance and sustainability. This makes such evaluation distinct from auditing and assessment, which focus on managerial and financial issues, and on measures of programme activities and performance as pre-determined in initial decisions about programme objectives. An emphasis on strategic effects is also central to the suggestion, outlined in a recent UNDP discussion paper on sustainable human development, that one of the indicators of sustainable human development is the extent to which social learning has taken place, as indicated by "... the emergence of new habits and routines (as) a critical indicator of social change" (Banuri, et al, 1994, p.28). The evaluation explored the extent to which the HDP activities had enhanced such social learning, and hence capacity development, by exploring a broader range of critical indicators, more accurately described as the emergence of new understanding and approaches, as the concepts of habits and routines were considered too static to describe either the activities of the Programme or the Programme's intended strategic effects. In line with the forward-looking purpose of the evaluation, and acknowledging the need for sustainable human development to grow out of people's lived experiences, the central issue of the evaluation process, reflecting the central question of the Programme itself, was "How can we, together, move forward?" In this case, the question was asked in relation to how we can use evaluation processes to generate a deeper understanding of the newer approaches to development practice. 2.2 The evaluation framework It was important that the evaluation be conducted in a manner consistent with the dynamic, and even opportunistic, nature of the Programme. The world keeps changing, the determinants and consequences of the HIV epidemic keep changing, and the opportunities available to a low-budget programme operating globally also keep changing. So does understanding of what works best to promote sustainable human development. Therefore, it was important that the evaluation avoid a result of "locking" the Programme into a fixed modus operandi. For example, an evaluation with a linear logic which simply considered inputs, outputs, or immediate impact would have been inappropriate. Such an approach would require some sense of "fixing" the Programme as a static entity, clarifying what were the precise goals and objectives and measuring the outputs at just one point of time and in one context: the world as it was at one point in time. Understanding the approach of the Programme as being based on a set of principles, from which arise methodologies and approaches, was therefore considered more accurate than attempts to define the Programme based on assumptions about clearly defined inputs and outputs. Early in the evaluation process, in July 1995, Programme staff developed a list of some of the concepts which the Programme was then using to describe its work and some of the concepts it was not then using. These were later used to generate discussion and reflection (see Stages 5 and 6 of the evaluation process, below) and are included in Appendix 2. The evaluation process was interactive and iterative. The interactive process involved working with the partners of the Programme, and others who use similar approaches in their own work, to develop together a deeper understanding of how the approaches work. The starting point was the Programme, but the evaluation enabled people to explore such issues with reference to their own lived experiences. This is consistent with the principle that, "The perspectives of and experiences of those persons who are served by applied programmes must be grasped, interpreted, and understood if solid, effective, applied programmes are to be put in place" (Denzin, 1989, p.105). The evaluation process was iterative in that it started with shared reflection about the nature of the Programme, went back into the field to explore how this understanding relates to people's lives, then returned to shared discussion about that, and so on. The process was, in these ways, based on the qualitative research method of interpretive interactionism (Denzin, 1989). This method emphasises the need to interpret real-life phenomena in order to reduce complex reality to understandable and debatable notions. It suggests doing this in an interactive way, because deeper understanding about the meanings and significance of experiences and relationships arises when people reflect and engage in dialogue with each other. Such an evaluation process meets two important requirements for the use of qualitative methodology: 1. The need to combine the collection of data with a rigorous interpretation of the significance of that data, as outlined by Patton (Patton, 1990, p.423):
2. The need to ensure that the people who conduct such interpretive analysis are those for whom the analysis is most important, as outlined by Denzin (Denzin, 1989, p.25):
The Conceptual Framework developed for the evaluation is outlined in detail in Appendix 1. Those interested in methodologies for evaluating processes might find it of interest. 2.3 The evaluation process: what was done Stage 1: Selection of evaluation team members The evaluation team consisted of four people who are themselves affected by the epidemic. They are all engaged in responding to the epidemic in their own contexts, and have all been involved in promoting sustainable human development as participants in international networks, as consultants, or as long-term staff in development projects. In response to the second evaluation requirement above, the team members have all been partners of the Programme. They reflect a diversity of experiences of the epidemic and come from different global regions. Directly including more people from developing countries in the evaluation team was planned but was, ultimately, not possible. The nature of the evaluation process ensured, however, that many people from developing countries had direct input in both provision of information and interpretation of results. The evaluation team members were Bruce Parnell (team leader, from Australia), Juan Jacobo Hernández (Mexico), Gro Lie (Norway) and Cindy Robins (Canada). Information about the evaluation team members is provided in Appendix 6. Stage 2: Selection of further participants in the evaluation process Decisions were made about countries to which field visits would be made and criteria for selection of participants in interactive discussions and in workshops. These decisions were based on the qualitative research method known as purposeful sampling, as defined by Patton (Patton, 1990, p.169):
The selection of evaluation participants was therefore based on determining which people were most likely to provide significant knowledge and insights which would contribute to a better understanding of how the approaches used work in practice. The Programme had used a variety of new and innovative programmatic and organisational approaches to assist governments, communities and organisations, and the selection of evaluation participants had to reflect these approaches. Criteria for selecting participants were that they represented groups with whom the Programme worked in partnerships, people who were involved in networks responding to the epidemic, people who had participated in SPR activities, and people who were identified by Programme partners or other participants as being people who work in ways that are compatible with the approaches used. The Programme's analysis had indicated that every facet of human, social and economic life can be affected by HIV and development, and that national responses must include collaboration between individuals, families, communities and governments. This mix was also reflected in the selection of evaluation participants, so that they included people from the governmental sector, non-government organisations, community groups, and grassroots movements as well as from the international community. The three countries visited were Zambia, the Philippines and Mexico. These reflect a diversity of regions, countries with different experience of the HIV epidemic and its impacts, and countries in which the SPR has had varying levels of contact. A fourth country, Senegal, was visited by the evaluation team leader for the purpose of conducting a limited number of interactive discussions. This choice was made because, although the Programme has worked in Senegal, availability of partners at the required time meant that it was not possible to conduct the full range of evaluation processes in Senegal. Further information about countries visited, and lists of evaluation participants, are included in Appendix 3. Details include summaries of prior HDP involvement within each country visited. Stage 3: Interactive interpretation of the Programme's approach The evaluation team leader worked with HDP staff in July 1995 to collate information about SPR activities and sustainable human development; to collectively reflect on the purpose, nature and approach of the Programme; to consider evaluation options; and to design a relevant evaluation process. A paper outlining the conceptual framework for the evaluation was prepared and distributed widely for comment amongst partners of the Programme and others interested in evaluation (attached as Appendix 1). It was developed as part of the evaluation process and was written with input from a number of people familiar with the work of HDP and evaluation methodologies to clarify the understanding of both Programme staff and evaluation team members. Stage 4: Developing further understanding of the approach of the Programme A planning workshop was held in Zambia in October 1995 for the evaluation team members, two members of the HIV and Development Programme staff and the UNDP HIV and Development national professional officer in Zambia, Margaret Mutambo. At this workshop, further consideration was given to the conceptual nature and approach of the HDP, with reference to selected publications on development practice, social change and evaluation which were tabled by workshop participants. The outline of the development practice framework for the HDP which is included in Section 4 of this report is based on the results of this period of the process. The process of interpretive interactionism was applied to exploration of workshop participants' own experiences of development, the HIV epidemic, and the Programme. This ensured that all evaluation team members were familiar with the concepts and methodologies of the evaluation process. It also ensured that the evaluation team commenced development of a collective understanding of the nature and significance of the approach of the Programme and a shared understanding of issues to be considered during the country field visits. Stage 5: Country field visits: interactive discussions Interactive discussions were held with selected participants from each visited country. This dialogue differed from a usual interview format, in that those interviewed were able to talk freely about what mattered to them, but the interviewers from the evaluation team were also free to pursue issues they considered important; hence, use of the description, "interactive discussions", rather than "interviews". The interactive discussions aimed to explore evaluation participants' own experiences of HIV and development and any involvement they had had with the Programme, and to engage with them in discussion about what approaches they find useful, regardless of whether or not their experience of those approaches was within SPR initiatives. These discussions followed the format outlined in Appendix 2, and lasted between one and three hours. The discussions explored issues through eliciting "thick description", as defined by Denzin (Denzin, 1989, p.83): A thick description does more than record what a person is doing. It goes beyond mere fact and surface appearances. It presents detail, context, emotion, and the webs of social relationships that join persons to one another. Thick description evokes emotionality and self-feelings. It inserts history into experience. It establishes the significance of an experience, or the sequence of events, for the person or persons in question. In thick description, the voices, feelings, actions, and meanings of interacting individuals are heard. Stage 6: Country field visits: interpretive workshops Interpretive workshops were held in each of the three primary countries visited in order to: - provide a chance for people from each country to interact with each other in interpreting understanding of how various approaches work within their countries; - provide a chance for larger numbers of people to participate in the evaluation process; and - ensure that interpretations made by evaluation team members were able to be further considered by the people whose perceptions of relevance and sustainability were being considered. The outline of those workshops is included in Appendix 2. Stage 7: Country field visits: review workshops People responding to the HIV epidemic and development in each country usually do so working with National HIV/AIDS Committees, UN agencies and international donor agencies. To further ensure that the findings of the evaluation process were based in reality, and to ascertain the extent to which new approaches in sustainable human development are understood and supported (or not) by these agencies in each country, a short review workshop was also held in each country, and these agencies were invited. At least some members of national HIV/AIDS committees in each country had been included in Stages 5 and 6 of the evaluation process. For this reason, very few also attended the first reporting back workshop review in Zambia, as this would have required yet further time commitment. Accepting this likely scenario, and in order to reduce the workload on those who had voluntarily offered to make the arrangements, it was decided that members of these committees should not be invited to attend the workshops in the Philippines or Mexico. Many staff of UN agencies and international donor agencies attended the review workshops in each of the three primary countries. The workshops had a simple format, in which the nature of the Programme and the evaluation process was explained, the lists of concepts currently used and not used by UNDP were presented (these lists are included in Appendix 2), and initial interpretation of data by evaluation team members was presented for further discussion. These workshops generated much interest, and dialogue ensued. Many participants indicated a good understanding of issues which were relevant within their countries of work, and demonstrated a keen interest in developing understanding of the interrelationships between the HIV epidemic and sustainable human development. Stage 8: Commissioning of case studies To complement information collected through field visits and consideration of published information, two case studies were commissioned to further explain the way the Programme worked in practice. The first of these, entitled "Enhancing National Capacity Through HIV Action Research", was written by Catherine Hankins, a partner of the HIV and Development Programme, and two members of the Programme staff. It summarises what occurred and some of the key lessons learnt through this component of the SPR. The initiative aimed to enhance the national ability of people to find out what they need to know and to incorporate what they find into policy and programme design (see Section 3.3). This case study is included to provide insight into the practical ways in which a specific project was developed within the framework of the SPR, to indicate some of the lessons learnt through these types of processes and, consequently, to provide an example of the way the Programme evolved over time in response to lessons learnt. This case study is included in the text of this report, after Section 5. The second case study, entitled "The Partnership Between the Salvation Army and the HIV and Development Programme", was written by Alison Rader and Captain Doctor Ian Campbell of the Salvation Army. It summarises the conceptual understanding about the nature of effective practices in sustainable human development which evolved over a long period of partnership between the Programme and the Salvation Army, and partnership between the Salvation Army's International Headquarters and its own partners in many countries. This paper's focus on development of conceptual understanding is deliberate, for two reasons. First, Salvation Army personnel and HDP staff reported that gradually enhanced understanding of development practice was one of the primary benefits to all parties engaged in this partnership. Second, evaluation team members felt it was useful to highlight specific lessons learnt through one particular partnership, as a typical example of the richness of what was achieved through an approach based on partnership. This case study is also included in the text of this report, after Section 6. Stage 9: Interpretation of data Collected data included information arising from shared discussions: discussions within the evaluation team and with Programme staff in the first four stages of the evaluation process, and discussions held with individuals and within workshops in Stages 5 to 7. In line with the processes of interpretive interactionism, this data was then collated, analysed and interpreted by the members of the evaluation team through a final workshop in New York over five days in November 1995. This workshop followed an open-ended format, enabling development as it progressed and input from evaluation team members and Programme staff. Once again reflecting the interactive nature of the evaluation process, participation of others in this workshop was encouraged. At various stages, the workshop was attended by all HDP staff, by Gary Engelberg (a partner of the programme based in Senegal) and by Alison Rader of the Salvation Army. During this workshop, the evaluation team identified the key concepts about which the evaluation process had led to deeper understanding, and shared information about what had been learnt about these concepts through the country field visits. The summary of understanding of how the Programme approach works in practice, which appears in Sections 5, 6 and 7 of this report, is based on this final workshop. Stage 10: Report writing The report was written following the final workshop. A brief report such as this cannot capture all that was learnt either through data collection or through the interpretive discussions. It is expected that further information about what has been learnt will be disseminated through the work and lives of the many people who participated in this process. Selected notes made by evaluation team members during the data collection process are available on request from the HIV and Development Programme. These provide summaries of interactive discussions with evaluation participants, and thus enable richer insights into the ways in which the approaches of the Programme have worked in practice. Statements made in confidence are not available for distribution, but these constitute only a small proportion of the data collected.
|
