Development and the HIV Epidemic: A forward-looking evaluation of the approach of the UNDP HIV and Development Programme APPENDIX 1: THE CONCEPTUAL FRAMEWORK OF THE EVALUATION
BACKGROUND During the fifth UNDP programming cycle (1992-96), Special Programme Resources (SPR) were made available to enhance the relevance, impact and effectiveness of UNDP's technical cooperation programmes. These resources were to be used to improve current procedures for and approaches to programme and project design, implementation and evaluation and for exploring new catalytic and innovative approaches. The HIV and Development Programme (HDP) was allocated $5m for the cycle, subsequently reduced to $3.3m. A broad-based consultation process was initiated and a workshop/ brainstorming held October 1992 with a diverse group of existing or potential partners to discuss the best use of these resources given the nature of the epidemic and their experience in responding to it, and given the aim of the SPR and the mandate of the HIV and Development Programme (DP/1991/57) to focus on the interrelations between development, the epidemic and the expertise it contained. This process of consultation led to an important set of understandings which were to inform the SPR: the nature of the HIV epidemic demands an urgent and effective response; governments, development professionals and other agents of change find themselves facing a new and highly complex phenomenon; and new knowledge and programming approaches are needed to respond in an effective and timely way to the multidimensional aspects of the epidemic. It was decided that the HDP SPR resources should be used to deepen understanding of the nature of effective responses to the epidemic, to determine what works and how, and what does not, and to take the best practices of development and apply them in this field. Because of the urgency of improving the effectiveness of the global response to the epidemic, wherever possible, these insights and lessons were to be drawn from ongoing initiatives. New initiatives were to be funded only when the Programme could not fulfill this condition. The resources were thus not to be considered as supplementary to existing UNDP IPFs, especially country IPFs, but rather were to be used to increase UNDP's capacity to programme its resources more effectively. By their very nature, activities funded from these resources would not duplicate those undertaken by others but, rather, be used to draw from them the critical lessons.
DEFINING CHARACTERISTICS OF THE USE OF THE HDP SPR The approach adopted by the HDP is interactive and iterative. Its expressed aim is to ensure that, in each area of focus, there is a critical mass of informed, engaged and committed people working together within a supportive milieu towards an effective response. The nature of the interactive processes between these partners and the Programme is intended to assist all those involved, including Programme staff, in understanding the epidemic differently, acting differently in their personal and professional lives and interacting differently with others in the various contexts within which they are responding to the epidemic. Consequently, the approach adopted is itself in a state of constant development, with frequent use of ongoing processes of innovation, reflection and review. The strands of the approach can be drawn out into the following:
The Programme staff experience these interactive processes as transformative. They lead to changes in their understanding, in ways of acting, in ways of being, changes which continually enrich and make more complex their analysis and practice. They take these changes back to their relationships with others and so change the nature of these relationships. It is inherent in this approach that the outcome of these interactive processes with the Programme's partners cannot be predetermined. Nor can that of subsequent interactions of its partners with others. Although the Programme may feel that certain things are important and can bring that analysis, those beliefs, that knowledge, into the interactive processes, what is decided to be done, how it is decided to do it, what agreement is reached, what beliefs people come to hold will be determined by the nature of the processes themselves. The areas of focus of the Programme have arisen from these interactive processes. They include:
The SPR is thus clearly situated within the overall conceptual and operational framework of UNDP's work and consistent with concepts centrally located therein, the concepts of sustainable human development, especially of human survival, of social capital formation, capacity building, social learning and partnership (Banuri et al, 1994), of expertise (Chambers, 1994), of gender responsiveness, of process consultation (Joy and Bennett, 1994), to which it adds the concepts of strategic questioning (Peavey, 1994) and governance. The Programme actively promotes questioning about the nature of expertise in an epidemic in which standard approaches have, by and large, failed, and promotes more active consideration of the role of expertise arising from people's lived experiences of the epidemic as it affects them in their own contexts. In drawing together these various but related perspectives, the Programme recognises that social change is crucial to the development of effective responses to the epidemic, but that the specific social changes will vary from place to place. One way to articulate the nature of the use of the SPR in summary would be to say that the programme is based on an interrelated series of active and reflective processes which are used to promote wider use of similar interrelated active and reflective processes. The desired outcome is the development of more effective and sustainable responses to the HIV epidemic, and the immediate impact is exemplified in an emerging discourse which places people, rather than the virus, at the centre of analysis of, and response to, the epidemic. In this sense, the SPR has been used as a means of evaluating broader responses to the HIV epidemic: to challenge commonly held assumptions about what drives the epidemic and what may work to slow it down, and to shift the focus of attention onto the processes of social change which will lead to effective community and national responses to the epidemic. Like other means of evaluation, the Programme challenges people to consider available data, carefully plan use of available resources, and continually question the assumptions on which they base their present responses. UNDP's manual on process consultation highlights the importance of approaches to programming which take account of the fact that each country is different and the world is rapidly changing:
The current evaluation process is being carried out in line with the principles of process consulting.
EVALUATION OF PROCESS-ORIENTED PROGRAMMES It is important that evaluation of such a Programme be conducted in a manner consistent with the dynamic, and even opportunistic, nature of the programme. Such evaluation should not attempt to "lock" the programme into a fixed modus operandi. This means that evaluation with a linear logic which simply considers inputs, outputs or immediate impact would be inappropriate. Such an approach would require some sense of "fixing" the programme as a static entity, clarifying what were the precise goals and objectives and measuring the outputs at just one point of time and in one context: the world as it is at one point in time. In fact, the world keeps changing, the determinants and consequences of the epidemic keep changing, and the opportunities available to this low-budget Programme operating globally also keep changing, as does understanding of what works best to develop capacity. Thus, the very aims and objectives of the Programme, which are often the starting points for developing evaluation which considers programme logic, inputs, outputs and outcomes, are necessarily in a constant state of change. It became clear during the initial design stage of the evaluation that Programme staff did not want their way of working to be reduced in ways which become too narrowly and inflexibly defined. Thus, for example, the reduction of the approach to a series of principles from which arise methodologies was considered far more favourably than attempts to define a programme logic based on assumptions which lead to clearly defined inputs and outputs. Programme staff developed a list of some of the concepts which the programme uses to describe its work and some of the concepts it is not using so that these could generate a process of discussion and reflection. It is important to note, however, that while the use of certain concepts to describe component elements in building capacity may be useful as a means of generating discussion about the approach, the evaluative process is not intended merely to lead to the replacement of one set of concepts with another. So, what type of evaluation would be appropriate to a Programme which is itself in a state of constant review and evaluation and which takes place in and acts upon an external context which is constantly changing and thus requires constant critical reflection? The answer to this question arises from consideration of the purpose of the evaluation, for evaluation is a process inextricably linked to a sense of purpose. There is no such thing as an all-purpose evaluation. Therefore, decisions have to be made about what kind of evaluation will be most useful for particular purposes. Importantly, the evaluative process is intended to be a forward looking assessment of the approach used. Those for whom the outcome may be of value and interest may not be the same people defined in many evaluation texts as "key stakeholders". Rather, given the Programme's concern with developing capacity, the intended users of an evaluation process, in this instance, could include primarily people involved in capacity development in their own contexts: "insiders" in their own communities, nations and regions. This group would primarily include the partners, direct and indirect, in the SPR initiatives. However, it also includes people and organisations involved in assisting and enabling these "insiders" to strengthen their capacity to respond: those "outsiders" who work in other people's contexts (National HIV/AIDS Program staff, staff of UN agencies, consultants, academics, etc.) For the current evaluation process what this suggests is that use of this process to enhance understanding about capacity building in relation to the HIV epidemic in general may be far more important and useful than enhancing understanding of the specific short term effects of the SPR initiatives funded to date. This does not mean ignoring the outputs of the SPR; rather, it leads to consideration of what was learnt through those programmes in a broader context, starting with the programmes and those directly involved as programme partners, but moving towards a broader understanding of the concepts underlying both those and other programmes intending to enhance capacity. In this way, the current evaluation fits firmly within the evaluation framework outlined by Benbouali in a paper intended to provide guiding principles for evaluation of UNDP programmes (Benbouali, 1995). Benbouali notes in relation to programmes such as the one under consideration that "The development objectives are finally the results of strategic effects of many programmes/projects and efforts" (p.2). It is therefore critical that evaluation processes consider strategic effects, not just inputs, outputs and immediate objectives. A means of enhancing understanding of strategic effects, as suggested by Benbouali, is to concentrate on issues of relevance and sustainability. These are defined as follows:
The concepts of relevance and sustainability underlie the description of the purpose of evaluation suggested by Benbouali, in order to draw attention to the difference between this type of evaluation and related but different concepts of programme auditing and assessment: "Evaluation (strictu sensu), in our new definition, should be focussing more on the relevance, impact and sustainability of the programme/object" (p.4). To the latter, we may add "approach". This makes such evaluation distinct from auditing and assessment, which focus more on managerial and financial issues, and on measures of the effectiveness and efficiency of programme activities in the light of their objectives, that is, on programme performance. The present evaluation process will be carried out in line with these evaluation concepts. In particular, given that this is not the only evaluative process used by the HIV and Development Programme, the programme itself is small and the budget and timeframe for the evaluation is also small, the present evaluation will, by conscious choice, focus primarily on relevance, impact and sustainability of the approach, that is, on the strategic effects of the Programme. In essence, this means that the current process will concentrate on evaluating the approach of the Programme , questioning how this approach works, what it means to those with whom the Programme works, and how the approach may be improved, regardless of which programme uses it. The focus on relevance and sustainability leads to a focus on questions about "how". Denzin (Denzin, 1989) suggests that much of value can be learnt from replacing the "why" question with the "how" question: "That is, how is social experience, or a sequence of social interaction, organized, perceived, and constructed by interacting individuals?" The use of these approaches to evaluation suggests that key questions about the use of these SPR funds may centre not so much around, "What did this Programme do?" or even "Why was that approach used?", as on:
Through such questions, a better understanding of capacity building will be reached. The classical focus of capacity development is on initiatives such as training, study tours, workshops, institutional development (Joy and Bennett, 1994). In the context of the HDP approach, capacity building is a complementary concept more closely tied to social capital formation through processes of human interaction. Its focus is in some sense on the non-measurable, on the norms, principles and values by which individuals and communities live their lives. It is these norms, principles and values which provide the accountability framework for institutional change and governance. In investigating these types of questions, it is essential to involve those who have been partners in programme initiatives in defining what strategic effects are most relevant to building capacity in their own contexts. This is consistent with an approach to evaluative research suggested by Denzin, who notes that, "The perspectives of and experiences of those persons who are served by applied programmes must be grasped, interpreted, and understood if solid, effective, applied programmes are to be put in place" (Denzin, 1989, p.105). These emphases on the investigation of the strategic effects of the Programme, the involvement of those who have been partners of the programme, and the exploration of relational issues require the development of new methodologies in evaluation. As a starting point, this evaluation will adapt an approach to qualitative investigation suggested by Denzin and called "Interpretive Interactionism" (Denzin, 1989).
A FRAMEWORK FOR THE EVALUATION PROCESS: INTERPRETIVE INTERACTIONISM The methods Denzin suggests for proceeding with interpretive interactionism provide useful ways to deal with Patton's suggestion that "...we must be constantly moving back and forth between the phenomenon of the program and our abstractions of that program, between the descriptions of what has occurred and our interpretations of those descriptions, between the complexity of reality and our simplifications of those complexities, between the circularities and interdependencies of human activity and our need for linear, ordered statements of cause and effect" (Patton). In particular, the methods suggested for interpretive interactionism involve constant consideration and re-conceptualisation of the phenomenon to be studied (i.e. the approach and methods of the programme in question) in close consultation with those directly affected by the programme, and in a context which takes for granted that value-free interpretive research is not possible. The method assumes that the most useful interpretations will arise if those involved in the process (both evaluators and programme participants) bring their own life experiences to the process, rather than attempting to act as some sort of independent objective observers or unengaged respondents:
Thus, with this SPR, a useful evaluative process could concentrate on further elucidating how the interactive processes of capacity development are organized, perceived and constructed by both "insiders" and "outsiders". Such an evaluation would then be grounded in the lived experiences of those directly involved with the Programme, and the forward looking component of the evaluation would grow out of those involved with the Programme, and as a means of relating those perspectives to further consideration of the assumptions and methodologies underlying this and similar programmes. Denzin suggests six phases, or stages for a process of interpretive interactionism, each of which is explained below and adapted for the current evaluation process:
Each of these phases is discussed here, with direct reference to the current evaluation process.
Phase I: Framing the evaluation question Denzin suggests a multiple-stage process for framing the research question. The current evaluation takes place in a context where much of this has already occurred, but processes for refining the evaluation questions will continue to be developed during and following the evaluation. This is also consistent with the suggestions that, "The researcher is led to seek out subjects who have experienced the types of experiences the researcher seeks to understand. The subject in the interpretive study elaborates and further defines the problem that organises the programme. Life experiences give greater substance and depth to the problem the researcher wishes to study" (Denzin, 1989, p.49). In this evaluation, the starting point will be with the appointment of a team of evaluators who are themselves people who have directly interacted with the Programme. In all cases, members of the team will be people who have been directly affected by the epidemic in their own social and geographical contexts, and who have also been involved as "outsiders" in development work aiming to build the capacity of others to respond to the epidemic. Thus, the team will be made up of people who are themselves direct "stakeholders" in developing a better understanding of the usefulness of the approaches to capacity building attempted throughout the SPR projects, and are therefore appropriate people to further define and clarify the purposes of the evaluation project itself. Denzin suggests the following stages for framing the research question. Denzin's stages assume the research focuses on personal "crises and epiphanies". In fact, while crises and epiphanies may be useful starting points, the current evaluation focuses on a broader range of interactive processes and outcomes, and focuses on effects of the programme on communities and nations, and so on individuals. It is therefore important that the following suggested stages be regarded as broadly defining parameters, not specific definitions of the evaluation process. Comments in italics indicate the way these stages will be used in the evaluation process. Stage 1. Locate, within one's own personal history, the problematic biographical experience to be studied. Researchers work outward from their own biographies.
Stage 2. Discover how this problem, as a private trouble, is, or is becoming a public issue that affects multiple lives, institutions, and social groups.
Stage 3. Locate the institutional formations, or sites, where persons with these troubles do things together.
Stage 4. Begin to ask not why but how it is that these experiences occur.
Stage 5. Attempt to formulate your research question into a single statement.
The approach which provides the framework for the SPR initiatives aims to enhance capacity to develop effective responses to the HIV epidemic through interactive processes.
Phase 2: Deconstruction and critical analysis of prior conceptions of the phenomenon As part of ongoing SPR development, this process has already been repeated again and again. Analyses of the phenomenon of capacity development with regard to responses to the HIV epidemic are included in the initial SPR project descriptions, in various discussion papers which have been written to draw together experiences and lessons learnt through the SPR projects, and in non-written formats through the use of interactive processes such as the workshops on HIV and Development and the processes used in the "Programme Development through Research" project. Concepts drawn from the work of the programme, which describe the its principles and methods, will be used for commencing discussions which will form the basis of this step of the evaluative process. Doing this necessitates reducing complex phenomena to simple words and phrases and this is a limiting factor of this process. However, it is considered a useful process for three reasons:
Phase 3: Capture In this phase, the evaluation team secures multiple instances of the experiences being studied. The usefulness of narrative processes in this phase is central. Denzin notes that, "Any story can be used, if it contributes to a general understanding of the phenomenon". It will be important to develop "thick descriptions" of instances of capacity building, built up from different perspectives, and looking towards the future as well as the past, in the sense that thick descriptions include not just what physically occurred (or is still occurring) but the way this relates to people's lived experiences. Denzin notes that:
The capture phase will be conducted in three ways. 1. Case studies of work done using SPR resources An inventory will be prepared of all activities supported from SPR resources. A number of case studies will be prepared focussing on some aspects of some of these initiatives, including the lessons learnt to date in the Programme Development through Research project, the partnership with the Salvation Army, and the genesis of the Network of African People Living With HIV and AIDS. The case studies will include consideration of issues around "how" the process of capacity building worked or could work better, not just "why" things were done the way they were. These case studies can therefore be used in the final evaluation report to give readers an insight into how this programme's approaches are useful in capacity building, rather than concentrating too much on the specific description of inputs, outputs, constraints and consequences that are often the main focus of such case studies used in evaluation reports. 2. Flow-on narratives Apart from the short personal stories to be written by team members, interviews will take place in each country visited with people who:
Such interviews will again focus on "how" capacity building works, aiming to encourage people to reflect on their own, their communities' and their countries' experiences with capacity building in relation to the HIV epidemic. The lists of Programme concepts will form the basis for open-ended strategic questions to be used in these interviews. In this way, the concepts which underlie the Programme's approach will be used to initiate reflective discussion, but people interviewed will be free to take issues in whatever directions they feel are important. 3. Group discussions In group discussions to be held in each country visited, thick descriptions of how capacity building has worked to date will be shared and discussed by participants. Those involved in developing the flow-on narratives will participate along with others interested in interactive processes and capacity building. In this way, participants will have already thought about key issues.
Phase 4: Bracketing or reduction Denzin notes that this leads the researcher to attempt to isolate the key or essential features of the processes under examination. In this case, evaluation team members would meet after conducting the interviews, to discuss together what concepts were of most concern to those interviewed, and what similarities, differences and points of interest would be relevant to explore further during the following group discussions. As much as possible, the bracketing or reduction should be done in close consultation with those interviewed. For example, towards the end of each interview, the interviewer could either suggest some key concepts which have been discussed to check if they are indeed good interpretations of the central concerns of the person interviewed, or the person interviewed could define such concepts themselves. Further bracketing or reduction can also form the basis of the start of the group workshops to be held as part of the next phase. For example, group process exercises could be used to enable the group to choose key concepts for commencing the discussion. The group workshops themselves can commence with consideration of a list of concepts, and go on to consideration of actual examples of capacity building which have occurred, or are ongoing, in the countries being visited. In this way, the group discussions will commence in the capture phase of the interpretive interaction process, develop through the bracketing or reduction phase, and then conclude with the next phase, construction.
Phase 5: Construction This phase involves, "the attempt to interpret the event or process". Denzin notes that this stage, "... classifies, orders, and reassembles the phenomenon back into a coherent [and hopefully complex] whole". In the most simple sense, this would involve group participants, after reflecting on key concepts in capacity building, actively discussing how capacity building might work best in their own countries in the near and more distant future. Such discussion might commence with questions like, "Given that these particular concepts are held to be important to capacity building, what do they indicate to you about how to move forward in this area?" In this part of the group discussions, a number of methodologies will be used, including the creative use of metaphor as a means of exploring the lived experience of participants, and in discussing how capacity building processes can be strengthened and expanded. This would serve as a means of both double-checking on shared meanings of particular concepts, as well as initiating further discussion on how these shared meanings might result in living practices. For example, if the groups have generated lists of key concepts in the morning, in the afternoon they might each be asked to consider the metaphors which underlie the concepts and see how these inform the way in which capacity building has been, or could be, undertaken. The identification of the metaphors which consciously or unconsciously underlie discourse and actions within the response to the epidemic is an important methodological tool since these metaphors are realized in programmatic approaches, in social practices, laws and social institutions. They impose a structure on life (Lakoff, 1992). Metaphors determine the distribution of power and define what constitutes moral issues. The dominant metaphor in the public discourse on the epidemic has been that of intervening, of social engineering, but there are emerging discourses based on different metaphors which the Programme has helped to shape. The group discussion would then focus on the further articulation and development of these metaphors and the discussion of the situations of their appropriate use in capacity building and other dimensions of the response to the epidemic. These metaphors would be included in the evaluation report. As well as group workshops with those whose lives are lived within the epidemic, that is Programme partners and others who have an interest in an interactive capacity building approach, further group discussion could be held in each country with others who contribute to capacity building in those countries (in-country staff of UN agencies, donors and NGOs and national HIV/AIDS programme staff, for example). Participants in the flow-on narratives and the group workshop would also participate and the discussion could explore how the group understands capacity building and their experience in trying to strengthen it.
Phase 6: Contextualisation This occurs when, "... the research locates the phenomenon back in the worlds of lived experience". For the purposes of the evaluation, it would involve the following components:
Bruce
Parnell
|
