LocationsThe challenge of learning from others
30 de Marzo de 2023
Sometimes, cases with the best indicators aren’t the most appropriate referents. For example, a student who wants to improve her grades may ask the student with the best grades: how do you do it? Often, the response she would get is frustrating; she concludes that she will hardly be able to improve her grades because she doesn't have the skills nor access to the resources the other student has. Thus, comparing ourselves to the best-performing case is unfair and counterproductive.
Alternatively, a better reference could be cases with unexpected success. For example, those students who may not have the best grades yet get better grades even with considerable resource limitations. But, how do I identify those unexpected success stories?
There is an analysis technique called "Data Powered Positive Deviance" (DPPD) that allows identifying cases with development results greater than expected. Authors such as Marsh (2004), Sterning et al. (2004) and Swartz (2013) have been pioneers in this technique.
Among the benefits of DPPD are recognizing extraordinary practices that can overcome adversities and giving back to people the leading role in designing solutions to their challenges. However, an additional challenge is that once success stories are identified, we don't always take the step to learn from them.
At the Accelerator Lab at UNDP Guatemala, we learned to use the DPPD in 2021, during a virtual camp organized by the global network of laboratories in partnership with researchers from the University of Manchester. Following, we share some lessons from our experience trying to take advantage of this innovative technique to accelerate sustainable human development.
Our first approach with DPPD was to help improve access to waste collection services. In Guatemala, on average, only 40 percent of the people in a municipality have access to this service, which results in greater contamination. Using DPPD with available data, in less than a week, we identified 6 municipalities with higher-than-expected service coverage. We prepared a report with this finding to share with projects and proposed a guide for conducting field visits. We were full of enthusiasm to show that there are municipalities that have been able to achieve greater coverage of the waste collection service in adverse conditions, and that learning from them could be the key to achieving greater results.
Six municipalities with unexpected success in the coverage of the garbage collection service: San Marcos La Laguna, Santa Catarina Palopó, San Pablo la Laguna and Santa Clara la Laguna in Sololá; Ipala in Chiquimula; and San Juan Cotzal in Quiché.
It has been enriching to receive questions about this technique. When answering, what are "positive deviances"? and how are they identified? It has been a challenge to explain in a short time, to audiences without training in statistics, that we’re referring to cases that have an observed performance considerably higher than the performance estimated by a predictive model. Similarly, questions about whether the predictive model is optimal are difficult to answer; beyond saying that although it has limitations, it's sufficiently valid, has a high predictive capacity (R2 > 0.6) and was obtained quickly with available data.
We know that among development practitioners, the results of the DPPD are strange. Traditionally, only extreme cases are highlighted: those where it´s urgent to intervene and those to which it´s necessary to aspire. Highlighting imperfect cases with potential to learn, as DPPD does, is unusual. Despite this, in this first use of the DPPD, it coincided that UNDP Guatemala implements local development programs in one of the municipalities identified with unexpected success: San Juan Cotzal in Quiché. Therefore, the use of DPPD contributed to support proposals to give continuity to existing programs and projects.
Our second use of the DPPD was to help reduce garbage burning in the Motagua River basin. UNDP Guatemala currently supports efforts to improve solid waste management in that basin, a challenge that also affects Guatemala's relationship with Honduras. Using DPPD, we identified 5 municipalities with a lower-than-expected incidence of garbage burning. This time, we presented a report that compared the results of DPPD with the results of traditional techniques and another report with visualizations. We believed that presenting different alternatives would make it easier to recognize the value of learning from unexpected successes before taking actions in the places of greatest need.
Contrast between DPPD results and greatest need analysis
When contrasting results between alternatives, new questions arise. On one side, it was considered a priority to intervene cases where there is a greater need to improve waste management rather than to learn from cases where there is unexpected success. On the other side, since the DPPD results don’t indicate why there is unexpected success, they don’t immediately offer a recommendation on how to act. It’s difficult to justify the use of resources destined for development projects without offering a direct development impact. It is possible that seeking learning, beyond the lessons learned during the implementation of a project, is considered secondary to obtaining results and a more academic exercise.
Our third use of the DPPD was to promote the electoral registration of indigenous populations. UNDP Guatemala supports the full inclusion, in the exercise of their civic rights, of historically marginalized populations such as indigenous peoples. On average, an indigenous municipality has 10.5% fewer electoral registered residents than a non-indigenous municipality. Using DPPD, we identified five municipalities with a high indigenous population and a higher-than-expected percentage of registrants. To us, the potential to learn about ways to promote participation relevant to the indigenous population seemed enormous.
Five municipalities with a high indigenous population and unexpected success in promoting registration: Chuarrancho (112) in Guatemala; Santa Catarina Barahona (316) in Sacatepéquez; Almolonga (913) in Quetzaltenango; and San Pedro la Laguna (718) and San José Chacayá (702) in Sololá.
On this occasion, the main barrier to use this result was that the identified municipalities were outside the departments prioritized by the project. Additionally, we’re pointed out that to start learning exercises in a territory, not only resources are needed, but also relationships of trust that take time and generate expectations. In other words, however, knowing that there are unexpected success stories generated interest to investigate more carefully in the future.
Our fourth use of the DPPD was to strengthen the management of local governments to promote development. UNDP Guatemala periodically prepares the National Human Development Report (INDH) which includes measurements of both the Municipal Human Development Index (IDH-M) and the Multidimensional Deprivation Index (IP-M). Using DPPD, municipalities with results in development indices higher than expected and lower than expected were identified. We hoped that linking learning source cases to learning target cases would make the value of learning from unexpected success stories more apparent.
Figure 4 compares municipal income, one of the predictive model variables with the greatest impact, with the deprivation index (IP-M). Each circle represents a municipality according to its income and deprivation. Each point on the line represents the expected value of deprivations for a municipality based on total income. The analysis suggests that it’s important to learn how a low-income municipality such as Santa María Visitación can have less deprivation than other municipalities such as San Marcos la Laguna or Santa Catarina Palopó.
Municipalities with less deprivation than expected in blue and greater deprivation than expected in red
Using the DPPD analysis to propose a learning activity, rather than a direct intervention to achieve development results, was strange, as just detecting unexpected successes doesn’t indicate how to achieve impact. However, the value of DPPD to facilitate exchanges of experiences between municipalities was recognized. With this validation, a prototype was prepared that would allow local governments to identify peers from whom to learn.
Prototype of a digital tool to make exchanges of successful experiences to achieve local development.
The digital tool would make it possible to select a municipality and then identify municipalities with similar characteristics that have been successful in different areas. An additional benefit is that the municipalities' development performance measurements are reformulated in a positive light.
Local governments often resist measurements of performance, because the result exposes them to overwhelming criticism. However, by offering peer-to-peer learning, attention is directed to recognizing opportunities about what is possible to achieve. Work is currently underway to take advantage of this variation of the DPPD analysis to promote the use of INDH results in local government transition.
In the network of UNDP Accelerator Labs, at a global level, the use of DPPD is being promoted to accelerate development in aspects such as creating safe public spaces for women, improving livestock practices for the sustainability of forests, improving grazing for water sustainability, the use of sensors to improve food security, among others. We hope that in Guatemala we continue to take advantage of this potential techniques such as DPPD to promote development by learning from unexpected stories of success that have been able to overcome adversity.
Written by:
- Javier Brolo, Head of experimentation: UNDP Accelerator Labs Guatemala.
- Juan Pablo Rustrián, Junior quantitative analyst: UNDP Accelerator Labs Guatemala.
