By Calum Handforth

When measuring success and impact in international development, we often focus on ‘Big Data’. This is particularly the case in the context of smart cities and digitalisation. However, arguably, the Lean Data approach is more relevant for many international development projects. It is an actionable approach to learning what does and does not work.

(As part of our Smart City Training Series, Katie Mountain (from 60 Decibels) shared the Lean Data approach. Watch her presentation here.)

Lean data is a portfolio approach to understanding impact, with a focus on shifting from impact measurement to impact management. It’s also about moving from top-down compliance to bottom-up accountability.  Fundamentally, Lean Data is a measurement approach tailored to meet the needs of the beneficiary. Lean Data is also particularly relevant when working to understand the impact of digital interventions.

The Lean Data process comprises five stages:

1. Defining goals: identifying what you want to learn. This includes in the context of project or programme design, and the intended outcomes for your customers or other beneficiaries. This is not a static process, either, as goals can shift or change completely over the course of a project. The agility of Lean Data methods is a strong asset here, as is leveraging the channels of relevance to your project or wider work.
2. Survey design: selecting or developing the right questions. These can come from surveys developed by established organisations, or be created from scratch. However, questions must avoid bias, be respectful, and also standardised to allow benchmarking. Questions should also cover ‘all’ bases (i.e. customer profile, business insights, and impact) and be a mix of quantitative and qualitative. The experience of 60 Decibels has highlighted that any survey should have a maximum of 40 questions.
3. Data collection: selecting and deploying the most appropriate method for your sample, and context. Data collection methods should align with this – including using approaches such as online, IVR, phone, SMS, WhatsApp, and in-person. Similarly, select the sampling strategy and confidence interval needed to understand impact – and the type of study most appropriate to doing so (such as one-off, baseline-follow up, longitudinal, or using a comparison group). Make sure you test the approach, and consistently check quality once the survey goes ‘live’.
4. Analyse: making sense of your data. The 60 Decibels team use sector-specific benchmarks to provide further detail here, allowing organisations to compare their performance to peers – or others operating in the wider sector. This is particularly relevant in the context of fast-moving challenges such as COVID-19 – including through leveraging tools such as dashboards.
5. Sharing and using: developing an action plan to identify how to use the data, and to track progress. This is an iterative process, developed over time – including modifying survey instruments as the data collection needs of the organisation (or product, or service) change.

The Lean Data Field Guide , developed by the 60 Decibels team, is a useful resource. It includes a number of core questions, guidance on sampling strategies, and other aspects of the above approach.

Importantly, Lean Data is also about dignity. Often, measurement approaches seek to collect as much data on lives and livelihoods as possible, without consideration for the privacy or rights of individuals. Lean Data highlights the importance of collecting only the data that is needed to measure impact – using tools and techniques that are part of the customer journey.

The value of lean data for cities

Lean Data has particular value in the context of digitalisation, and smart city initiatives more broadly. In a digital product or service, every step needs to add value to the user – or they will go elsewhere. This is a lesson highlighted by other organisations in the sector. Measurement needs to be agile. Similarly, measurement cannot be extractive: digital products and services must demonstrate value, before looking to engage from a measurement perspective.

Digital products and services have a particular advantage in comparison to ‘typical’ development interventions. Their very nature allows ‘invisible’ measurement, drawing on the data and metrics collected through user interactions and from the ‘backend’ of systems.  This type of measurement does not require any formal input, can be measured over the longer-term, and at very regular intervals. However, this reaffirms the importance of defining measurement goals and balancing measurement priorities – including collecting data to improve the product or service, but also to understand its impact.

As an aside, measuring digital products and services can present an inherent challenge. Digital allows exploration of very granular user experiences – down to the level of single users on some platforms. However, this can encourage an excessive focus on individual datapoints – at the risk of missing or failing to understand broader insights that can drive iteration and eventual impact. This misdirected focus could even cause bias, as fixating on the experience of a certain group of users may affect how products and services are developed.

Both Big Data and Lean Data are important ways of exploring and understanding impact in international development. Each approach has enormous value, and each should be considered as a tool in a broader measurement toolkit – neither is ‘better’. What they highlight, though, is that impact measurement is non-negotiable. We have a responsibility to identify – and learn from – what is and is not working. Similarly, as digital interventions continue to increase, we need to shape tools and approaches to measure their success.  Measurement is a fundamental asset in developing products, services, and initiatives to improve lives and livelihoods – whether in smart cities, or elsewhere.