On Monday, Tony Loyd was nice enough to include me in his great series of Social Entrepreneur podcasts. We covered a lot of topics but dug in most deeply on Lean Data, particularly on how we are using it at Acumen to amplify the voice of low-income customers so our entrepreneurs can better serve them. It was a fun conversation.
(if you’re not seeing the embedded link click here)
If this kind of thing is up your alley, you might want to sign up to receive the specialized newsletter we’ve created to share hot-off-the-press insights on what we’re learning through Lean Data. We send it out once every six weeks or so, so it won’t clog your inbox, and it’s full of great stuff.
Every day, more than 5 million new cellphones are sold. That’s more than 10 times the number of babies born each day. We are barreling towards a world where a cellphone will be in every pocket by 2020, and a smartphone in every pocket soon after that.
This revolution is making the unimaginable real— in the near future, we will have the opportunity to start a dialogue with literally every person on the planet. This new two-way conversation, where everyone participates, will pull billions of people into the mainstream by connecting them with one another.
Since starting this work in 2014, one of the most important lessons we’ve learned is that a cellphone in every pocket is just a starting point. The art of every Lean Data project is in the questions we ask. Ask the wrong questions, and you get back little of value. Ask the right ones, and you can move from data to information to actionable insights.
Great questions connect with customers and give them an opportunity to share their voice. But crafting a great question is no easy task. The slightest shifts in word choice can affect understanding; the smallest differences in intonation alter perceptions of sincerity. All of these nuances can bias the data and diminish its value.
For example, in trying to understand the usage of solar home systems in Kenya, we started with the question, “How often are you currently using (product/service)?” After testing this question over SMS, we received feedback suggesting we omit the word “often” and make the question more simple and direct. We quickly amended the question to “When do you use (product/service)?,” provided sample multiple choice replies, and received a higher level of understanding.
Getting questions right is not a new idea. Indeed, Angus Deaton’s recent Nobel Prize was largely the result of his foundational work on designing household surveys. What’s new is trying to gather rich data over a cellphone. While you can run an effective focus group with a loose guide of topics and you can cover a lot of ground in a 90-minute one-on-one interview, a typical SMS survey is limited to 10 questions and 150 characters per question. These constraints are a powerful pressure-cooker for the questions we ask. We’ve got to make every word and every question count.
So what makes a great question?
For us, a great question is one that is easily and consistently understood by customers. It’s one that makes the complex simple. And it’s one that yields insight around what matters to the customer and the social enterprise trying to serve them.
One of the biggest challenges in impact measurement and international development is understanding not just the breadth but the depth of impact. In Acumen’s case, depth is defined by the degree of change in their well-being a customer experiences from one of our investments’ products or services. For example, we know that a solar light is a better solution than a kerosene lamp, but exactly how much better and why is tricky to figure out. This isn’t an academic exercise for Acumen or our companies. Ultimately, we need to understand our customers’ needs to know where to direct our capital to drive the greatest impact, and without impact data we are simply flying blind.
Because we work across multiple sectors addressing a number of the problems of poverty, our challenge extends beyond just figuring out the quantitative impact of owning a solar light or sending a child to a low-cost private school. Our goal is to go one step further and understand the qualitative difference in value that our customers experience when comparing the various products and services available to them.
Can we really compare the impact of a year of schooling to owning a solar home system? We’re not sure, but we think it’s worth a shot. We believe that trying to understand these comparisons from a customer’s perspective will push us to listen harder and deeper, and it will test the limits of our ability to get rich data through mobile phones.
We asked ourselves if we could create a question or a set of questions that get at this topic directly, helping our customers share what they value most and why.
While a single question to cut through the complexity of our work seemed far-fetched, we knew that similar attempts have been made before. Twelve years ago, Frederick F. Reichheld, Rob Markey and Bain & Company developed the Net Promoter Score® (NPS). According to the Harvard Business Review, the NPS “substitut[ed] a single question for the complex black box of the typical customer satisfaction survey.” Today, it’s become widely adopted by the Fortune 500 as one of the most effective ways to measure customer loyalty. Just as NPS provides companies with a method to effectively judge performance and generate qualitative customer feedback, we wanted to create a single, unifying question to compare social impact.
We started by asking ourselves whether the NPS question — “How likely is it that you would recommend [product/service] to a friend or colleague?” [1–10 scale]” — could serve as a good proxy for how much impact a product had for our customers. We wanted to test this by asking NPS questions together with our depth of impact questions to see if products with a higher NPS also had a higher depth of impact.
We piloted this approach in Kenya and India in two surveys, and the initial results were not as promising as we had hoped.
Despite the proven success of NPS with more affluent, educated customers, the question didn’t seem to perform well with our customers who are typically poor, have limited formal education and little experience with surveys. In follow-up conversations, we heard that the 0–10 scale was hard for them to understand and the hypothetical “would recommend” language didn’t translate well.
Lean Data surveys are short and inexpensive to conduct, so it’s easy to test and refine questions. We experimented with four different versions of the question before landing on a question, inspired by NPS, that seems to perform well: “Have you ever recommended product/service to a friend?” We also played with three different answer scales and arrived at a workable solution. Instead of a 0–10 scale, customers choose between three responses: “Yes, I’ve told many friends;” “Yes I’ve told some friends;” or “No, I have not.”
Once we saw the effectiveness of this question, we wanted to go further, to learn not only whether or not customers recommended a product but also the drivers of meaningfulness of that impact. Drawing on the concept of Constituent Voice developed by Keystone Accountability, we developed a second question, asking customers to respond from “strongly agree” to “strongly disagree” to the statement: “There have been changes in my home because of (product/service).”
In the early tests we’ve run, we’ve seen correlation between reported depth of impact and the strength of agreement to this “meaningfulness” question. For example, owners of solar lights who “strongly agree” with the statement reported an 83 percent reduction in expenditure kerosene, while the customers who said “agree” only reported a 69 percent savings on kerosene. These are just preliminary results, but we’re starting to see that this question might allow us to compare across different interventions, so that customers can tell us what they value the most and why.
While we’re still fine-tuning both of these questions, the progress we’ve made is exciting. Low-income customers are enthusiastic to engage in dialogue, and we are seeing that it’s possible — if you work at it — to develop new questions that capture rich, meaningful data about the wants and preferences of this emerging set of customers. At the end of one of our surveys, one happy customer expressed her satisfaction with the service she received at a health clinic and then added, “I really enjoyed being interviewed.” Clearly, we’re on to something.
While Lean Data is, today, being used mostly by startup social enterprises, our work in learning to ask the right questions over mobile phones is universal. The low-income customer of today is the low middle-income customer of tomorrow. Hundreds of millions of people in the developing world are poised to improve their well-being, but this depends on how well we, as a society, listen to them and adjust our efforts to meet their needs.
So much of this rests on the simple act of caring enough to ask the right questions.
10 years ago, if you wanted to get into the social impact/social enterprise sector, it was enough to say “I think I want to find ways to take a business approach to solving social problems. That makes so much sense!”
5 years ago, if you wanted to do this work you needed to show that you had some direct, relevant experience, a spike of some sort that allowed folks to connect the dots between things that you’ve done and the work you’re proposing to do now.
Today, the expectation is significant direct experience that matters.
If you want to work with social enterprises in the developing world, the expectation is that you’ve spent real time in the developing world doing related work – a couple of years, not a couple of months.
If you want to be a marketer for a great cause, the pool of applicants shooting for that job have been in the great cause marketing business for a while already.
If you want to invest overseas, the expectation is that you have both investing chops and a direct understanding of the markets and businesses you’d like to invest in.
The great news is, unlike 10 years ago, when you had to a make a giant leap, there are countless opportunities for smoother, more gradual transitions.
I’m looking forward to speaking at the Catalyst for Social Change event this coming Thursday, November 12. I’ll be speaking together with Jake Porway, the founder of DataKind and Samuel Sia, one of MIT’s Innovators under 35.
The event is at Fordham Law School at 7pm, and there are still a few seats left – you can get tickets here.
We’ll be talking about innovative approaches to data and measurement, and using them to make the world a better place. It should be a lot of fun.
While I don’t know exactly where the conversation will go, I suspect that if you’re the kind of person who finds this image funny then you’ll have a blast. Hope to see you there.
This post originally appeared on Acumen Ideas, our new channel on Medium.com. If you’re interested in the nitty-gritty of impact measurement, be one of the first to sign up for Acumen’s new Impact Matters e-newsletter that will come out monthly. You’ll get great content at the cutting edge of impact measurement, and we’ll also make sure let you know when our full piece on Lean Data comes out in Stanford Social Innovation Review this winter.
In 2006, Sam Goldman and Ned Tozun set out to eradicate kerosene as a source of light in the developing world. As a Peace Corps volunteer in Benin, Sam witnessed the damage kerosene could do when an overturned lantern created a fire that nearly killed his neighbor’s son. They also saw what a scourge the dirty fuel was for poor, rural families without access to energy, eating up 15 percent of their spending.
Sam and Ned decided to start d.light design, a social enterprise that would solve this problem once and for all. With funding from Acumen and others, d.light set out to create a business providing low-cost solar lanterns to poor customers. Since then, the company has sold tens of millions of solar-powered lights across more than 40 countries.
So is d.light a success? By one measure, absolutely. They are seeing demand for their product and on track to reach 100 million customers by 2020. That’s nearly 10 percent of the more than 1.3 billion people globally without access to electricity. But for entrepreneurs like Sam and Ned — and all of us at Acumen with a mission to make a real dent in poverty — just reaching a large number of people isn’t good enough.
But while it makes us proud to say we’ve helped a million people acquire a reliable solar light or 10,000 women give birth in a high-quality, low-cost hospital, we need more than just big numbers to tell us if we are actually changing people’s lives.
How can we know if we are making a real difference?
Over the last 10 years, impact investing has attracted lots of attention and dollars. Thanks to the success of d.light and other ventures like it, today there are hundreds of impact investors putting their money behind companies that aim to deliver a social and financial return.
Despite this growth, impact investors have done a terrible job of analyzing whether or not these enterprises are creating meaningful social impact.
For example, in June, the Global Impact Investing Network and Cambridge Associates published the Impact Investing Benchmark, the first comprehensive analysis of the performance of impact investors. The report does an outstanding job of analyzing the financial results of impact investing funds, but it says virtually nothing about social performance. That’s a problem.
You’d assume impact investors must be good at measuring social impact. How else could we call ourselves “impact” investors? Not surprisingly, 95 percent of impact investors say they measure impact. But, if you scratch the surface, you’ll discover their definition of impact is mostly limited to big, flashy numbers: number of farmers using an improved kind of seed, number of kids attending school or, as in the case of d.light, number of lights sold.
This is a start, but it’s not good enough.Typical impact investors may know how many farmers a company has reached, but they don’t have a clue if these farmers are better off. They may know how many kids attend schools, but they can’t tell you if the students are from low-income communities or just transplants from the private school down the street. They may know how many households bought a new solar lantern, but they don’t understand if the children in these homes are still dying from kerosene fires.
There’s a good reason impact investors have been falling short : the existing tools for measuring social impact are nearly useless to a social entrepreneur.
These tools, mostly inherited from large-scale, international development organizations, are cumbersome, expensive and typically take a matter of months or even years to produce any real data. For a cash-strapped, resource-constrained social entrepreneur trying to build a fledgling business in tough, emerging markets, these tools don’t make sense.
The good news is, we have an opportunity to change this. Unlike five or 10 years ago, the majority of the 2.5 billion people living in poverty now have access to a cellphone and, in another five years, virtually everyone will be reachable by phone or SMS. At Acumen, we’ve developed a new approach to impact measurement that takes advantage of this shift. Our approach is optimized for entrepreneurs building social enterprises in the developing world, and it capitalizes on today’s information revolution to gather data directly from low-income customers. Our goal is to use this infrastructure to understand our social impact and better serve the poor. We call this approach Lean Data.
Unlike traditional impact measurement, Lean Data is designed to quickly and affordably generate quality customer insights that can immediately drive entrepreneurs’ decisions.
It reframes impact measurement as customer feedback by applying Lean Startup experimentation principles to the collection and use of social impact data. While Lean Startup aims to understand product-market fit with questions like “Do you like this product?” and “Will you buy this product?,” Lean Data goes a step further by working to understand how a purchased product is — or is not — changing a customer’s life.
By asking questions via mobile phones and other existing customer touchpoints (such as a salesperson’s visit to a customer’s home or a company’s call center), Lean Data allows enterprises to get social performance data in a matter of weeks and at a fraction of the cost of traditional measurement approaches.
In the last year, Acumen has helped 12 of our companies measure their social performance by surveying more than 5,000 customers across seven countries. Each of these projects took weeks, not months, and cost thousands, not hundreds of thousands, of dollars.
Lean Data leverages technology, so enterprises can communicate directly with their customers. It is now possible to get reliable, meaningful data directly from low-income customers either through calls or SMS messages. For example, we worked with Ziqitza, a healthcare company that provides low-cost emergency services in India to understand what percentage of its customers in Orissa and Punjab live below the local poverty line. Our results showed that 75 percent of customers live on less than $2.50 a day. In another case, we worked with Juhudi Kilimo, a microfinance enterprise servicing smallholder farmers in Kenya, to measure its social performance using a 10-question SMS survey. The survey showed that the loans Juhudi Kilimo provided to purchase dairy cows are helping farmers see an increase in milk yields of 60 percent.
Lean Data puts the customer first, not the investor. As an investor in social enterprises, Acumen needs impact data to manage its own performance. But we believe social enterprises should first and foremost be accountable “downward” to their customers before worrying about “upward” accountability to their funders. Social enterprises set out to solve meaningful problems for their customers, and they should only systematically collect impact data if that information helps them understand how their products or services are making a difference in their customers’ lives. The information should also be shared “upward” with funders, but that cannot be the primary reason for collecting data.
Lean Data gets underneath not just the “what” but also the “why” of product-market fit. Lean Startup principles focus on product-market fit: is there a demand for a new product in a given customer set? How satisfied are customers with the new product? Social enterprises can take this a step further, asking not just whether there is product-market fit, but why that fit exists. This is the first step towards understanding impact. When we discover why products are purchased, how well or often they are being used, and which problems they solve or fail to solve — like improved productivity, increases in household savings or fewer sick days — we empower customers to articulate what impact means to them. This kind of insight is invaluable to entrepreneurs looking to drive lifetime value, customer loyalty and social impact.
We’ve been developing Lean Data for a little more than a year and, while it is still in its early days, we see huge promise.
If we can give more entrepreneurs like Sam and Ned the right tools to understand their social impact and hear from their customers, they will, for the first time, have actionable data that can tell them, in real time, how to improve their products and create meaningful change.
The truth of this work is that the big, glossy numbers allow us to sing our own praises and raise more money, but they do little to help us improve the lives of the people we aim to serve. It’s time to dig deeper, to use technology to talk directly to our customers, so that our work can realize its full potential.
Yesterday night at bedtime, my son handed me this envelope.
Earlier this year I visited with his Fifth Grade class and had a 90 minute discussion about Acumen. If you ever want to figure out how to explain what you do simply and clearly, I strongly recommend talking to a group of 5th graders. You’ll have to boil everything down to its essence in a great way.
I was blown away by how sophisticated and prepared the kids were: having 5th graders in a nice suburb of New York City ask me how things are going with the rice gassifiers that Husk Power Systems runs in Bihar gave me a lot of hope for the world.
What I didn’t know, until last night, was that a few of my son’s classmates took it upon themselves to raise money for Acumen as a thank you. They self-organized and went around to other classes in the school, talking about Acumen and asking kids to bring in bottles they would return for the 5 cent deposits.
So the $22.05 represents 441 bottles that 5th graders collected and returned because they wanted to help people they’ve never met who have less than they do.
The feeling I had upon receiving this envelope reminded of one of the most powerful impacts that fundraising can have on you. When someone you trust and respect shakes your hand, looks you in the eye, and agrees to give you their hard-earned money to support the work you do…well, I find that moment always motivates me to do the work harder and better, to do right by them and to earn the trust that they are placing in me and in my team.
To know the work that these kids went through – learning about Acumen, courageously going up to their friends and asking them to bring bottles from home, having those kids then go home and explain the same story to their parents – makes me want to work twice as hard and twice as smart.
I hope this story inspires you as much as it’s inspired me.
What would be fabulous is if more kids choose to do a similar project – to collect and turn in 441 bottles to support Acumen or another cause of their choice.
A few years into my job as a fundraiser, one of the things I grew to hate was being sent lists of rich people. What, exactly, was I supposed to do with them? Of course, by definition the people who can give a lot of money are the people who have a lot of money, but that qualifier alone means next to nothing.
As Seth keeps reminding us, our work is to find our tribe, people who share a worldview, and to communicate to them that “people like us do things like this.”
This Wall Street Journal ad nails it for me. While the WSJ no doubt has all the obvious data you’d hope they have – about income levels and geography and demographics and and and – about their readers, the ad boils everything down to:
People who don’t have time to read the Wall Street Journal make time to read the Wall Street Journal.
People like us – busy, successful, top of the heap enough that we now wear multiple hats – believe that the WSJ helps keep us where we are, believe that it is our access to this kind of content allows us to continue to be the thing we are so proud to be.
It’s specific and aspirational for the group that people who identify with that worldview, people for whom that story resonates.
What is your group? What is their worldview? What story do they want to be a part of?