I was speaking to an impact investor recently, and he was saying that investment decisions are ultimately based on an intuitive sense the investor has about the company: the deal, the team, the market opportunity. And shouldn’t we just use our intuition to assess impact?
This is the most common unspoken premise used by impact investors to justify not collecting impact data.
So, where does this intuition come from? And is there such a thing as good and bad impact intuition?
Intuition is subconscious pattern recognition. And patterns are the sum total of the information we’ve taken in. If that information, and our ability to understand and process it, is of high quality, then we develop good intuition. If not, not.
A good investor is awash in quantitative and qualitative data that inform her investment intuition. For example, on the quantitative side, she’ll know what she expected gross margins to be, the predicted length of a company’s working capital cycle, and how many years she forecasted it would take for the company to get to profitability.
But that original financial model will have a very short shelf life: after the investment, she’ll get reams of data to show whether her predictions were right or wrong.
But in the world of impact, she’ll handle things differently.
She’ll look at research and benchmarks to develop a thesis. And she’ll stop there and multiply products sold by those benchmarks [e.g. 10 lights sold * X predicted impact/light = 10x impact].
This is like creating a pre-investment financial model of a company and then, two years later, when asked how the company is performing, using the model’s original variables to answer that question.
Not only would this answer not be any good, but her impact intuition would never improve.
Why do we accept the idea that we can understand the impact we are creating in people’s lives by looking at comparables? Why do we nod when told that it’s hard to get better data (it’s not)? How can we say that “we know impact when we see it” if we don’t gather data to understand actual performance?
The only explanation is this: we are not the people whose lives are, or are not, improved by a given intervention; we are not personally affected by a positive or negative ROI on a “better” solution; and the difference between potential and actual impact doesn’t land on our doorstep, or in our pocketbooks, or in our child’s cough or the quality of the education he receives.
The only way we’ll create better impact intuition is if we apply ourselves seriously to the question of learning what does and doesn’t improve people’s lives.
We don’t settle for “it’s too hard” anywhere else but here.
I had the chance to participate on a panel at SOCAP about the future of impact measurement, and was surprised how challenging I found Karim Harji’s framing question:
Where is impact measurement headed over the next decade? What is it going to take to get there?
After pondering on this question for a while, I ended up at the conclusion that the future is very bright at the level of company-customer interaction.
I say this because in the coming decade, social enterprises, like all companies, will necessarily begin accessing and managing much more customer data gathered remotely through devices. It’s a bit easier to see this future when one is in San Francisco staying at an Air BnB and taking Lyfts everywhere: as mobile phones become both the communications and transaction platforms for nearly everything in everyone’s lives, companies, no matter what their specific business or the customers they serve, will have more data about us. While it’s true that poorer, more remote customers will lag millennials in San Francisco in terms of how soon they get on this conveyor belt, the direction this is heading, for everyone, is clear.
With that in mind, the only question at the company-customer intersection is whether and to what extent companies will incorporate data about social impact into their growing data flow. My thesis is that doing so will be a competitive advantage, allowing companies that move first to better understand how well their products and services are improving their customer’s lives, thereby driving greater loyalty, share of wallet, and share of mind and voice.
I believe this because, as our Acumen impact team has worked with companies on Lean Data projects, it’s become increasingly clear that value creation is impact when you’re dealing with critical goods and services like electricity or education or healthcare: if the customer who buys her first solar lamp stops using kerosene, uses the lamp to keep her business open later at night, and also uses a second lamp for her kids to study at night, then that lamp is creating deep and meaningful value (impact) for her. And all our data show that this same customer is nearly always a net promoter of the company, a source of positive word of mouth, and a high-value and loyal customer.
If this thesis plays out over time, then we’re about to be riding a huge, powerful wave that we’ll simply have to redirect slightly to incorporate thoughtful impact data capture and to drive towards impact management. Soon, even resource-strapped, impact-focused companies in the developing world will have no choice but to gather and utilize more data (including impact data) from end customers if they want to serve these customers effectively.
The question I find harder to answer, interestingly, is: What is going to happen to the capital market for impact? Here, things seem a bit muddier.
In order for capital to increasingly flow towards high-impact opportunities, there has to be some standardization in terms of how impact is measured and communicated, so that an investor looking to compare impact performance can compare opportunity A and B in the same way she compares financial performance for these same two opportunities.
I believe this evolution is a very important one, indeed it might be the most important development that needs to happen if the impact investing marketplace is to realize its full potential. However, unlike the evolution at the company-customer level, it’s less clear to me that there’s forward momentum pushing us in the right direction. It seems possible that we are due for a step-change in terms of how investors deploy capital for impact, and it seems just as possible that five or ten years from now things will be as bespoke and hard to decipher as they are today.
My best guess is that what’s needed to make a shift here is that a handful of highly influential and interconnected players – those holding large amounts of capital that they distribute through a large ecosystem of connected funders – need to establish their own higher, clearer impact measurement standards that they will use to deploy capital, such that their new standards flow all the way down the chain and slowly shift expectations for, and raise the bar for, everyone in the space. This was the role that the U.S. Government played with LEED certification through the GSA, which owns 9,600 buildings in 2,200 communities across the U.S., and I suspect it’s the pattern that needs to play out in impact investing too.
On Monday, Tony Loyd was nice enough to include me in his great series of Social Entrepreneur podcasts. We covered a lot of topics but dug in most deeply on Lean Data, particularly on how we are using it at Acumen to amplify the voice of low-income customers so our entrepreneurs can better serve them. It was a fun conversation.
(if you’re not seeing the embedded link click here)
If this kind of thing is up your alley, you might want to sign up to receive the specialized newsletter we’ve created to share hot-off-the-press insights on what we’re learning through Lean Data. We send it out once every six weeks or so, so it won’t clog your inbox, and it’s full of great stuff.
I spent last week at the annual meeting of the Global Impact Investing Network (GIIN), and I was struck by three trends that could take our sector to the next level.
The first is around taking impact seriously. The second is how different the impact measurement challenge looks depending on where you sit. The third is the acceleration of the rate at which mainstream financial capital is entering our space.
Throughout the GIIN conference, impact — the role it plays in defining our work and how to improve the quality of our impact data — was front and center in a way that I’ve not felt before. For example, one of the first panels kicking off this year’s event was on market segmentation. While segmentation is not a new topic in impact investing, the panel was titled “Market Segmentation through an Impact Lens.” The panelists — from Skopos Impact Fund, Tideline, Athena Capital Management and Omidyar — discussed their research and client-facing efforts to make sense of impact investing from the perspective of impact objectives.
This shouldn’t be brand new, but it is. An orientation to start segmentation with an impact lens runs against the natural tendency to segment investors by asset class or sector strategy, and it’s certainly a far cry from accepting that “intentionality” (as in: my intention is to make such-and-such happen with limited accountability on the data to figure out whether or not real change is happening) is a high-enough bar to set for the sector in terms of impact.
If we could pull off organizing ourselves, as impact investors, by the change we’re trying to make in the world rather than by the investing strategies we’re using to make that happen, that would be a big step forward.
Second, we need much better impact data AND we need to help people who are drowning in too much indecipherable, low-quality data.
I had the chance to participate in two panels focused squarely on advances in impact measurement. What I learned from these panels is that better impact data isn’t enough — there’s a huge desire for simplification too.
At Acumen, our Lean Data work has focused relentlessly on going directly to the low-income customers we aim to serve so we can understand what they have to say. Our objective is to improve the quality of impact data we have by scaling up our capacity to listen to the voices these customers, so we and our investees can better serve them.
While I’m convinced that this kind of listening must to be the foundation of everything we do as a sector, it’s not enough. Listening to my fellow panelists — from Goldman Sachs, Zurich Re, Abraaj Capital and Leapfrog — I heard that big institutions with large, diverse portfolios of impact investments not only desire better impact data but they also need help simplifying and clarifying the reams of impact data they already feel they receive.
Ironically, these large institutions have too much data coming in and most of it’s not very good. Our job is both to improve the strength of the signal and also lessen the noise.
Lastly, it was impossible not to notice that more and more big-name financial players are coming to the table.
The simple fact of having an impact measurement conversation between Acumen and Leapfrog on the one hand (two organizations that are essentially growing startups, with between $100M and $1B in capital under management), and Goldman Sachs, Zurich Re and Abraaj Capital on the other means that there are innovations in impact management happening across the spectrum of impact capital. That’s hugely positive.
Then, at the end of the day, we got to hear Former Governor Deval Patrick and Deborah Winshel discuss the impact investing strategies they began implementing in the last year at Bain Capital and Blackrock. Both articulated their goals to fully integrate impact into the global practices of these uber-blue chip firms, firms that collectively represent more than $4.5 trillion in assets. While it’s early in the journey for both Bain and Blackrock, it’s clear that their actions could have a huge influence with other mainstream financial players and beyond.
As I left the conference and made my way back to New York, I was struck with the feeling that we are entering a new phase in our sector. Having passed through the teething pains of our early days and our loud, sometimes impulsive childhood, we’re ready to start growing up a bit. This means harnessing — rather than just shouting about — the increased momentum building in our space, thanks to the entrance of major new players, while also taking a much more sober and serious look at the ultimate goal of this work, which is to make a real, large-scale and lasting difference in the well-being of people and the planet.
If, in this next chapter, we can find a way to have impact investing go deeper on impact and bigger in terms of scale and reach, we will truly be in a position to take this work to the next level.
[Note: you can also follow the conversation about this post on Medium]
Tomorrow, Wednesday, February 17th at 12 noon Eastern, I’m helping run a Twitter chat that Acumen is hosting to talk about Lean Data and measuring social performance. It’s all about the finding the next frontier in impact measurement, in a discussion with Acumen, Omidyar, Stanford Social Innovation Review, the Aspen Network for Development Entrepreneurs and Root Capital.
Every day, more than 5 million new cellphones are sold. That’s more than 10 times the number of babies born each day. We are barreling towards a world where a cellphone will be in every pocket by 2020, and a smartphone in every pocket soon after that.
This revolution is making the unimaginable real— in the near future, we will have the opportunity to start a dialogue with literally every person on the planet. This new two-way conversation, where everyone participates, will pull billions of people into the mainstream by connecting them with one another.
Since starting this work in 2014, one of the most important lessons we’ve learned is that a cellphone in every pocket is just a starting point. The art of every Lean Data project is in the questions we ask. Ask the wrong questions, and you get back little of value. Ask the right ones, and you can move from data to information to actionable insights.
Great questions connect with customers and give them an opportunity to share their voice. But crafting a great question is no easy task. The slightest shifts in word choice can affect understanding; the smallest differences in intonation alter perceptions of sincerity. All of these nuances can bias the data and diminish its value.
For example, in trying to understand the usage of solar home systems in Kenya, we started with the question, “How often are you currently using (product/service)?” After testing this question over SMS, we received feedback suggesting we omit the word “often” and make the question more simple and direct. We quickly amended the question to “When do you use (product/service)?,” provided sample multiple choice replies, and received a higher level of understanding.
Getting questions right is not a new idea. Indeed, Angus Deaton’s recent Nobel Prize was largely the result of his foundational work on designing household surveys. What’s new is trying to gather rich data over a cellphone. While you can run an effective focus group with a loose guide of topics and you can cover a lot of ground in a 90-minute one-on-one interview, a typical SMS survey is limited to 10 questions and 150 characters per question. These constraints are a powerful pressure-cooker for the questions we ask. We’ve got to make every word and every question count.
So what makes a great question?
For us, a great question is one that is easily and consistently understood by customers. It’s one that makes the complex simple. And it’s one that yields insight around what matters to the customer and the social enterprise trying to serve them.
One of the biggest challenges in impact measurement and international development is understanding not just the breadth but the depth of impact. In Acumen’s case, depth is defined by the degree of change in their well-being a customer experiences from one of our investments’ products or services. For example, we know that a solar light is a better solution than a kerosene lamp, but exactly how much better and why is tricky to figure out. This isn’t an academic exercise for Acumen or our companies. Ultimately, we need to understand our customers’ needs to know where to direct our capital to drive the greatest impact, and without impact data we are simply flying blind.
Because we work across multiple sectors addressing a number of the problems of poverty, our challenge extends beyond just figuring out the quantitative impact of owning a solar light or sending a child to a low-cost private school. Our goal is to go one step further and understand the qualitative difference in value that our customers experience when comparing the various products and services available to them.
Can we really compare the impact of a year of schooling to owning a solar home system? We’re not sure, but we think it’s worth a shot. We believe that trying to understand these comparisons from a customer’s perspective will push us to listen harder and deeper, and it will test the limits of our ability to get rich data through mobile phones.
We asked ourselves if we could create a question or a set of questions that get at this topic directly, helping our customers share what they value most and why.
While a single question to cut through the complexity of our work seemed far-fetched, we knew that similar attempts have been made before. Twelve years ago, Frederick F. Reichheld, Rob Markey and Bain & Company developed the Net Promoter Score® (NPS). According to the Harvard Business Review, the NPS “substitut[ed] a single question for the complex black box of the typical customer satisfaction survey.” Today, it’s become widely adopted by the Fortune 500 as one of the most effective ways to measure customer loyalty. Just as NPS provides companies with a method to effectively judge performance and generate qualitative customer feedback, we wanted to create a single, unifying question to compare social impact.
We started by asking ourselves whether the NPS question — “How likely is it that you would recommend [product/service] to a friend or colleague?” [1–10 scale]” — could serve as a good proxy for how much impact a product had for our customers. We wanted to test this by asking NPS questions together with our depth of impact questions to see if products with a higher NPS also had a higher depth of impact.
We piloted this approach in Kenya and India in two surveys, and the initial results were not as promising as we had hoped.
Despite the proven success of NPS with more affluent, educated customers, the question didn’t seem to perform well with our customers who are typically poor, have limited formal education and little experience with surveys. In follow-up conversations, we heard that the 0–10 scale was hard for them to understand and the hypothetical “would recommend” language didn’t translate well.
Lean Data surveys are short and inexpensive to conduct, so it’s easy to test and refine questions. We experimented with four different versions of the question before landing on a question, inspired by NPS, that seems to perform well: “Have you ever recommended product/service to a friend?” We also played with three different answer scales and arrived at a workable solution. Instead of a 0–10 scale, customers choose between three responses: “Yes, I’ve told many friends;” “Yes I’ve told some friends;” or “No, I have not.”
Once we saw the effectiveness of this question, we wanted to go further, to learn not only whether or not customers recommended a product but also the drivers of meaningfulness of that impact. Drawing on the concept of Constituent Voice developed by Keystone Accountability, we developed a second question, asking customers to respond from “strongly agree” to “strongly disagree” to the statement: “There have been changes in my home because of (product/service).”
In the early tests we’ve run, we’ve seen correlation between reported depth of impact and the strength of agreement to this “meaningfulness” question. For example, owners of solar lights who “strongly agree” with the statement reported an 83 percent reduction in expenditure kerosene, while the customers who said “agree” only reported a 69 percent savings on kerosene. These are just preliminary results, but we’re starting to see that this question might allow us to compare across different interventions, so that customers can tell us what they value the most and why.
While we’re still fine-tuning both of these questions, the progress we’ve made is exciting. Low-income customers are enthusiastic to engage in dialogue, and we are seeing that it’s possible — if you work at it — to develop new questions that capture rich, meaningful data about the wants and preferences of this emerging set of customers. At the end of one of our surveys, one happy customer expressed her satisfaction with the service she received at a health clinic and then added, “I really enjoyed being interviewed.” Clearly, we’re on to something.
While Lean Data is, today, being used mostly by startup social enterprises, our work in learning to ask the right questions over mobile phones is universal. The low-income customer of today is the low middle-income customer of tomorrow. Hundreds of millions of people in the developing world are poised to improve their well-being, but this depends on how well we, as a society, listen to them and adjust our efforts to meet their needs.
So much of this rests on the simple act of caring enough to ask the right questions.