The New Measures for Improving Nonprofit Performance
In this era of scarce economic resources, the pressure on nonprofit managers to show quantifiable results is greater than ever. Alnoor S. Ebrahim and philanthropist Mario Morino discuss the differences and similarities between performance measurement in the for-profit and nonprofit sectors.
For-profit businesses have a common goal: create value for owners or shareholders by creating value for customers.
It's a focus that must seem enviably straightforward from the perspective of nonprofit organizations and social enterprises obliged to navigate a path under the watchful eye of multiple stakeholders, including funders, boards, and clients—all while staying true to a core mission and values.
In today's climate of scarce economic resources, the pressure for nonprofits to show quantifiable results is greater than ever; as a result, an organization without a strong sense of strategic direction and the internal data to understand its own strengths and weaknesses can be overly influenced by outside demands for metrics that may not always be relevant to its ultimate success.
"Without understanding outcomes, you can't get at the issue of what works and what doesn't."
How can nonprofit leaders address such conflicting demands? Harvard Business School Working Knowledge recently spoke with two leaders in the field of nonprofit performance management. Mario Morino is cofounder and chairman of Venture Philanthropy Partners, which strategically invests money and expertise to improve the lives of children and youth of low-income families in the Washington, D.C., region. Morino is the author of Leap of Reason: Managing to Outcomes in an Era of Scarcity.
Alnoor S. Ebrahim is an associate professor in the General Management Unit and the Social Enterprise Initiative at Harvard Business School. He has published extensively on the challenges of accountability and performance management facing nonprofit organizations, including the award-winning book NGOs and Organizational Change: Discourse, Reporting, and Learning and the edited volume Global Accountabilities: Participation, Pluralism, and Public Ethics. Ebrahim teaches the required MBA course Leadership and Organizational Behavior, and chairs the Executive Education program Governing for Nonprofit Excellence.
Q: Mario, tell us about Venture Philanthropy Partners (VPP) and how it contrasts with your past experience in the for-profit sector?
Mario Morino: My background was in the IT industry, working in systems management, which is fundamentally about tracking the performance and integrity of large computing platforms. What I experienced back in the 1970s and '80s was very similar to what I'm seeing today in the nonprofit sector. Back then, our clients were being asked to make significant investments in IT, so of course they asked, what's my return? As a result I got very engaged in how one collects and presents information to show how the IT function was being productive and what value it was creating for the corporation—which was actually very abstract and difficult to do.
I left the for-profit sector in 1992 and spent some time involved with different nonprofits, getting a contextual view of what was possible. That led to the creation of VPP in 2000. My belief was that if a grant or investment in a nonprofit was handled strategically it would yield a greater impact for the people served. In an organization with compelling leadership, vision, and evidence of outcomes that were making a material difference in the lives of those served, our view was that we could come in and make a substantial investment of $3 million to $5 million and work extensively with the management team as strategic advisors to help them build that organization. And in fact, some of our investment partners have made significant strides—real changes to the services provided to those they serve, in both quantity and quality. Performance data was always at the heart of those efforts, but it wasn't a leading factor.
Q: What is your primary focus?
Morino: Performance really starts with the leadership of the organization. I didn't set out to write a book, but in 2009 I sat in three different meetings with leading philanthropists in the country, discussing the issue of leadership in nonprofit organizations. But there wasn't a single nonprofit executive in those meetings, and that ticked me off. So I wrote an article, which led to my book, Leap of Reason.
Alnoor Ebrahim: I'd like to pick up on a couple of your comments, Mario, that I believe are critical to emphasize. Going back to your point about the differences and similarities of performance measurement in the for-profit and nonprofit sectors, there are three things that really stand out.
The first is that we tend to think of for-profit organizations as being focused on tangible hard metrics like profit or market capitalization. But for-profits also do a lot of organizational capacity-building work. Venture capital investments are very much about building the organization, by making investments in human and systems capacities that then lead to results.
Ironically, that kind of organization building is pretty rare in the nonprofit sector. That's what VPP and other organizations like it are bringing to the table. For the past 20 to 40 years there's been a lot of talk about capacity building in nonprofits, but to actually figure out how to build and support organizations is something we're learning how to do through the for-profit world. It's absolutely critical to have that in place first—if you don't, it's hard to figure out what are the important things to measure. So that's number one: the integral relationship between capacity building and performance management.
Morino: The term "capacity building" is sometimes confused with the implementation of technical systems. But it's really about building the organization as a whole. We lose sight of that.
Ebrahim: That's right. And it needs to be supported by the second challenge, which is aligning the interests of different stakeholders. When you think about a business, ideally the customers are paying for a product that generates revenue for the firm, which ultimately generates value in the form of profit or some sort of return for the owners of the firm (if it's privately held) or for shareholders (if it's public). The interests of the customers, the firm, and the owners become aligned by the customer actually paying for the product.
In the nonprofit sector you rarely have this kind of alignment because the people who pay for the work of the nonprofit—foundations, governments, donors—are usually different from the people benefiting from the work of the nonprofit. There are some exceptions in situations where fees are paid for services. But generally speaking a nonprofit CEO or board deals with a whole set of stakeholders with very different interests. The challenge then becomes enabling this alignment, which requires the CEO to prioritize based on the mission of the organization.
In practice, we find that many leaders have trouble prioritizing because they're leaning toward the interests of the donors. Funders may have their own legitimate interests, but if the CEO finds that those are in tension with the interests of the organization or the beneficiary, then you have a real problem. And in this economy, people are unlikely to turn away funding when it comes their way. You do have instances of enlightened funders that are willing to work with the organization as it's building its capacity and figuring out its accountability priorities. But that's still relatively rare.
The third issue is the difference between financial and social metrics. We have a system of generally agreed-upon financial metrics for businesses, such as profit and revenues. Those are generally backward-looking measures. Then there are share prices and market capitalization, which are forward-looking measures that anticipate the value of a firm. There may be some debate over what is important at a particular stage of an organization's life cycle, but people speak the same language for the most part. In the social sector we don't have nearly that degree of agreement over what constitutes useful, common metrics, even in subsectors like health care, education, and the environment.
At the moment there's a lot of debate about developing a common set of metrics, so that if people are going to invest in the sector there's some way to measure that return in social terms. It's a tough struggle that a lot of experienced and devoted people are trying to crack right now.
Q: There seems to be an increasing need for social services at a time when government at all levels is cutting programs. How will this affect the evolution of performance management?
Morino: Relatively speaking, there's been remarkable progress in the nonprofit sector in the area of performance management, but the potential for change is still enormous. The challenge is that we can't possibly solve an organization's problems at higher levels without the organization looking at outcomes internally. Without understanding outcomes, you can't get at the issue of what works and what doesn't.
The hammer is coming in the economy—it's going to force a major disruption. What we hope is that smart decisions will be made as to where available money is allocated. But the impact will be severe and cause a lot of change in the sector.
Ebrahim: The conversation about results and performance management is occurring globally, not just here in the United States. The US Agency for International Development, the Millennium Challenge Corporation, and their counterparts in other OECD countries are all involved. Several years ago the various ministries of developed and developing countries got together to identify how to improve the performance and quality of international aid—resulting in the Paris Declaration on Aid Effectiveness. The fourth such global meeting was just held in early December in Busan, South Korea.
"We need to encourage risk-taking by organizations, provided that it's accompanied with rigorous data collection."
In this particular economy the natural tendency of governments is to cut back on foreign or development aid, which creates a similar pressure to what we're seeing in the United States. The emphasis on results is far stronger. There's a shift toward trying to get countries (in the case of aid) or organizations (in the case of foundation support) to build up their theories of change and their organizational capacities. That, I think, is a positive development. But it's coming at a time when need is even greater and resources are diminishing. It's really a double-edged sword.
Q: It sounds like it's hard to say how it's all going to play out.
Morino: Having lived in the for-profit and nonprofit world, when the hammer does hit, you only have a few ways of getting through it. One of them is making smart decisions. When everything hits, all the sacred cows disappear.
In one particular experience I had with a nonprofit, a factor that worked in our favor was that we had developed multiyear financial forecast models. We were dealing with bank workout teams, which is very rugged. But because we had that information, we stunned them. They didn't expect that coming from a nonprofit—that gave us time and credibility.
In addition, we have to pay attention to the fact that there are real differences in the ecosystems of each sector.
Ebrahim: Mario raises a crucial distinction. In the for-profit world, we tend to see individual firms as the central unit of analysis. Firms make profit, and investors buy equity in firms. But in the social sector, the results are not at the firm level but in a community or society. To really see impact, we have to ratchet it up a notch and talk about entire ecosystems—collections of organizations working on a common set of problems. We're not going to be able to address problems of poverty, education, health care, and the environment at the organizational level of analysis unless we figure out a way to measure collective performance when multiple actors are involved. But funders tend to want to reward individual organizations. We need new ways of collaboratively figuring out what is a theory of change when you have 100 organizations working on an interconnected set of issues.
Morino: I'm afraid there's also a tremendous reluctance within the sector to deal with performance results in an open, honest, and transparent way. Funders don't want to show that things aren't working, and nonprofits don't want to show that their program may not be what people think it is. It's the culture. We use this information to penalize people, when instead we should be thinking about how we can make things better. It's a huge mental and cultural shift that's going to have to take place in the sector.
Q: Can you each describe organizations that are effective in managing to outcomes? What are they doing differently?
Ebrahim: I have two examples that are rather different from each other.
The Acumen Fund is an intermediary funder that seeks support from foundations and investors and channels this support to for-profit social enterprises, in the form of a debt or equity relationship. The idea is to recover a portion of that investment, at least 1x or 2x, and cycle it back into supporting other organizations. What I find interesting about Acumen, and this is a point of considerable debate in the sector, is that the metrics it uses to track its investments are largely output oriented, not outcome oriented.
As an example, Acumen has invested in a Kenyan business that builds toilet and shower facilities in central business districts as well as in slums, and it is measuring the number of customer visits to those facilities. That doesn't tell you anything about the social impact—it doesn't tell you whether people's health has improved as a result, or whether environmental and sanitation conditions have improved, though Acumen might decide to measure that somewhere down the road.
Acumen's argument is that it has a theory of change for this intervention, based on existing research that suggests urban sanitation improves environmental and human health. But it has made a conscious decision that measuring such longer-term outcomes is beyond the scale of its investment at the moment. It would stretch the entrepreneur in a direction that, at least at this stage, would not be helpful in building the business.
What I like about this approach is that it's very sensitive to building the organization. So you're not applying 50 metrics; you're using 10 or 15. It's very much focused on the learning of the entrepreneur and building the business, but it's controversial because some people say Acumen should be measuring outcomes. Ultimately, it does want a sense of those outcomes—but building the organization is the high priority now.
The second is the Millennium Challenge Corporation [MCC], a US government agency that supports economic development in various countries. Its measures are typically very long-term outcomes and impacts: for example, increases in farmer incomes. This is a fairly simple, clear, and measurable impact. The hard part is not the measurement, but the strategy for how you actually get to those results. For example, the MCC made a $550 million grant to the government of Ghana that was focused on infrastructure such as irrigation and roads that could get farm goods to market, as well as agricultural credit and supplies to increase productivity. The MCC was interested in supporting government policies that would help establish this agricultural market, as well as in figuring out what it could do to bring those value-added goods to world markets.
The MCC doesn't have any illusion that its grant alone can increase farmer incomes. It is interested in working this into the broader goals of the host country to fit with the government's development priorities. What MCC has in common with Acumen is that it's using the grant as leverage to build the capacity of government agencies to implement long-term change. Ultimately it's the outcome measure (increasing farmer incomes) that becomes the "True North" that holds everyone's feet to the fire. At some point you need to check as to whether farmer incomes actually increased as a result of this huge investment. MCC's time horizon is around 20 years for these expected benefits.
Q: Mario, what has your experience been in working with nonprofits through VPP?
Morino: Our approach at VPP is to work with organizations to figure out what every executive needs right now to run their nonprofit and have an impact. I think that represents a fundamental shift for this sector. We're simply trying to ask organizations what they need to run effectively and course-correct when needed. It's that basic.
One of our portfolio organizations, the Latin American Youth Center, had to course-correct when it found that a new domestic violence prevention program was actually changing attitudes in the wrong direction. The only reason LAYC knew this was because it collected information on participants' attitudes both before and after the program. It was very transparent of LAYC to admit this, and it was able to make changes (in this case, moving from a mixed-gender to a single-sex class environment) that brought positive, statistically significant change to every cohort.
In another case, the Friendship Public Charter School, a group of four traditional public schools and six public charter schools in the Washington, D.C.-Baltimore area, worked with consultants, teachers, and administrators to design a performance-management system that was truly effective in making information accessible and actionable. The data was easily understandable and useful for students and parents—everyone was involved, and as result the schools saw real, positive results by working smarter, not just harder.
What we're talking about is very real, very doable. Roca, a Boston-based organization that reaches out to disengaged, high-risk youth, has undergone a fundamental shift in its culture thanks to the leadership of its executive director, Molly Baldwin. In addition, it has moved to a single intervention model that has led to greater focus and the ability to construct meaningful performance objectives and standards across the agency. Roca has been using performance-tracking and client profile data to ensure optimal service delivery and optimal youth/caseworker matches. It has also been figuring out methods to orient new staff more quickly and thoroughly, and how to optimize caseworker compliance with data-gathering goals and objectives. The fact that Roca is even aware of sophisticated issues like the impact of consistency and service delivery on program outcomes is a testament to how far advanced it is, and it shows that these things can be done.
Q: How is a transparent, collaborative culture cultivated in an organization?
Morino: It starts with leadership that is very outreaching. It starts with getting the right people in the right seats in the organization. You have to model the behavior that you want the organization to adopt. And you have to ground it in a set of values and guiding principles, which I think many people tend to skip over. Without a moral or ethical framework, it's very difficult to build a culture. Having clarity of vision is essential to building a culture. People need to know what's expected of them, and it's important to give feedback in a constructive way. Using information is essential to a good culture because it allows people to get engaged, to investigate, to explore, to be curious, to learn, and to do continuous improvement, both for the individuals and for the programs. It's a process. It's slow.
Ebrahim: I couldn't agree more. For a nonprofit leader, it's risky to reveal that your work is not having the impact you desire, particularly in this economy. If we can flip that attitude so that the funding environment is receptive to disclosure about failure and risk—where it's actually encouraged, because it can foster creative problem-solving—then we will have made progress. We need to encourage risk-taking by organizations, provided that it's accompanied with rigorous data collection, so we can figure out what's working and what isn't.
On Mario's insight about leadership, I would add that this is also a concern at the board level. Many nonprofit boards see their work as simply looking after the financial integrity of the organization and following the letter of the law. All of that is important, but an equal if not greater priority is actually focusing on the mission of the organization and making sure there's a strategy that can be implemented and measured in terms of performance targets. You need an environment that encourages lively debate, where the CEO feels comfortable coming to the board and saying, here's a problem I've been having, can you help me figure it out?
Morino: When we make investments at VPP we work with the organization to set milestones for that year: outputs, related outcomes, and accomplishments. The tension always arises when a target is missed. Some foundations walk away at that moment in time. Our belief is that you can miss a target—that's life. We try to get into how are you going to get by it, how will you make it better going forward? We reset expectations for the following year. But I think that's an important step to make in the funder-grantee relationship, to have that openness and integrity in the relationship so that you can do that. If you can't agree on what's going to happen going forward, you can back out. But the point is that you don't walk away because a target is missed. You use it as an opportunity to say, how can we help you get better in that area?
One of the biggest challenges I've found is that not enough attention is paid to rewarding and encouraging management. In the private sector, you're rewarded significantly for being a good executive manager. Your perks are tied to that. In the nonprofit world, you don't see that same recognition or appreciation.
We tend to "knight" managers rather than develop people in roles. I think that's a huge cultural difference in the sectors. The issue is particularly pronounced in foundations.
Q: Give us a sense of new developments in the field of performance-management measurement. Alnoor, can you bring us up to speed on what's going on in that area?
Ebrahim: Sure. First, there's been quite a lot of development in terms of methods not just for performance measurement, but also for linking those methods to strategy and management more generally. We've had the classic tools in place for a long time—logic models and the balanced scorecard, for example. But in international development we've seen an increasing critique of those methods on the basis that sometimes they are too linear and don't capture the complex, total ecosystem challenges that we were just discussing. There's been a lot of conversation lately about using systems thinking and complexity theory to help us understand the challenges at the ecosystem level.
There's also growing attention to using randomized control trials in the social sector, which we tend to associate with drug trials. The Jameel Poverty Action Lab at MIT has been very active in that area. The real question is, when are these different methods appropriate, and under what circumstances?
In international development there's been a lot of experimentation that hasn't quite made its way to the United States yet. A Canadian agency called the International Development Research Center has developed a tool called "outcome mapping" that could potentially be useful for measuring ecosystem outcomes. It involves bringing together a diverse set of stakeholders that are working on a similar problem to map out their different roles and various theories of change. It's about relationship building and using the wisdom of the crowds, if you will, to begin to identify critical pathways for change.
There's also been growth in "constituency feedback" tools, both in the United States and internationally. One of the big conversations is about what are the appropriate measurement approaches for enabling learning and capacity building inside an organization, as compared to evaluative methods that are used after the fact.
What Mario and I are concerned about is building up the learning-based approaches that support taking risk, looking at failure, analyzing data, and then doing various midcourse corrections. Ultimately, you need both evaluative and learning-based methods.
Finally, there have been some efforts to get intermediary funders to agree on a set of metrics for the performance of their grantees that are then aggregated in order to develop an asset class, such as the Impact Reporting & Investment Standards (IRIS). And social impact bonds that use private investment to support government and nonprofit projects are now coming from the United Kingdom to the United States—Massachusetts might be one of the first states to try to use them. I'm in the process of writing a case on them.
Morino: The Achilles' heel comes back to the question of what information the organization has about its performance. The information starts with what the business creates about its own performance. We have that turned around in the nonprofit sector.
Ebrahim: This is a really critical point. Developments like social impact bonds are important for attracting private investment to proven programs, while reducing costs to government and society. But you're right that this still doesn't address the question of what information will help nonprofits or social enterprises improve their work, regardless of what is happening at the funder level.
That's not to say that nonprofits don't have a core sense of purpose. Many do. Many are strongly committed, are driven by their missions, and have a highly devoted and committed staff. But ultimately that needs to be extended to the willingness to look critically at what you're doing well and what you're doing poorly and then trying to improve it.
Q: I'm curious to know more about your interest in this topic. Do you have a personal connection to it?
Morino: All of VPP's work has been around three words: youth, learning, community. That's been my passion point. When you go into work in those environments there are remarkably good people there. But the question in my mind is always, how can we get better? A culture of continuous improvement to help the kids we're serving is what drives me.
Q: Am I right in thinking that you grew up in a poorer environment?
Morino: My family was blue collar, but we were never poor. There's a distinction between being in deep poverty and living in a low-income world. We had food and clothing. And the world was different in the 1950s. Even a tough neighborhood still had a social fabric to it. Today the social fabric is gone. The economic base for that income level has been almost fundamentally wiped out. And there's a prevalence of guns and drugs. Those three things together create a deadly cycle for a community.
When I grew up I had all kinds of people encouraging me, helping me get through things. That's what's missing in communities today. In metrics, we don't track the existence of caring relationships with adults in a student's life. Yet it's the biggest reason a kid succeeds in school. I'm trying to apply the analytical world to the passion of how do you help people, how do you make a difference in their lives.
Q: How about you, Alnoor?
Ebrahim: I was born in Tanzania. Prior to that, my family's roots are in India, where we still have many relatives in rural communities. We were well off when we were living in East Africa, but when we left during a period of political turmoil in the 1970s, we came to Canada as immigrants with very little. But like Mario says, economic poverty is not the same as social poverty. I never felt disadvantaged because I had a supportive family and community, and access to education through the public school system in Canada. So there are personal, historical ties that inform this work for me.
At an intellectual level this work on social performance is stimulating because the challenges are both difficult and deeply meaningful. And it's particularly useful to do this work at a business school where there's a lot of insight and research on what it takes to create and lead high-performing organizations.