Take the growth approach to evaluating startup nonprofits, not the marginal approach

In its first 2 years, Google made no revenue. Did this indicate it was a bad idea to invest or work there?

We spent the summer in Y Combinator, and one of the main things we learned about is how Y Combinator identifies the best startups. What we learned made me worry that many in the effective altruism community are taking the wrong approach to evaluating startup nonprofits.

In summary, I’ll argue:

1. There’s two broad approaches to assessing projects – the marginal cost-effectiveness approach and the growth approach.
2. The community today often wrongly applies the marginal approach to fast growing startups.
3. This means we’re supporting the wrong projects and not investing enough in growth.

At the end I’ll give some guidelines on how to use the growth approach to evaluate nonprofits.

Two approaches to assessing projects

Suppose you want to identify the startup that will generate the most (time-discounted) profit. The social impact equivalent is to find the project that will generate the largest long-run social impact per dollar invested. What rules of thumb should you use to identify these projects?

Here’s two broad approaches (with the social impact equivalent in brackets):

1. The marginal approach: Which project generated the largest profit (social impact minus costs) per dollar invested over the last year? Can we expect these levels of return to hold up at the margin? If so, invest.

2. The growth approach: Does the project have a high growth rate, large total market (address an important problem), good product (have an effective solution to this problem) and great team?

The growth approach is how Y Combinator evaluates startups, and this seems clearly the right approach in their case. If you’d only invest in startups that had been profitable over the previous year, you would have missed almost all of the biggest winners e.g. Google didn’t make any revenue at all for the first two years; Amazon still isn’t profitable. And this makes sense – if you have the opportunity to be 100x larger, then you should invest all your spare money in growth rather than run a profit. All the value comes from the scenario where you get really big and that’s all you should focus on.

The marginal approach can be applied to companies that are already big and relatively stable, like Coca Cola or a utility company. But it doesn’t make sense to apply to a startup.

Here’s the problem. People in the effective altruism community often evaluate startup nonprofits using the marginal approach rather than the growth approach.

I know I was making this mistake several years ago. What attracted me to working at 80,000 Hours was the high probability of persuading a couple of extra people into high impact careers, and achieving a high short-run leverage ratio. I was vaguely aware the project had high long-term upside – perhaps 10% of all college grads could be users – but didn’t explicitly think much about the possibility of getting really big in ten years time. Over time, however, I realised we should be focusing on the scenario where we get really big. Going through Y Combinator made me more acutely aware of the differences.

I think many donors in the community also mostly think in terms of the marginal approach. Responding to demand, this is why many startups in the community put some kind of short-run leverage ratio at the centre of their fundraising: e.g. Giving What We Can reports pledge donations per dollar, and GiveWell reports money moved against costs. Many discussions about which organisation to donate to focus on (i) the robustness of these multipliers (ii) whether similar levels of return can be projected into the future of not. This was notable during recent discussion about whether to donate to Giving What We Can (GWWC), and I’ve noticed it in many other discussions about where to donate. I’ve seen much less little attention given to which projects have the best growth prospects.

I guess the community has a bias towards the marginal approach because in the past it has put a lot of focus on supporting evidence-backed global health interventions. These interventions have the character that $1 donated leads to some quantified effects, such as malaria nets being given out and QALYs saved. In this case, it makes sense to apply the marginal approach to evaluating the project. The mistake is to then try to apply the same approach to a tiny organisation that’s growing fast. (And to be clear, I’m not saying everyone is making this mistake – there’s plenty who aren’t, including GiveWell themselves – GiveWell broadly applies the marginal approach to their top recommended charities, but wouldn’t evaluate a startup nonprofit using the same criteria, and also originally recommended GiveDirectly partly due to growth potential).

These backward looking ratios are not the best guide to future potential. For instance, if you’d looked at GiveWell’s ratios in its first couple of years, then costs exceeded money moved, especially if you included the opportunity cost of staff time. GiveWell wouldn’t have been “cost-effective” by this measure, and so you wouldn’t have donated, which would have been a mistake. A few years later in 2012, GiveWell had about a 1:10 leverage ratio, so you would have thought it was good, though not exceptional. But again that would have been a mistake: GiveWell was about to partner with Good Ventures, a foundation with thousands of times more money than what GiveWell had moved to that point.

What problems might this be causing?

1) Radically underinvesting in growth. GWWC claims $8 of donations made per $1 of costs, and over $50 of value of future donations. If GWWC is under pressure not to let this ratio drop over the next year, then the maximum GWWC can spend next year is one eighth of the total amount of money GWWC expects to influence in that year. It seems likely that the optimal amount to spend is much higher than that if you want to have the greatest possible long-run impact. In particular, GWWC has little incentive to make investments that will pay off in more than a year, because this only lowers their short-term ratio and could cause their donors to abandon them. But there probably are good opportunities that will pay off in longer than a year.

All of this means we might be investing far less in growing small organisations than we should. (This is particularly true if you also incorrectly assume diminishing returns, which comes naturally with the marginal approach. See this post on why small organisations may have increasing marginal returns). We may also be incorrectly prioritising among organisations with similar ratios, but where one has much more growth potential than another (e.g. some in the community donate to organisations that they believe to have strongly diminishing returns to scale, which the growth approach wouldn’t advise).

2) Starting an overly narrow range of projects. Several of the organisations that have been founded recently seem to be motivated in large part by the possibility of obtaining a provably high cost-effectiveness ratio in the short-term, or providing some immediate value to the community. Charity Science and Raising for Effective Giving are notable examples. These are great projects, of course, and they clearly win over earning to give for the people involved. My worry is that there are whole areas of projects we haven’t even considered that could lead to even higher long-run impact.

3) Not optimising organisations for long-run growth, e.g. by investing in impact evaluation too early. If you’re thinking with the marginal approach, then you’ll want to do a rigorous impact evaluation as soon as possible to prove that you’re (short-run) cost-effective. And many effective altruist organisations do exactly that, sometimes taking evaluation and transparency efforts to extreme levels. Thinking in terms of the growth approach, however, it’s not so obvious this is the right move. Impact evaluation takes up a significant fraction of executive time. When you consider that nonprofit executives also have to spend a lot of time fundraising (unlike for-profit counterparts), a focus on early impact evaluation means executives are left with little time to make sure the programs are actually good and you have a good team. Under the growth approach, it may make sense to delay in-depth impact evaluation until you have more resources and capacity.

Another example is hiring too early. In the growth approach, you spend the first couple of years exploring different products, with the aim of hitting one that “takes off” achieving exponential growth. New hires however, unless exceptionally able and up to speed, aren’t very good at working out what product to make, so slow you down in this stage. Later on, once you achieve ‘product-market fit’, you want to hire as fast as you can. If you’re thinking with the marginal approach, however, then you should hire whenever it won’t decrease your short-run leverage ratio, which normally means hiring a steady stream of a small number of people. In the very early stage of CEA, I think we hired too many people, especially interns, because we were thinking with the marginal approach. There’s other examples too – for instance if your aim is to make an organisation that’s huge in ten years time, then having a good organisational culture is extremely important, which is another reason to hire more slowly early on than you would with the marginal approach.

The combined result of these three mistakes is that we’ll end up with lots of small organisations, but no big wins. And because the most of the impact comes from the big wins (because the distribution of outcomes has a log-normal or power law tail), the community will end up having far less impact than we could have.

It’s not obvious the community is making these mistakes, but the comparative lack of focus on the growth approach makes me concerned we are.

Why favour the marginal approach?

The benefit supporting projects that look good according marginal approach is that you can be confident you’re at least having some impact, whereas it’s harder to make accurate assessments with the growth approach (e.g. it’s easier to be duped into supporting something with zero impact on the promise of long-term benefits). If you’re relatively skeptical, you could argue this is the best we can do.

Taken to the extreme, this view can’t be right as it would preclude properly testing new projects. For example, if you had followed this line of argument, you would have never invested in early GiveWell, and that would have been a mistake.

A more defensible view is that donors with more limited time should focus on the marginal approach (or just give to GiveWell recommended charities), whereas growth donations to startups should be left to “angel” donors, who have the substantial time and expertise required to make good assessments.

So, I don’t think everyone in the community should move to using the growth approach, but I think the balance should move in that direction.

How to evaluate projects with the growth approach?

Here’s some questions to ask if you’d like to evaluate social impact projects using the growth approach¹ (in brackets I’ve written the for-profit equivalent):

1. Does the project have a replicable approach to solving an important problem? (Is the product good?) E.g. with GWWC the question is ‘do they have a repeatable approach for encouraging more people to take the pledge’?

2. Do they have a method to do this at a much larger scale, such that if done, the value created would be much larger than costs? (Do they have a distribution strategy, and if done at scale, would the cost of customer acquisition be lower than the value of a customer?) E.g. can GWWC acquire many more members at less than the value of a pledge? Note that in the nonprofit sector this can also include options like having your intervention taken over by the government, or implemented by other organisations.

3. If the project solved the problem it’s addressing, how good would it be? (market size and growth rate). E.g. if GWWC’s entire addressable market took the pledge, how much impact would that have?

4. Is the cause the project is addressing neglected and tractable? (competitive advantage and timing) E.g. For GWWC, are there lots of other people trying to promote effective giving relative to the size of the problem?

5. Is the project’s impact growing quickly, and is it plausibly exponential? (growth rate) E.g. For GWWC, what’s the growth rate of people taking the pledge? (Bear in mind assessing the growth rate isn’t appropriate at the earliest stages on the project, before you have product-market fit).

6. Is the team altruistic, smart and determined? Do they have a track record? Do they have the relevant skills and experience?

If you want to learn more about the growth approach, check out How to Start a Startup by Y Combinator or the Startup Playbook.