Quanta Corporate Citizenship The dangers of using off-the-shelf charity ranks
| Quanta Corporate Citizenship |
It has become very fashionable over the last few years to use ranking agencies. For full disclosure, I confess that this is how I first became involved in the sector. Alongside a group of partners from Oxford and Cambridge, we tried to develop a rating mechanism for environmental charities. Although receiving some attention from the media, my conclusion after a lot of energy spent on it, was that we were trying to create a smokescreen that would eventually cause more damage than good.
Using standardised rankings is appealing. They provide clear-cut answers to difficult questions and take away the headache of having to think for yourself. Besides, they eliminate the hard job of gathering data. Contacting hundreds, if not thousands, of charities and social enterprises and then understanding and later tabulating the data is cumbersome. It demands time and money. But the dangers offset the benefits of using those standardised formats.
First, many standard rankings available from different sources fail to differentiate input from output. Because measuring input is easier than measuring output, ranking/rating agencies have an incentive to focus on the former, although it should be focusing on the later. Worse, these inputs reflect what the agencies believe is ideal, but are often based on nothing truly testable. In other words, they are unable to correlate the input and the output. Furthermore, even if there were ways of linking input and output, they are unable to recognise which are the relevant inputs to which outputs, because of the number of inputs and the diversity of NGOs (as no two NGOs do exactly the same work).
Secondly, they provide a one dimensional view of the world. From good to bad. From high to low, failing to take into account that different donors have different needs, that different objectives need different set-ups, and different models generate different outcomes. In the same way that some clients will choose a restaurant based on affordability, others on accessibility, others on style of food, others on calorie count and others on taste. It simply makes no sense to compare a three star Michelin restaurant in Paris with a cheap fast food outlet down the road, in the same that it makes no sense to try to fit all charities and social enterprises in the same scale.
Furthermore, even if they were able to provide some sense of comparison in terms of output, how could the different types of output be comparable? How can one elephant saved in Kenya be compared with one child vaccinated in Thailand and one school receiving a computer in London? What does it really mean? Without inserting the donor’s strategic objectives into the equation, it simply makes no sense. And once the donor’s objectives are inserted, the standardised ranks can no longer hold, as the donor will have very specific needs.
This brings us to the problem of using predefined metrics. Several of the proxies used by rating agencies to assess the value of the outcome generated by the investees are corporate proxies that didn't work even for the corporate world (otherwise Enron, WorldCom and Lehman would still be around, and the likes of Google and Facebook would not exist).
Also, as Michael Edwards, from Duke University, states, “rankings imply attribution, which is impossible in the world of social change because results are never driven by one project or organization acting on its own, so who exactly is being rewarded, and is that fair?”[1] In other words, how to say that specific social or environmental achievements were the results of the efforts of one particular organisation or project if the problems they try to solve are generally the result of several different factors tackled by several different organisations and projects in many different ways?
There is also a clear tendency to confuse charities and social enterprises and try to use the same tools to evaluate both, although they are fundamentally different models. Even within the same group, different organisations will have different objectives, donors and investors, and different ways of tackling their problems. Trying to compare the outcomes of an organisation working to change the law about discrimination against minorities with another organisation trying to deal with the consequence of that discrimination, with a third organisation trying to provide legal assistance for those suffering the discrimination is very difficult, if not impossible. They are all dealing with the same problem: discriminatory behaviours, but while the outcomes of the first organisation – changing the law – might take years to be realised and will probably not be able to be directly attributable to only the efforts of that particular organisation, the outcomes of the second organisation is much easier to measure if we define success for this organisation as the number of calls received. If, however, we define success as the emotional comfort for all those who knew the organisation was there for them at the end of the phone, even if they never needed it, then it once more becomes very difficult to measure. On the other hand, our third organisation can clearly measure its success by the number and percentage of cases won. How impactful it is compared to the first organisation that is trying to avoid the problem happening, and the second one, that is covering a much broader scope? If we do not understand the donor’s specific strategic needs and characteristics, it becomes impossible to determine which one, if any, offers the best potential fit.
The table below, extracted from a well-known charity rating organisation, shows a variation of the same problem: the top 4 organisations – the only ones receiving 3 stars – are organisations that, due to the nature of their work, produce quantifiable outputs, whilst other organisations that focus on more qualitative things (e.g., advocacy, behaviour changing, policy making, awareness raising etc), that might be equally as important, are unable to compete in the same level playing field.
Furthermore, because ranking agencies need to somehow use quantitative elements to compare a large number of different organisations operating in many different fields and places, there is a recurring use of standardised metrics that generate a myriad of negative incentives to potential charity participators.
Using standardised rankings is appealing. They provide clear-cut answers to difficult questions and take away the headache of having to think for yourself. Besides, they eliminate the hard job of gathering data. Contacting hundreds, if not thousands, of charities and social enterprises and then understanding and later tabulating the data is cumbersome. It demands time and money. But the dangers offset the benefits of using those standardised formats.
First, many standard rankings available from different sources fail to differentiate input from output. Because measuring input is easier than measuring output, ranking/rating agencies have an incentive to focus on the former, although it should be focusing on the later. Worse, these inputs reflect what the agencies believe is ideal, but are often based on nothing truly testable. In other words, they are unable to correlate the input and the output. Furthermore, even if there were ways of linking input and output, they are unable to recognise which are the relevant inputs to which outputs, because of the number of inputs and the diversity of NGOs (as no two NGOs do exactly the same work).
Secondly, they provide a one dimensional view of the world. From good to bad. From high to low, failing to take into account that different donors have different needs, that different objectives need different set-ups, and different models generate different outcomes. In the same way that some clients will choose a restaurant based on affordability, others on accessibility, others on style of food, others on calorie count and others on taste. It simply makes no sense to compare a three star Michelin restaurant in Paris with a cheap fast food outlet down the road, in the same that it makes no sense to try to fit all charities and social enterprises in the same scale.
Furthermore, even if they were able to provide some sense of comparison in terms of output, how could the different types of output be comparable? How can one elephant saved in Kenya be compared with one child vaccinated in Thailand and one school receiving a computer in London? What does it really mean? Without inserting the donor’s strategic objectives into the equation, it simply makes no sense. And once the donor’s objectives are inserted, the standardised ranks can no longer hold, as the donor will have very specific needs.
This brings us to the problem of using predefined metrics. Several of the proxies used by rating agencies to assess the value of the outcome generated by the investees are corporate proxies that didn't work even for the corporate world (otherwise Enron, WorldCom and Lehman would still be around, and the likes of Google and Facebook would not exist).
Also, as Michael Edwards, from Duke University, states, “rankings imply attribution, which is impossible in the world of social change because results are never driven by one project or organization acting on its own, so who exactly is being rewarded, and is that fair?”[1] In other words, how to say that specific social or environmental achievements were the results of the efforts of one particular organisation or project if the problems they try to solve are generally the result of several different factors tackled by several different organisations and projects in many different ways?
There is also a clear tendency to confuse charities and social enterprises and try to use the same tools to evaluate both, although they are fundamentally different models. Even within the same group, different organisations will have different objectives, donors and investors, and different ways of tackling their problems. Trying to compare the outcomes of an organisation working to change the law about discrimination against minorities with another organisation trying to deal with the consequence of that discrimination, with a third organisation trying to provide legal assistance for those suffering the discrimination is very difficult, if not impossible. They are all dealing with the same problem: discriminatory behaviours, but while the outcomes of the first organisation – changing the law – might take years to be realised and will probably not be able to be directly attributable to only the efforts of that particular organisation, the outcomes of the second organisation is much easier to measure if we define success for this organisation as the number of calls received. If, however, we define success as the emotional comfort for all those who knew the organisation was there for them at the end of the phone, even if they never needed it, then it once more becomes very difficult to measure. On the other hand, our third organisation can clearly measure its success by the number and percentage of cases won. How impactful it is compared to the first organisation that is trying to avoid the problem happening, and the second one, that is covering a much broader scope? If we do not understand the donor’s specific strategic needs and characteristics, it becomes impossible to determine which one, if any, offers the best potential fit.
The table below, extracted from a well-known charity rating organisation, shows a variation of the same problem: the top 4 organisations – the only ones receiving 3 stars – are organisations that, due to the nature of their work, produce quantifiable outputs, whilst other organisations that focus on more qualitative things (e.g., advocacy, behaviour changing, policy making, awareness raising etc), that might be equally as important, are unable to compete in the same level playing field.
Furthermore, because ranking agencies need to somehow use quantitative elements to compare a large number of different organisations operating in many different fields and places, there is a recurring use of standardised metrics that generate a myriad of negative incentives to potential charity participators.
First, ranking agencies demand time from these investees to prepare, collect and provide the data requested. That would be a potentially doable task and would have minimum impact on the charities work if not for the fact that there is not only one rating agency out there. There are tens, if not hundreds of them, all using different sets of metrics. That is not just distracting, but also confusing to the thousands of charities and social entrepreneurs who do not want to risk staying out of any of those ranks because they know how risky it is to not be seen there by funders.
Moreover, the ranks end up setting an incentive for an NGO to deviate from its objectives: to efficiently deliver high-quality and relevant services to their beneficiaries, and instead focus on building its organisation around the metrics and reports demanded by the rating agencies. By trying to solve a common problem, the rating agencies end up creating a second one – distracting the NGOs – and failing to solve the first problem, by creating a negative incentive for the NGOs to play the assessment game.
Not long ago business schools faced the same problem: there were so many different business publications trying to assess them at the same time, so many that several schools felt overwhelmed. Worst, a few started playing the rating game, focusing on what mattered to the assessors and not what mattered to the students. The simple volatility in the ranking of some of those schools year on year clearly shows how some of them have mastered the art of manipulating the system. If business schools, with their well-off beneficiaries and relatively stable finances felt the pressure of the assessment game and decided to play with it, we can all imagine the consequence for cash-starving charities.
In the same way that some business schools started focusing on ways to improve their ranking by playing with their selection criteria for professors and students, which was later captured by the rating agencies, instead of focusing on improving the quality and affordability of their courses, and on helping students achieve their personal aims, many NGOs have started racing to metrics and not to impact. In addition, creating this race to standardise metrics, the rating game leaves outside all those without a track record that could right now be building the game changing organisations of tomorrow. All because they cannot show results yet or because the results they can show, or that are relevant to them, cannot be captured using the standardised metrics developed by the rating agencies.
Another two problems linked to this conundrum are the hundreds of thousands of NGOs who are left outside the rank because they cannot afford – financially or time-wise - to get distracted with the production of those metrics, but nevertheless are delivering great work. In some cases they may be completely unaware that such assessment is even being performed because they are too focused on delivering results to their beneficiaries.
And then there is the question of how reliable the data presented by the NGOs is. If the data is presented and produced by the NGOs, the assessment agency needs to audit it and that has a cost that is inevitably paid directly by the NGOs. (i.e., they will have to pay the fees themselves) or indirectly (i.e., donors will be asked to pay the fees and therefore even less money reaches the NGOs’ beneficiaries).
But the worst problem by far is that it completely eliminates the amazing innovation possibilities that the sector is renowned for. Some of the most effective work I've seen was by organisations that wouldn't stand a chance of checking a single box on standard templates. That does not mean they are not delivering, it means that the rating agencies haven't put their fingers on what makes them outperform their peer group.
The idea of developing an assessment mechanism that helps society, donors, governments and partners to distinguish the sheep from the wolves in one of the largest sectors on the planet is extremely attractive and that is why so many rating agencies have a go at. It is why so many organisations desperate for a measurement tool will jump on the first boat they see without (a) understanding and (b) questioning the adequacy of that tool for their purpose.
Organisations and individuals aiming to make philanthropic investments need to avoid using standardised assessment models that deliver on the wishful thinking of rating agencies but do not reflect the reality of the NGOs true outputs. To ignore this reality is to risk the diversion of good money to underperforming charities at the cost of great funding opportunities in charities that simply do not fit standardised frameworks.
Assessing potential investees and comparing them is extremely important; but first and foremost, an organisation willing to take a strategic approach to its philanthropic endeavours needs to understand all of its own objectives, its own “personality” and culture and its own requirements. Then comes the opportunity to design a meaningful selection process for fund recipients that takes all of these important peculiarities into account.
Moreover, the ranks end up setting an incentive for an NGO to deviate from its objectives: to efficiently deliver high-quality and relevant services to their beneficiaries, and instead focus on building its organisation around the metrics and reports demanded by the rating agencies. By trying to solve a common problem, the rating agencies end up creating a second one – distracting the NGOs – and failing to solve the first problem, by creating a negative incentive for the NGOs to play the assessment game.
Not long ago business schools faced the same problem: there were so many different business publications trying to assess them at the same time, so many that several schools felt overwhelmed. Worst, a few started playing the rating game, focusing on what mattered to the assessors and not what mattered to the students. The simple volatility in the ranking of some of those schools year on year clearly shows how some of them have mastered the art of manipulating the system. If business schools, with their well-off beneficiaries and relatively stable finances felt the pressure of the assessment game and decided to play with it, we can all imagine the consequence for cash-starving charities.
In the same way that some business schools started focusing on ways to improve their ranking by playing with their selection criteria for professors and students, which was later captured by the rating agencies, instead of focusing on improving the quality and affordability of their courses, and on helping students achieve their personal aims, many NGOs have started racing to metrics and not to impact. In addition, creating this race to standardise metrics, the rating game leaves outside all those without a track record that could right now be building the game changing organisations of tomorrow. All because they cannot show results yet or because the results they can show, or that are relevant to them, cannot be captured using the standardised metrics developed by the rating agencies.
Another two problems linked to this conundrum are the hundreds of thousands of NGOs who are left outside the rank because they cannot afford – financially or time-wise - to get distracted with the production of those metrics, but nevertheless are delivering great work. In some cases they may be completely unaware that such assessment is even being performed because they are too focused on delivering results to their beneficiaries.
And then there is the question of how reliable the data presented by the NGOs is. If the data is presented and produced by the NGOs, the assessment agency needs to audit it and that has a cost that is inevitably paid directly by the NGOs. (i.e., they will have to pay the fees themselves) or indirectly (i.e., donors will be asked to pay the fees and therefore even less money reaches the NGOs’ beneficiaries).
But the worst problem by far is that it completely eliminates the amazing innovation possibilities that the sector is renowned for. Some of the most effective work I've seen was by organisations that wouldn't stand a chance of checking a single box on standard templates. That does not mean they are not delivering, it means that the rating agencies haven't put their fingers on what makes them outperform their peer group.
The idea of developing an assessment mechanism that helps society, donors, governments and partners to distinguish the sheep from the wolves in one of the largest sectors on the planet is extremely attractive and that is why so many rating agencies have a go at. It is why so many organisations desperate for a measurement tool will jump on the first boat they see without (a) understanding and (b) questioning the adequacy of that tool for their purpose.
Organisations and individuals aiming to make philanthropic investments need to avoid using standardised assessment models that deliver on the wishful thinking of rating agencies but do not reflect the reality of the NGOs true outputs. To ignore this reality is to risk the diversion of good money to underperforming charities at the cost of great funding opportunities in charities that simply do not fit standardised frameworks.
Assessing potential investees and comparing them is extremely important; but first and foremost, an organisation willing to take a strategic approach to its philanthropic endeavours needs to understand all of its own objectives, its own “personality” and culture and its own requirements. Then comes the opportunity to design a meaningful selection process for fund recipients that takes all of these important peculiarities into account.
The author, Gus Romano, is a partner and co-founder of Quanta Corporate Citizenship
