Today we present an ongoing debate between Chris and Johannes over the past couple of weeks. It’s become more intense and fun since Martin Ravallion, Director of the World Bank's Development Research Group, put out a Working Paper highlighting the various difficulties, shortcomings and flaws of a growing industry of "mash-up" development indexes. Whereas Chris is pleased someone has voiced his own concerns about the value of these indexes in a systematic and rigorous way, Johannes is much less convinced about the overall negative assessment of composite indicators. He feels criticism of the many development-related composite indicators ignores their main strength, which is to put otherwise ignored development issues on the table for discussion by policy makers, rather than relegating them to academic discussions in peer-reviewed journals.
Witness the following imaginative coffee talk between them both:
JOHANNES: Hey Chris, have you seen this?! Finally, somebody – and not just anybody, but the head of research at the World Bank –put down on paper some of your reservations about all these increasingly popular composite indicators, such as the Multidimensional Poverty Index, the World Gender Gap Index, the Human Development Index, etc – you must be quite pleased, aren’t you?
CHRIS: Well, Johannes, as you're an admitted fan of these "mash-up" indexes, I am surprised to hear about this from you, but sure, I think it’s not only great but also timely. UNDP just launched last week its HDR report using Foster and Alkire’s new methodology for measuring international poverty and yes, I must say, I have problems with it. For sure, development doesn't depend on economic growth alone, so a single measure like GDP isn't sufficient to measure progress. But what use are development indexes or composite indicators at all, like the UN's Human Development Index , when we can’t even keep track of development progress along a few commonly agreed dimensions of progress, such as the eight goals that make up the internationally agreed upon Millennium Development Goals?
JOHANNES: Well, Chris, good point. I think we agree that measuring development is a messy business. As you know there are literally hundreds of measures used by international organizations, such as the UN, the World Bank, and the OECD to compare and benchmark countries along different dimensions of development. For example, the MDGs try to boil down progress on development into 8 simple measurable objectives. Eight goals seem easy enough to handle, right? But, if you dig a little deeper into the official UN documentation on the MDGs, you realize that the 8 goals are actually only the tip of the iceberg. Each goal is associated with up to six related targets. Each of these targets then is associated with up to seven related indicators. All in all, that means that to measure development using the MDGs, you need to keep track of eight goals, 21 targets, and 60 indicators! See link
Try explaining that to the average person in the street-- let alone, to a president or prime minister!
Thus the need for composite indicators. They allow different dimensions of development to be combined into one measure that is simple to understand and explain. And what's more and often forgot, is that these development indexes aren't made to please ivory tower Harvard PhD researchers; instead their main aim is to alert policy makers to important issues. Countries are ranked according to these indexes, which kicks off a healthy debate about issues that otherwise would not even be noticed. The media likes them so much that you can even get them interested in such boring topics as the time it takes to open up a business! Do you know that probably the most influential "mash-up" index in recent memory was IFC's Doing Business index? It's had real success in getting countries to pay attention to the concerns and importance of small businesses.
CHRIS: Ugh! Don't me remind me of the dark ages! Actually, I am surprised that you defend the Doing Business indicators so much as they clearly show how ideology can drive the composition of these indexes. In 2008, for example, the World Bank’s own Independent Evaluation Group criticized the indicators as being overly biased towards deregulation and having no statistically significant relationship with economic growth and development. I suspect this is why you can sometimes draw bizarre conclusions from these indexes, like the fact that according to the ease of starting a business indicator, Liberia ranks higher than Germany , where many would argue that the dynamism of its small business sector has helped make the country one of the strongest exporters worldwide.
JOHANNES: Okay, Chris, good point. Paying too much attention to the index ranking can miss the point. And ranking countries can be harmful if they focus debate on the ranking, rather than on the issue itself. What is really important is why the data makes one country score better than another in a given dimension. But the value of ranking as an advocacy tool shouldn't be underestimated. Countries don't like to see themselves ranked low on an index or composite indicator. Often it can be perceived as bad for business and investment in a country. In this way, index rankings can help policymakers focus on an issue of concern and try to build political will for reform to improve their country's ranking. This is where composite indicators can be particularly valuable by combining related measures so that an important issue, like, for example, gender inequality, is tackled holistically by policymakers and is confronted as a cross-cutting problem.
CHRIS: But doesn't it all depend on how the data is combined in a composite measure? Aggregation of only loosely related components in these indexes and the use of arbitrary weighting can make these indexes troublesome as well – you risk adding up apples, oranges, and cherries. If, for example, your index is based on the average of GDP per capita, educational attainment, and life expectancy, you are assuming that lack of economic growth can be compensated for by increases in education or increases in life expectancy. Not everyone would agree on such strong assumptions! How a composite indicator combines and weights its constituent parts is an important issue. Ideally, the weighting and combination of various indicators should be grounded in a theory of how the various parts work together. Put simply, you should only combine apples and oranges if you are interested in fruits as a whole! Establishing how a composite indicator should be weighted is therefore open to debate in a large number of cases (e.g., some people think fruit as whole is interesting, while others only find discussion of citrus fruits to be valuable!).
JOHANNES: Interestingly enough, there is increasing openness to discussion of the weightings and assumptions underlying composite indicators. For example, the Human Development Report 2010, has introduced new methodologies for examining the holistic nature of human development and poverty through it's new IHDI and MPI measures. One way to address these concerns is actually related to the word "mash-up" used in the title of Ravallion's working paper , which was borrowed from the web 2.0 jargon.
How to use today's technology to permit this greater openness is demonstrated by a web application used on the OECD’s Social Insitution and Gender Index’s “Build My Own Gender Index” website. Rather than rely solely on the weighting provided by the authors of the index, the site allows users to both drop and add as well as change components of the index and observe how the rankings change in real time. Users can also drill down into the underlying data for each country and see the qualitative research that informed the quantitative data.
CHRIS: I totally agree about the value of these online tools for making indexes more transparent, Johannes. The latest developments in these technologies will be the focus of a joint seminar organized by the OECD, World Bank and Statistics South Africa in Cape Town in December. Although the seminar is called "Turning statistics into knowledge", I think it will also be a good chance to discuss turning knowledge into "implementation and action”!
CG and JJ
No comments:
Post a Comment