Why College Rankings Keep Deans Awake at Night

Can parents and prospective students trust college rankings? Bill Kirby unpacks this complex system, including what “world-class” actually means, what rankings don’t take into account, and how schools are learning to game an imperfect system.

Subscribe on iTunes  Follow on Libsyn

Subscribe on iTunes  Follow on Libsyn

Podcast Transcript

Brian Kenny: In September of 2013, President Obama sent shock waves through American higher education by announcing a plan for a federal rating system that would allow parents and students to easily compare colleges. He said he would urge Congress to pass legislation to link student aid to the rating system. The president abandoned the idea after many college presidents complained that there is no simple way to reliably rate and rank institutions of higher learning given their vast differences.

That reality, however, hasn't slowed the rising tide of annual college rankings from all corners of the world, rating schools in every category imaginable. Today we'll hear from Professor Bill Kirby about his case study entitled "World-class" Universities: Rankings and Reputation in Higher Education.

Bill Kirby is a historian who examines contemporary China's business, economic and political development in an international context. In addition to many books and articles, he's written over 40 cases on China. I guess this counts as one of those, Bill. Thanks for joining me.

Bill Kirby: It's great to be here, Brian. Thanks for asking.

Brian Kenny: I found this case to be very, very interesting. I think many of our listeners have probably referred to those rankings at some point either for themselves or for their kids so I think they'll be able to relate to this as well. Put us in the context for this case. The protagonist it sounds like might be China itself?

Kirby: Well since my area is China, it's China in part. But I'm writing a book looking at the future of universities and what country or what national system is going to lead universities in the world of higher education in the 21st century. Is it going to be Europe, whose great institutions really defined what a modern university would be in the 19th century? The United States, which is without question still at the moment the dominant player in research universities? Or is it China, a place into which more resources are going than any other place in the world, and a place with more extraordinary human capital than any other place in the world, and a place (maybe because it's a communist country) that is more obsessed with hierarchy and rankings than any place I've ever been?

Kenny: When people see the case, they'll see that “world-class” is in quotations. Why did you decide to do that?

Kirby: Because no one knows what it is. Everyone wants to be “world-class” (no one has yet gone for the entire solar system), so we're just content to be successful on this planet at the moment. It is since every dean, every provost, every president particularly outside of the United States has an aspiration, in a university more than just a college, an aspiration to be ranked among global leaders. They are taken seriously in every corner of the world. I was giving a lecture couple of years ago in Vienna and the dean of this faculty in Vienna (actually it was the president of the University of Vienna) was there and the Shanghai Jiao Tong University rankings of global universities had just come out and he was really upset. Vienna was number 65. How can this be? I thought to myself how can it be that the president of the University of Vienna cares so much what the Shanghai Jiao Tong University says about his university, but he does. These rankings have come to define what individual institutions do to get ahead.

Kenny: For some schools it maybe an exaggeration to say they live and die by this, but they really do pay close attention to these.

Kirby: They do pay close attention to them. I think for good and bad reasons. There is a lot that we can gain by looking at rankings or looking at kind of movements, broad movements in rankings. We also understand that this system is extraordinarily new. The US News and World Report, which everyone knows in the United States, started its college rankings in 1983 because it was a failing news magazine that needed another business to stay in print. That's why they did it and they suddenly got taken seriously even if it's purely reputational rankings to begin with. Now they have global rankings. The Shanghai Jiao Tong, the Times of London system, the QS systems, these are the three dominant global rankings that everyone takes very seriously in higher education around the world. They are no more than 10 to 15 years old but they have, with enormous speed, gained traction in how people think about universities.

Kenny: This is not an exact science. I alluded to that in the introduction, and every institution is different. Can you dig into the rankings a little bit and talk about what the key elements are that they look at on the global rankings?

Kirby: There are two basic fundamental systems. One is by reputation and one is by output. What output do you measure? Usually today it is primarily research output in a certain number of publications, almost exclusively in the English language and largely in science and engineering and applied science but in social science as well. Sometimes it ranks education to the degree that it can on faculty-student ratios. Some of them take a very specific number of journals and these are the only ones that they take seriously for ranking. Shanghai Jiao Tong is probably in some sense the most objective in that regard. The one thing that none of them rank well, which is something that parents care about enormously, is teaching. Teaching and education, inspiration, mentorship—not a single one of them has a real system for trying to figure out who gets educated. You can have a great research faculty but how do they learn, how do the students learn, how do the students come to know them? This is entirely missing in this system.

Kenny: That's a big gap. Particularly as we think about the student experience. The reputational piece applies directly to consumer point of view, right? If you're looking for a college you want to know that your child is going to be well cared for there, that they're going to have a good experience. The students themselves are thinking about, “I want to go to a place that's nice and comfortable and people are good to me and the food is good.” You've got a lot of rankings out there that are looking at seemingly pretty unimportant things like the taste of the food but maybe to that end consumer that's a really important element.

Kirby: There are two types of consumers in education. There are the students, who tend to know not too much about the product that they're about to acquire or be part of and, much more important, the parents because they're the ones who ultimately get to choose. I remember telling my own children, "You can apply to any college you want. These are the ones I'll pay for."

Kenny: There is some leverage in that.

Kirby: There is a little bit of leverage. Or at least “I'll help pay for." The lack of comparative information helps these rankings gain traction also because it gives you a sense of what are deemed to be the best liberal arts colleges, if you really want to go to a small college. What are the best places if you want to be pre-med or if you're thinking of going to law school or if you simply have no idea what you're doing? What are the places that per capita turn out some of the extraordinary talent? It's very interesting, one would think if you really wanted to go on to a PhD in biology that you might go to a place that's attached to a major medical school. In fact, a higher percentage of individuals go on to doctorates in the life sciences from places like Williams, Carlton, Middlebury—great liberal arts colleges where people get hands-on experience with scientists and with labs and who have a close working relationship with their faculty.

Kenny: Does that show up in the rankings? Does that somehow make itself apparent?

Kirby: No, because usually Williams, Middlebury these places are ranked as liberal arts colleges and they're not in the rankings of universities.

Kenny: Can you game the system? Is there a way to acquire the pieces that you need to elevate yourself in the rankings?

Kirby: Well you would know this better than me, maybe. The very sad thing that I see happening (and these rankings should take this into account and discount these statistics) is selectivity. For US News and World Report, for example, historically one of their criterion has been: how selective is the college? Do you take 20 percent of everybody who applies, do you take 50 percent of everybody who applies, or in the case of Harvard College less than 5% of those who apply? One of the sad things is that it is led many colleges to, how should we put it, “gin up” the number of applicants—getting many more people to apply who would otherwise not have applied so that they appear to be more selective, and including people who are not necessarily qualified or for whom it's simply not the right fit or the right place. I find that really very sad. They're more selective but it adds unnecessarily and almost cruelly to the anxiety of parents and students who are now told, “You need to apply to 10 to 15 places,” which I think is a huge mistake. Ten is more than enough for anybody, but you need to apply to all of these places because all of them are getting in many more applications than quite frankly they need or deserve.

Kenny: Right, so you've taught this case in class before?

Kirby: I've taught it this term to my MBAs and to my undergraduates.

Kenny: I'm curious about what their reaction was, since many of them probably are fresh off using one of these guide books?

Kirby: That's right. I asked them, "How did they choose their own college?" The good news is that their parents and their guidance counselors do pay some attention to these rankings but on the whole students much less so. They pay attention much more to reputation, to geography, and to where X or Y has gone that they know. The students looked at it and they looked very critically at the methodology of the rankings and particularly the MBAs understood how flawed these are or how idiosyncratic they are one by one by one. They serve, I have to say, different purposes. It's not by accident that the British universities tend to fair rather well in the Times of London polls, the American universities overwhelmingly well in the US News and World Report, by contrast probably the most heartless one of all is the Shanghai Jioa Tong ranking. Chinese universities Tsinghua and Beijing University, these are in the top 20, 30 or 40 depending on the rankings in the other rankings, but they are 150th and beyond in the one that is based purely on publication in certain journals.

Kenny: Do you see that shifting over time? If we were to look at the turn of the next century do you think all of a sudden the Chinese Universities would be displacing some of the universities that are commonly at the top of these rankings?

Kirby: I do. I think without question there will be one or two Chinese universities soon in the top 20, and in most of these rankings in another 10 to 15 years, two in the top 10, if the level of resources and if the kind of policies that I see being enacted in Chinese universities allow for this to happen. The biggest challenge though is not money, that's a huge challenge anywhere. It's not talent. A place like China has extraordinary talent in every dimension. It's governance. Who makes the decision of what you invest in for research? Who hires the faculty? How do you bring in the best students? How do you make sure that the faculty and students interact and that the faculty actually teach the students once they're there? How do you give the faculty the academic freedom and the intellectual freedom? What the Germans called in the 19th century “lehrfreiheit,” the freedom to teach, and for the students “lernefreheit,” the freedom to learn. This is at the moment what is sadly constrained at least in the humanities and social sciences in China on any matter political.

The other thing I will say if you look at European universities and particularly at German universities, which really led the world until the beginning of the twentieth century, they're trying to lead again. They are re-inventing themselves again. These rankings have come as a kind of shock to that system and so they have initiated what is called an “excellence initiative” in which enormous amounts of money are going to fund truly innovative efforts of teaching and learning across the major German universities. The French are trying to do the same. The European universities are not standing still, they're growing very strong.

The place that I worry about the most, mostly because I'm here, is the United States. I worry less about places like Harvard and Yale and the big private universities. I worry that we too will begin to decline if the great public systems of this country decline. One of the things that I talk about in another case, a case on the University of California, Berkeley—this is arguably the greatest public university in the United States, perhaps in the world, but it has suffered enormously in terms of budgetary and other pressures over the last decade and more and longer than that. If it declines as it possibly may in a significant way, then places like Harvard will decline too. We compete with them for the same graduate students. For the same faculty. We're part of a highly competitive environment in this country, globally now as well. If such a great system declines it hurts all of us. When one thinks of these rankings, we should pay as national policy much greater attention than we do to the state of our great public universities.

 Read more

Podcast Transcript

Brian Kenny: In September of 2013, President Obama sent shock waves through American higher education by announcing a plan for a federal rating system that would allow parents and students to easily compare colleges. He said he would urge Congress to pass legislation to link student aid to the rating system. The president abandoned the idea after many college presidents complained that there is no simple way to reliably rate and rank institutions of higher learning given their vast differences.

That reality, however, hasn't slowed the rising tide of annual college rankings from all corners of the world, rating schools in every category imaginable. Today we'll hear from Professor Bill Kirby about his case study entitled "World-class" Universities: Rankings and Reputation in Higher Education.

Bill Kirby is a historian who examines contemporary China's business, economic and political development in an international context. In addition to many books and articles, he's written over 40 cases on China. I guess this counts as one of those, Bill. Thanks for joining me.

Bill Kirby: It's great to be here, Brian. Thanks for asking.

Brian Kenny: I found this case to be very, very interesting. I think many of our listeners have probably referred to those rankings at some point either for themselves or for their kids so I think they'll be able to relate to this as well. Put us in the context for this case. The protagonist it sounds like might be China itself?

Kirby: Well since my area is China, it's China in part. But I'm writing a book looking at the future of universities and what country or what national system is going to lead universities in the world of higher education in the 21st century. Is it going to be Europe, whose great institutions really defined what a modern university would be in the 19th century? The United States, which is without question still at the moment the dominant player in research universities? Or is it China, a place into which more resources are going than any other place in the world, and a place with more extraordinary human capital than any other place in the world, and a place (maybe because it's a communist country) that is more obsessed with hierarchy and rankings than any place I've ever been?

Kenny: When people see the case, they'll see that “world-class” is in quotations. Why did you decide to do that?

Kirby: Because no one knows what it is. Everyone wants to be “world-class” (no one has yet gone for the entire solar system), so we're just content to be successful on this planet at the moment. It is since every dean, every provost, every president particularly outside of the United States has an aspiration, in a university more than just a college, an aspiration to be ranked among global leaders. They are taken seriously in every corner of the world. I was giving a lecture couple of years ago in Vienna and the dean of this faculty in Vienna (actually it was the president of the University of Vienna) was there and the Shanghai Jiao Tong University rankings of global universities had just come out and he was really upset. Vienna was number 65. How can this be? I thought to myself how can it be that the president of the University of Vienna cares so much what the Shanghai Jiao Tong University says about his university, but he does. These rankings have come to define what individual institutions do to get ahead.

Kenny: For some schools it maybe an exaggeration to say they live and die by this, but they really do pay close attention to these.

Kirby: They do pay close attention to them. I think for good and bad reasons. There is a lot that we can gain by looking at rankings or looking at kind of movements, broad movements in rankings. We also understand that this system is extraordinarily new. The US News and World Report, which everyone knows in the United States, started its college rankings in 1983 because it was a failing news magazine that needed another business to stay in print. That's why they did it and they suddenly got taken seriously even if it's purely reputational rankings to begin with. Now they have global rankings. The Shanghai Jiao Tong, the Times of London system, the QS systems, these are the three dominant global rankings that everyone takes very seriously in higher education around the world. They are no more than 10 to 15 years old but they have, with enormous speed, gained traction in how people think about universities.

Kenny: This is not an exact science. I alluded to that in the introduction, and every institution is different. Can you dig into the rankings a little bit and talk about what the key elements are that they look at on the global rankings?

Kirby: There are two basic fundamental systems. One is by reputation and one is by output. What output do you measure? Usually today it is primarily research output in a certain number of publications, almost exclusively in the English language and largely in science and engineering and applied science but in social science as well. Sometimes it ranks education to the degree that it can on faculty-student ratios. Some of them take a very specific number of journals and these are the only ones that they take seriously for ranking. Shanghai Jiao Tong is probably in some sense the most objective in that regard. The one thing that none of them rank well, which is something that parents care about enormously, is teaching. Teaching and education, inspiration, mentorship—not a single one of them has a real system for trying to figure out who gets educated. You can have a great research faculty but how do they learn, how do the students learn, how do the students come to know them? This is entirely missing in this system.

Kenny: That's a big gap. Particularly as we think about the student experience. The reputational piece applies directly to consumer point of view, right? If you're looking for a college you want to know that your child is going to be well cared for there, that they're going to have a good experience. The students themselves are thinking about, “I want to go to a place that's nice and comfortable and people are good to me and the food is good.” You've got a lot of rankings out there that are looking at seemingly pretty unimportant things like the taste of the food but maybe to that end consumer that's a really important element.

Kirby: There are two types of consumers in education. There are the students, who tend to know not too much about the product that they're about to acquire or be part of and, much more important, the parents because they're the ones who ultimately get to choose. I remember telling my own children, "You can apply to any college you want. These are the ones I'll pay for."

Kenny: There is some leverage in that.

Kirby: There is a little bit of leverage. Or at least “I'll help pay for." The lack of comparative information helps these rankings gain traction also because it gives you a sense of what are deemed to be the best liberal arts colleges, if you really want to go to a small college. What are the best places if you want to be pre-med or if you're thinking of going to law school or if you simply have no idea what you're doing? What are the places that per capita turn out some of the extraordinary talent? It's very interesting, one would think if you really wanted to go on to a PhD in biology that you might go to a place that's attached to a major medical school. In fact, a higher percentage of individuals go on to doctorates in the life sciences from places like Williams, Carlton, Middlebury—great liberal arts colleges where people get hands-on experience with scientists and with labs and who have a close working relationship with their faculty.

Kenny: Does that show up in the rankings? Does that somehow make itself apparent?

Kirby: No, because usually Williams, Middlebury these places are ranked as liberal arts colleges and they're not in the rankings of universities.

Kenny: Can you game the system? Is there a way to acquire the pieces that you need to elevate yourself in the rankings?

Kirby: Well you would know this better than me, maybe. The very sad thing that I see happening (and these rankings should take this into account and discount these statistics) is selectivity. For US News and World Report, for example, historically one of their criterion has been: how selective is the college? Do you take 20 percent of everybody who applies, do you take 50 percent of everybody who applies, or in the case of Harvard College less than 5% of those who apply? One of the sad things is that it is led many colleges to, how should we put it, “gin up” the number of applicants—getting many more people to apply who would otherwise not have applied so that they appear to be more selective, and including people who are not necessarily qualified or for whom it's simply not the right fit or the right place. I find that really very sad. They're more selective but it adds unnecessarily and almost cruelly to the anxiety of parents and students who are now told, “You need to apply to 10 to 15 places,” which I think is a huge mistake. Ten is more than enough for anybody, but you need to apply to all of these places because all of them are getting in many more applications than quite frankly they need or deserve.

Kenny: Right, so you've taught this case in class before?

Kirby: I've taught it this term to my MBAs and to my undergraduates.

Kenny: I'm curious about what their reaction was, since many of them probably are fresh off using one of these guide books?

Kirby: That's right. I asked them, "How did they choose their own college?" The good news is that their parents and their guidance counselors do pay some attention to these rankings but on the whole students much less so. They pay attention much more to reputation, to geography, and to where X or Y has gone that they know. The students looked at it and they looked very critically at the methodology of the rankings and particularly the MBAs understood how flawed these are or how idiosyncratic they are one by one by one. They serve, I have to say, different purposes. It's not by accident that the British universities tend to fair rather well in the Times of London polls, the American universities overwhelmingly well in the US News and World Report, by contrast probably the most heartless one of all is the Shanghai Jioa Tong ranking. Chinese universities Tsinghua and Beijing University, these are in the top 20, 30 or 40 depending on the rankings in the other rankings, but they are 150th and beyond in the one that is based purely on publication in certain journals.

Kenny: Do you see that shifting over time? If we were to look at the turn of the next century do you think all of a sudden the Chinese Universities would be displacing some of the universities that are commonly at the top of these rankings?

Kirby: I do. I think without question there will be one or two Chinese universities soon in the top 20, and in most of these rankings in another 10 to 15 years, two in the top 10, if the level of resources and if the kind of policies that I see being enacted in Chinese universities allow for this to happen. The biggest challenge though is not money, that's a huge challenge anywhere. It's not talent. A place like China has extraordinary talent in every dimension. It's governance. Who makes the decision of what you invest in for research? Who hires the faculty? How do you bring in the best students? How do you make sure that the faculty and students interact and that the faculty actually teach the students once they're there? How do you give the faculty the academic freedom and the intellectual freedom? What the Germans called in the 19th century “lehrfreiheit,” the freedom to teach, and for the students “lernefreheit,” the freedom to learn. This is at the moment what is sadly constrained at least in the humanities and social sciences in China on any matter political.

The other thing I will say if you look at European universities and particularly at German universities, which really led the world until the beginning of the twentieth century, they're trying to lead again. They are re-inventing themselves again. These rankings have come as a kind of shock to that system and so they have initiated what is called an “excellence initiative” in which enormous amounts of money are going to fund truly innovative efforts of teaching and learning across the major German universities. The French are trying to do the same. The European universities are not standing still, they're growing very strong.

The place that I worry about the most, mostly because I'm here, is the United States. I worry less about places like Harvard and Yale and the big private universities. I worry that we too will begin to decline if the great public systems of this country decline. One of the things that I talk about in another case, a case on the University of California, Berkeley—this is arguably the greatest public university in the United States, perhaps in the world, but it has suffered enormously in terms of budgetary and other pressures over the last decade and more and longer than that. If it declines as it possibly may in a significant way, then places like Harvard will decline too. We compete with them for the same graduate students. For the same faculty. We're part of a highly competitive environment in this country, globally now as well. If such a great system declines it hurts all of us. When one thinks of these rankings, we should pay as national policy much greater attention than we do to the state of our great public universities.

Post A Comment

In order to be published, comments must be on-topic and civil in tone, with no name calling or personal attacks. Your comment may be edited for clarity and length.