(2022-02-28) ColumbiaU CollegeRanking Scandal

Columbia University is facing a scandal over inaccuracy of the data provided to USNews for its college ranking.

The data has been challenged by Columbia math professor Michael Thaddeus. §1: A DIZZYING ASCENT

In sections 2 through 5, we examine some of the numerical data on students and faculty reported by Columbia to U.S. News

In each case, we find discrepancies, sometimes quite large, and always in Columbia’s favor, between the two sets of figures.

In section 6, we consider the financial data underpinning the U.S. News Financial Resources subscore. It is largely based on instructional expenditures, but, as we show, Columbia’s stated instructional expenditures are implausibly large

Finally, in section 7, we turn to graduation rates and the other “outcome measures” which account for more than one-third of the overall U.S. News ranking. We show that Columbia’s performance on some, perhaps even most, of these measures would plunge if its many transfer students were included.

Mar07: Andrew Gelman: “An Investigation of the Facts Behind Columbia’s U.S. News Ranking” | Statistical Modeling, Causal Inference, and Social Science

Michael Thaddeus writes: Nearly forty years after their inception, the U.S. News rankings of colleges and universities continue to fascinate students, parents, and alumni... Can we be sure that the data accurately reflect the reality of life within the university? Regrettably, the answer is no. As we will see, several of the key figures supporting Columbia’s high ranking are inaccurate, dubious, or highly misleading.

Thaddeus continues by shredding the administration’s figures on “percentage of faculty who are full time” (no, it seems that it’s not really “96.5%”), the “student-faculty ratio” (no, it seems that it’s not really “6 to 1”), “spending on instruction” (no, it seems that it’s not really higher than the corresponding figures for Harvard, Yale, and Princeton combined), “graduation and retention rates” (the reported numbers appear not to include transfer students).

it is reasonable to conclude that the true . . . percentage, among Columbia courses enrolling undergraduates, of those with under 20 students — probably lies somewhere between 62.7% and 66.9%. We can be quite confident that it is nowhere near the figure of 82.5% claimed by Columbia.

The picture coming into focus is that of a two-tier university, which educates, side by side in the same classrooms, two large and quite distinct groups of undergraduates: non-transfer students and transfer students

No one would design a university this way, but it has been the status quo at Columbia for years. The situation is tolerated only because it is not widely understood.

No one should try to reform or rehabilitate the ranking. It is irredeemable. . . . Students are poorly served by rankings. To be sure, they need information when applying to colleges, but rankings provide the wrong information. . .

A broad categorization of institutions, like the Carnegie Classification, may also be helpful — for it is perfectly true that some colleges are simply in a different league from others — but this is a far cry from a linear ranking.

Eighty percent of the U.S. News ranking of a university is based on information reported by the university itself.

Michael Thaddeus and I are teachers at the university and we have academic freedom. If you’re a spokesman, though, you can’t just say what you think, right?

Mar17, Andrew Gelman: “The 100 percent figure was rounded up”. The most stunning thing to me about the whole Columbia-cooking-the-books-on-their-U.S.-News-ranking story is that this information was all out there, and it took so long for someone (math prof Michael Thaddeus) to notice. The second most stunning thing is that whoever’s in charge of this hasn’t apologized and fixed things yet

Instead we get this: The 100 percent figure was rounded up, officials said, and they believed they were allowed some leeway . .

OK, the whole “terminal degree” thing is pretty much unimportant

I think Columbia is a wonderful institution. I love it here, and my colleagues throughout the university do cutting-edge research and quality teaching, with student involvement at all levels

I get it, everybody makes mistakes, there’s lots of pressure to keep that ranking high

But what they did here seems like a few steps too far, and this kind of defensive reply is distressing in itself.

Mar17 NyTimes: U.S. News Ranked Columbia No. 2, but a Math Professor Has His Doubts

A professor identified several data discrepancies that Columbia University provided to U.S. News & World Report, renewing the debate over the value and accuracy of college rankings.

In a 21-page blistering critique on his website, Dr. Thaddeus is not only challenging the rating but redoubling the debate over whether college rankings — used by millions of prospective college students and their parents — are valuable or even accurate.

Columbia said it stood by its data.

“I think the majority of institutions would be happy if the rankings went away,” said Colin Diver, a former president of Reed College, who has a book coming out about college rankings.

“To say, as U.S. News does, we’re going to rank 392 institutions from one to 392 is just absurd, and it forces all the wonderful diversity in that group into a single template,” Mr. Diver said.

Last year, a former dean of Temple University’s business school was found guilty of using fraudulent data between 2014 and 2018

The school’s online M.B.A. program was ranked best in the country by U.S. News & World Report in the years that he falsified data.

Over the years, other schools like Iona College, Claremont McKenna College and Emory University have been found to have falsified or manipulated data.

Dr. Thaddeus has not done a systematic analysis of universities other than Columbia, but he does have a bigger agenda. He believes that all rankings are “inherently suspect,” he said, because they are based on information from the institution being ranked.

Columbia claimed that 100 percent of its faculty had “terminal degrees,” the highest in their field; Harvard, for instance, claimed 91 percent, he said. By poring through the 958 full-time faculty members of Columbia College listed on its website (the only public list he could find), Dr. Thaddeus came up with 69 people (he has since corrected it to 66) whose highest degree, if any, was a bachelor’s or master’s degree

“Columbia would surely be a lesser place without them,” Dr. Thaddeus wrote.

On the financial front, Dr. Thaddeus found that Columbia’s claim that it spent $3.1 billion annually on instruction was “implausibly large” — more than Harvard, Yale and Princeton combined. He said that Columbia appeared to be including patient care in the spending, something he noted that New York University, for instance, does not do, to its detriment in the rankings.

Dr. Thaddeus, who has taught at Columbia for 24 years, has made a hobby of provoking his employer. He said he had been radicalized by the experience of being chair of the math department from 2017 to 2020, when he discovered how secretive the university was.

Apr24, Andrew Gelman: More news on Columbia University’s apparent manipulation of numbers that go into its U.S. News ranking. (Spoiler: it’s not good news.) It seems like a clear case of the Armstrong principle: If you push people to promise more than they can deliver, they’re motivated to cheat.

In the meantime, Thaddeus seems to have become addicted to this data-detective thing, and his next step was to look into some of the numbers for the engineering school. His new article is called “The U.S. News Ranking of Columbia’s Online Engineering Programs,”

The Columbia Video Network (CVN) is a distance education initiative of Columbia Engineering. . . . Total enrollments in CVN have grown dramatically in recent years

For the last five years in a row, CVN has been ranked #1 nationally by U.S. News in its Best Online Master’s in Engineering Programs ranking

Although class size counts for only 3.75% of the overall ranking, it has clearly been a key factor in providing Columbia with the competitive edge which has made it #1 year after year.

Columbia reported that fully 88.7% of its classes enroll under 10 students. This is an astonishing figure, miles ahead of the competition

Online class sections, typically very small, are listed alongside in-person sections of the same course, taught by the same instructor and typically much larger.

In the example shown below, an in-person Robotics class with 101 students was accompanied by an online section with 5 students.

online and in-person students are in the same section for all practical purposes. Online students appear on the same course roster as in-person students

CVN students reap none of the purported benefits of small class sizes. Their class sizes are small in name only.

The punchline: The class size figures on which Columbia’s #1 ranking rests are therefore meaningless

He loops back by reporting that the administrator responsible for this program, the Vice Provost for Teaching and Learning, has been promoted to Senior Vice Provost, and one of his achievements was listed by the provost as “propelling online engineering programs at Columbia to be ranked first in the nation for the past five years.” (looks like Soulaymane Kachani)

P.P.S. If Columbia’s screwing with the numbers, I guess lots of other universities are doing it too, in which case the U.S. News rankings are something like what rich executives’ tax returns would look like if the IRS didn’t do regular audits. To decide not to cheat could really be bad for your ranking! It’s the Armstrong Principle all over again.

May27: Andrew Gelman: More implausible numbers from Columbia University’s U.S. News ranking. Thaddeus has dug up more! These are all for the engineering program....

I agree with Thaddeus that these numbers seem implausible.

Just to be clear: I’m not claiming that anyone at Columbia is trying to lie or mislead. My guess is that it’s someone’s job to fill out these forms, and paperwork is no fun.

Sept12 NyTimes: Columbia University Drops in U.S. News Rankings, From No. 2 to No. 18. The change comes about eight months after a Columbia math professor accused the university of fudging some statistics used in the rankings.

Almost every year, U.S. News announces that it has found discrepancies in data submitted by universities for the rankings. But Columbia, an Ivy League institution, is probably the most storied and prestigious university in recent memory to be accused of providing incorrect data.

Some experts said that there also seemed to be a vindictive element to Columbia’s demotion

“We are not confident in the accuracy of the data Columbia submitted and did not use any prior year data,” U.S. News said. “For the 2022-2023 rankings, Columbia’s rank was calculated with data from the U.S. Department of Education’s National Center for Education Statistics, data from the peer assessment survey conducted by U.S. News, the College Scorecard and assigned competitive set values for data where no third-party data exists. Based on those data sets, Columbia ranks No. 18 in national universities.”