 |
Schultz |
By Silas Polkinghorne
University surveys recently released by Maclean’s magazine and the Globe and Mail can be a good starting point for those considering where to study, but both fail to provide more useful and specific program-level rankings, says Robert Schultz, director of Institutional Analysis at the U of S.
“If you say, ‘I’m interested in the engineering program or a specific discipline within engineering, or commerce or the accounting program,’ they fall completely flat on that,” Schultz said in an interview last week.
Each institution organizes its units differently – math, statistics, and computer science might be joined together, they might be separate, or in some other combination – so comparing programs is difficult. “To drill down to a department or a college level to get information is almost impossible at this point.”
But the Globe and Mail, together with the Educational Policy Institute, has indicated it wants to work toward providing program comparisons. As well, the recently launched Common University Data Ontario (CUDO) is designed to give public access to some program-level information from members of the Council of Ontario Universities. The public can use the data to draw its own conclusions, Schultz said.
But the focus is often on the single ranking. This year, the U of S maintained its Maclean’s position of 10th among medical-doctoral universities in Canada. Schultz commends the magazine for taking regional differences in data collection into account, and for checking responses with third-party sources.
The main problem with Maclean’s data, however, is the use of a percentile method to compile the rankings, a practice that magnifies minute differences between institutions, he said.
For instance, the range of admission averages are very tight in the U of S’s peer category – about 7 or 8 points between top and bottom – and “a good chunk” of the difference can be attributed to varying methods of calculating the data. Nevertheless, “the top institution gets all the points for this indicator and the bottom institution gets almost none.”
Schultz disagrees with some of his colleagues who view the Maclean’s rankings as a scientific study. “It’s more of a public relations tool that I see. Yes, there is some scientific methodology applied to the information … (but) there’s so much noise in some of this stuff. You’re trying to make a precise calculation or decision on data that just doesn’t support that level of decision-making.”
Other indicators, like class size, do show a wide variance between schools and lend themselves better to the percentile method.
Maclean’s can also be criticized for the data they collect on reputation, a difficult thing to measure since very few people have a broad knowledge of universities in Canada.
In addition, rankings put a competitive edge to institutional research, which is otherwise “very collegial.” Because of the public relations that play into the Maclean’s survey, “there is lot of pressure to try to make sure you stay in your place in the rankings.”
Still, Schultz doesn’t think rankings or surveys are going to go away. He adds that a national repository for institutional data, like those in the U.S. and Australia, would be beneficial.
“I think Peter (MacKinnon) hit it right, saying we don’t necessarily want to rank institutions, but we want to provide the information for people who want or need it to make decisions about the institution, be it parents or students, taxpayers, government.”
Other portions of the Maclean’s and Globe and Mail content give useful information for students. Carrie Winter, a research analyst with Institutional Analysis, said Maclean’s University Guidebook may be more valuable than its rankings. “That’s where the institution gets to give you its strengths, or what it feels are its strengths.”
The Globe and Mail’s online University Navigator allows students to choose up to five indicators that are important to them, and rather than ranking institutions, it divides them into three tiers. However, Schultz notes the numbers are based on student perceptions, even for indicators like class sizes.
The University Navigator also attempts to encourage prospective students to do some of their own research. Users can click a button on the website to ask a university representative to contact them.
“You really need to start digging in and doing your homework on the institution yourself, and don’t rely on this third-party source to do your work for you,” Schultz said.
Login
