Quality Control in Education

It's important to measure how our universities are doing. The only problem is we don't have a good way to do it.

The idea of measuring university quality conjures up a stream of clichés: pinning water to a wall; catching air in a net; throwing a far less desirable substance against the wall and seeing what sticks.

This isn’t because university quality doesn’t matter. We need our institutions to offer the best learning experience to ensure the success of our students, the strength of our economy, and the health of our society. Rather, the tendency towards metaphor in discussions about higher education quality is due to one overwhelming problem: it is so darn hard to measure.

This creates a certain amount of consternation in policy circles. Measuring quality is important. We need to know that the public funding received by universities is being used effectively, and if it isn’t, how best to solve the problem. Competition for public resources is increasingly fierce, and citizens are becoming more hawkish with their tax dollars. If universities can’t demonstrate value, they will lose out on the funds they need.

This way of thinking is now common all over the globe, and has led to a kind of public policy gold rush. Governments are clamoring for ways to quantify the performance of the university system, and institutions are happy to oblige in the hopes of receiving greater public investment.

In theory, there are three things to look at when measuring university quality: inputs, such as public funding received or entrance averages of students; processes, like teaching and learning resources; and outputs, which include graduation and employment rates. Inputs and outputs are fairly easy to measure. Entrance averages are collected by institutions, and government funding is a matter of public record. Graduation and employment rates – although far from unproblematic – can be measured through student records and surveys. Since they deal with results, output measures are useful to governments keen to convince voters they’re spending tax dollars well. In Ontario, “key performance indicators” – basically a set of output measures – have been used to assess university performance for years.

The actual processes of university quality are much more problematic for policy-makers; an unfortunate reality, since this is where the actual learning occurs. The classroom experience, including engagement with faculty, opportunities for independent inquiry, and the breadth of educational resources like libraries and information technology, is very difficult to measure in a useful way. We do have a few tools at our disposal though. For example, the (http://nsse.iub.edu/html/about.cfm) National Survey of Student Engagement (NSSE) is a recent American import to Canada. It provides significant insight into how certain aspects of the learning experience influence student outcomes. But it is a long way from being a truly useful policy tool. At present, it’s hard to demonstrate in concrete terms that our universities are providing a high quality educational experience. Which is not to say they are underperforming; they just can’t demonstrate their success in a way that satisfies the current data mania in government.

In response to this confusion, governments have intensified their focus on what they can easily gauge. They are adding new and more nuanced output measures, shifting accountability structures away from the complexities of the classroom. This is harmful. It forestalls work on developing tools that can be applied to assess the learning process. While difficult to develop, such tools have much more to tell us about improving university quality. Outputs are great, but if we don’t know what combination of factors produced those outputs, their value is limited.

The focus on outcomes also precludes the use of informative, though less scientific, proxy measures. Take student-to-faculty ratios. We know that student engagement with their professors is a key element of a quality education. While the student-to-faculty ratio doesn’t measure the actual quality of this interaction at a given university, it does speak to an intuitive truth: if there aren’t enough professors, students can’t get the individual attention they need. This is an extremely important insight. However, it is being excised from discussion about higher education quality because it doesn’t have the clinical precision of an output measure. And, in many cases, process measures reveal more problems than outputs. Take Ontario, which now has the worst student-to-faculty ratio in Canada (27:1). There’s no political incentive to include this measure in the province’s quality framework, as it implies there may be some serious structural problems that need to be fixed. And, more likely than not, the solution won’t be cheap.

The current drive for better quality measurement thus creates a series of perverse results: on the one hand, a collection of outputs divorced from context, better at telling us “what” than “why” or “how.” On the other, a pile of under-used proxy indicators that have much to tell us, at least until deeper process measures can be developed. The classroom experience – and all it entails – is central to the meaningful university experience from which all other economic and social benefits of higher education flow. If we’re not interested in analyzing it, then we must never have been very interested in quality in the first place.