Goodhart's Law in Education
Writing is no trouble: you just jot down ideas as they occur. Jotting is simple - it is the occurring which is difficult.
—Stephen Leacock
Goodhart's law deserves to be better known since it is one of the most fundamental laws in the social sciences, at least as far as "laws" in the social sciences go. To recap, it states that “when a measure becomes a target, it ceases to be a good measure”.
In a generalized form, it is valid even in evolutionary biology, as I tried to show in one of the previous posts. Once a person understands Goodhart's law, it is hard not to notice it almost everywhere around him.
It usually occurs in situations where one actor has insufficient information about the characteristics of another actor, and this information is relevant for entering into a relationship with that actor, for example, for potential cooperation. Actors in those situations cling to certain indirect, but observable, indicators of those characteristics, or proxies, which are usually correlated with those characteristics.
Once some observable property of an actor becomes a proxy (or a measure) of some positive characteristic, that "proxy" becomes a "target." Because of this, over time the proxy loses its quality as a proxy for this positive characteristic and thus ceases to be a good measure.
For some time I have been fascinated by the workings of Goodhart's Law within the educational and scientific systems.
Let's take the PISA test as an example. If a country's goal is to improve its ranking among other countries in the PISA test, it may try to exclude from the test those regions or those populations that are expected to perform poorly on the test. Therefore, a country's score on the PISA test, although originally conceived as a measure of the level of education of fifteen-year-olds, ceases to be a (good) measure of the level of education of fifteen-year-olds.
If the measure of the quality of a scientist becomes the number of citations, cliques are formed that cite each other ("citation rings"), reviewers in journals ask for their works to be cited, and so on.
If the measure of the quality of scientists becomes the number of published publications, there is an increase in the publication of papers of poor quality with poor methodology, which in turn leads to wrong findings and “replication crises”.
If student evaluations become the measure of teacher’s quality, there will be inflation of good grades and reduced scope of work for students. If students' good grades become the measure of a teacher’s quality, we get "studying for exams" and an emphasis on short-term learning.
(It should be emphasized that if a certain measure of quality starts losing its quality as a measure of quality after becoming the "target" of optimization does not necessarily mean that it has completely lost its value as a quality measure - it may still be usable, but the question is to what extent and for how long.)
Similarly, a completed degree is a signal of intelligence, conscientiousness, diligence, work ethic, and similar qualities valued by employers. And this signaling aspect of education largely explains the "college wage premium", i.e. the fact that the highly educated on average have higher salaries than the less educated.
An alternative model says that people in educational institutions acquire knowledge and skills that future employers will value and that this explains the education premium. In contrast, the signaling model of education says that the educational system only "certifies" the qualities of the individual that he possesses independently of the educational system.
Now, one thing I haven't seen that the signaling model points out is that signaling itself is subject to Goodhart's Law: when a proxy for quality becomes the goal of optimization, the proxy ceases to be a proxy for quality, i.e. ceases to be correlated with quality. Put differently: when something like education becomes a signal of something desirable, actors start appearing who invest efforts in acquiring the signal even though they themselves do not possess the basic qualities of which education is the original signal.
Diplomas and doctorates, if they were once a good measure of quality, now cease to be, at least to an equal extent. The goal is what a diploma or doctorate usually brings: prestige for one person, a sinecure for another. Hence, various affairs with plagiarism ensue.
However, preserving the credibility of quality signals is one of the greatest tasks of a civilization. The absence of a way to identify quality and distinguish it from non-quality carries with it a huge social cost. It’s like an arms race between a parasite and its host: the host has to keep changing because the constant gives the parasite an opportunity to find "holes" in the host's system in order to exploit him. He has to run to stand still. But the education system is too slow to "run", and that is why it is an easy target for various parasites.
Some time ago I was reading a discussion on an online group. A person is a Ph.D. candidate and complains that he has no ideas for his PhD and asks others "How to come up with ideas?". I remember thinking to myself, here's an idea: have the % of uncreative minds increased over time, say in the last 50-100 years, in academia?
If you have no ideas for a PhD, perhaps you should consider the possibility that a PhD is not for you. It's no longer "I have an idea and I want to explore it through a doctorate", but "I want a doctorate and I need an idea to get one". Status signal. And over time it gets poorer and poorer, at least when it comes to intelligence. A way to impress the mediocre, "to be taken more seriously".
Of course, no one is going to suggest to that person to consider that possibility. "Yes, you can do it! Audacity of hope!", they say. Whatever! And then others describe to that person "how to generate ideas" ("first identify the problem, then...."), although it can't really be explained. Ideas come by themselves. Or they don't.
(First published in April 2021)