The Strategic Logic of Overconfidence
Why does overconfidence exist? Isn't it irrational?
Why does overconfidence exist? Isn't it irrational? And if not, how can excessive (which implies an inaccurate representation of reality) self-confidence be rational? Isn't it always more rational for beliefs to better approximate reality?
Here is one consideration of the logic behind the overconfidence effect.
Self-confidence is, in short, an evaluation of the subject by the subject himself. The real world is full of information asymmetries, i.e. situations where a person A knows more about himself (or something else, say a car he wants to sell) than a person B with whom A enters into a relationship in which person B may incur a certain cost if he does not evaluate person A accurately (or something that person A knows better and B has an interest in). For example, the relationship between the employer and the employee, the seller and the buyer of some thing or service, marriage or friendship.
In conditions where they know nothing about the agent being evaluated, people “cling” to that agent's level of self-confidence as a quick-and-dirty heuristic used to infer something about the agent in the absence of currently available better alternatives. They cling to self-confidence as a proxy for the “quality” of an agent in a certain domain. The logic, in short, is: "If a subject evaluates himself or herself positively in domain X, assume he or she possesses positive properties in that domain." Since person A knows more about himself than person B does, it is assumed that this knowledge will be reflected in the greater or lesser self-confidence of agent A and that it can therefore serve person B to obtain information about person A.
The problem is that as soon as this logic is established, self-confidence becomes an object of manipulation: pressure arises to develop excessive self-confidence as an instrument of persuading the other party. Excessive self-confidence becomes a strategic move in the arms race between “deceivers” and “deceiver-detectors”. And this brings us to Goodhart's law: when a measure becomes a target, it ceases to be a (good) measure.
To illustrate Goodhart’s law: if a country, say, wants to improve its ranking among other countries in the PISA tests, it may try to drop those regions or populations that are thought to perform worse on the test. Therefore, the country's score on the PISA test, although originally conceived as a measure of the level of education of 15-year-olds, ceases to be so when it becomes a target of optimization.
Formulated more generally, Goodhart's law states that when a proxy for quality becomes the goal of optimization, the proxy ceases to be a (good) proxy for quality, i.e. the proxy ceases to be correlated with quality or the correlation weakens.
In this context, it means that when self-confidence as a measure of quality becomes the goal of optimization, self-confidence ceases to be a (good) measure of quality.
In a romantic context, the lack of information about the other side has a higher cost for women as they have a higher cost of choosing a suboptimal partner. Therefore, women "like confident men", and in response, men show excessive self-confidence. A further move in the “battle of the sexes” would be for women to either reduce the weight they place on the value of self-esteem, or to find a factor that is a proxy for overconfidence; i.e. a factor that has a high correlation with the “inaccurate” or “strategic” self-confidence, and low (or 0) correlation with the “accurate” self-confidence.
Politics is another domain in which overconfidence pays off. It is assumed that self-confidence is a quality of a good leader. One of the main tasks for a politician is to win the trust of people (voters) who do not know him and in order to do that he has to advertise his competence. Donald Trump probably did this in his own public appearances. In his 1987 book The Art of the Deal, he writes:
The final key to the way I promote is bravado. I play to people's fantasies. People may not always think big themselves. but they can get very excited by those who do. That is why a little hyperbole never hurts. People want to believe that something is the biggest, the greatest and the most spectacular.
Similarly, behavioral economists often point out that the business and financial worlds are full of "irrational" overconfidence: an analyst who sells stock market forecasts will generally err in forecasting—otherwise he would not work as a forecaster. However, the analyst’s behavior is "ecologically rational" in the sense that showing self-confidence is the only way to convince others to use his forecasting services, in other words—a good PR.
Here are some of the hypotheses that we can derive from the model:
People will mostly use a person’s level of self-confidence to find out something about a person when they don’t have enough information about that person. Related: increasing information about the person we are assessing leads to a reduction in the weight we place on self-confidence as a measure of quality.
People will mostly be encouraged to show excessive self-confidence when the other person does not have enough information about them. Related: increasing the information that the other person possesses about us leads to a decrease in overconfidence.
Whatever parameter (if possible) replaces self-confidence as a proxy for quality, it will become subject to equivalent (not necessarily conscious) manipulation. Related: When the proxy for quality becomes the goal of optimization, an incentive arises to find another proxy.