A well-written, sensation seeking book marred by fundamental analytical flaws

Kjetil K. Haugen and Alex Krumer
Faculty of Business Administration and Social Sciences, Molde University College

Introduction

While this is a well-written book in terms of the stories described and non-trivial findings that the average sports fan may like, we would refrain from labeling it as one that is based on sound and rigorous scientific work. The reason is the fundamental flaws in the approach and the analyses, as will be described in the two sections below. In the first section, we discuss the oversimplified statistical analyses, which mostly rely on simple correlations. The author, however, admits in the book that the conclusions presented are not causal and that the interpretation of the cognitive mechanisms is speculative. But then the question is: What is the value of such a book? What do we really learn from it?

Although Geir Jordet exerts a lot of effort to base his stories on scientific grounds, these stories are based on many studies that suffer from severe methodological flaws that include, but are not limited to, omitted variable bias, a small number of observations, arbitrary definitions, lack of proper statistical tests, selection on outcome, and many other problems.

For example, the author seems to completely disregard the basic statistical rule that in a non-random environment, a simple comparison of means between two groups makes no sense for the causal interpretation of the findings. Some studies mentioned in this book are equivalent, from the methodological point of view, to a “successful” medical experiment where a treated group of young and healthy people had significantly better health outcomes after a treatment compared to control group of old and sick people that received a placebo.

In the second section, we discuss another fundamental flaw of the book that disregards the strategic and interactive nature of a penalty kick between a kicker and a goalkeeper that requires a game-theoretical approach. While the author discusses that it makes little sense to repeat the same kicking strategy time after time, he disregards the obvious mixed-strategy equilibrium that should follow such a statement. In this equilibrium both the kicker and the goalkeeper should randomize their actions to maximize the probability of success. There is a wide literature on that, references to which are, unfortunately, absent from this book.

Statistics

To explain the many problems with the statistical analyses, let us start with a simple example. There is a well-known positive and significant correlation between the consumption of mozzarella cheese and the award of civil engineering PhDs in the US. Does this mean anything? Of course not. The problem with this correlation is that both mozzarella consumption and the award of civil engineering PhDs correlate with other variables (GDP per capita, R&D expenditure, etc.) that are omitted from the analyses. This is a well-known omitted variable problem that imply that the correlation is simply nonsense in terms of any causal interpretation.

Jordet claims that Messi is a below average penalty-taker based on the following. Messi scored 109 out of 140 penalties he took, giving him a success rate of 77.9%. The average success rate of all the players who played in Ligue 1 in 2022-2023 was 81.9%.

Just as with the mozzarella case, in penalty shootouts there are many non-random features, such as the different levels of pressure in different kicks, that the first kick is usually executed by the best player, players that kicked more in the past may have a different probability to score than those kicking for the first time, etc. Thus, if these variables correlate with both the dependent and independent variables in question, they should be included into the analyses, otherwise the results would be biased and not reliable due to the above-mentioned omitted variable problem.

Unfortunately, the many studies presented in this book suffer from this omitted variable problem. For example, the author explicitly says that the importance of the kick is a crucial predictor of the successful penalty kick, as also one of his early studies, that is not mentioned in this book finds [3]. However, for some reason the author does not include this variable in his follow-up studies, exposing his analyses to the omitted variable bias. It would be really interesting to listen to Geir Jordet’s explanation of the reason to leave this variable outside of estimations in his follow-up studies, especially after he knew about its importance.

To illustrate, let us discuss one of the main results of Geir Jordet’s work in general that relate to the preparation to the kick (e.g. waiting time, placement of the ball). For example, one of the mentioned studies in the book [4] finds that a longer waiting time is associated with higher scoring rate. However, the methodology of that paper raises many questions. First, the results are simple correlations between the waiting time and success rate without considering the importance of the kick and players’ experience, all of which may correlate with both waiting time and the success rate (some of these relationships are discussed in the book). Moreover, there were somewhat arbitrary intervals of waiting time whose choice was never discussed[1]. The results showed that some longer intervals of waiting time had significantly higher success rates than waiting time of 0-0.2 seconds. However, it seems that there was no difference between waiting 0.3-0.4 seconds and waiting more than 1.1 seconds. Thus, it remains a question whether the results are robust to different definition of intervals (and inclusion of other control variables).

Later in the book, the author again refers to this study, claiming that “players who take more time physically placing the ball on the penalty spot scored more goals than those who took shorter time” (page 75). Besides the fact that the analysis was again based on simple correlation without any controls, the statement is simply false from the statistical point of view. This is because in the referred study, there were again five arbitrary groups of time intervals (< 1.1 seconds, 1.2-1.5 seconds, 1.6-2.2 seconds, 2.3-2.9 seconds, above 3 seconds). Most importantly, none of the intervals was significantly different from the base group (< 1.1 seconds) in terms of scoring probabilities at the 5% level.

This was not the only time in the book when the author made false statistical statements. Beyond many descriptive numbers without any statistical inference, it is worth noting the example about Leo Messi. In that example, Jordet claims that Messi is a below average penalty-taker based on the following. Messi scored 109 out of 140 penalties he took, giving him a success rate of 77.9%. The average success rate of all the players who played in Ligue 1 in 2022-2023 was 81.9%. Without going into issues on importance of each penalty and the number of penalties per player, a simple two-sided binomial test shows that Messi’s success rate is not significantly different from 81.9% (p = 0.227). So, from the scientific point of view (which is what is expected from a professor in academia), the statement is wrong. The correct statement should be that there is no significant difference at conventional levels acceptable in science between success rate of Messi and an average player in Ligue 1. But we can understand that this sounds less fancy than a sensational statement that Geir Jordet chose to use in his book. After all, it is likely that sensations sell much more than a rigorous analysis.

Unfortunately, the list of such examples is too long for this short report, so we don’t wish to exhaust the reader, hoping that we were able to transfer the main message of a weak statistical foundation of the results discussed in the book.

Game Theory

It is astonishing to read a book about penalties which does not mention the word equilibrium once. The reason is of course that a penalty kick is a game, and it seems weird to write a 300-page book on penalties without even mentioning the terms game theory or equilibrium. We do not know whether the author intentionally has omitted this, in our opinion, quite important part of penalty kick research,^[2] or if this is simply a lack of knowledge of this branch of literature. Anyway, as we see it, adding this dimension would greatly have improved the scientific quality of the book.

So, what does this brand of research tell us? It tells us that a reasonable mathematical model of a penalty kick should result in a unique Nash equilibrium in mixed strategies. That is, penalty kickers (and keepers) should randomize on how to shoot (and try to save) their penalties. Hence, rolling dice could be the way to choose how to perform both the kicker and keeper tasks in a penalty kick.

From an intuitive point of view, this should not be very surprising. Suppose a certain football player has decided that all penalties he will take is of the “Pankenka”-type^[3]. Obviously, a “Panenka” penalty is extremely easy to save if the goalkeeper can predict it. So, if this player follows this strategy, after some penalties, the goalkeepers will start saving. If the player is not a complete idiot, he will realize this, and start making variations, that is sometimes using the Pananka (P), other times using, say, some alternative choice (A), perhaps initially in some kind of predefined pattern like; A,P,A,A,A,P… Unfortunately, a predefined pattern is both observable and recognizable, forcing him to realize that he has to be unpredictable. Then, the mixed strategy Nash equilibrium follows by itself. The only way to be perfectly unpredictable is to randomize.

As both intuition and theory indicates randomization, the way to perform real-world empirical testing on the topic should not be too far away. And indeed, this is the case. Many researchers did spend time on testing whether such randomizing behaviour could be identified from real world penalty data. In most of the cases, they identify such behaviour. See for instance: [2], [5] and even quite recently, [6].

So, why is the lack of this research knowledge a problem in Pressure? Jordet reflects a quite simple research pattern, involving observing penalties, kickers and keepers with the aim of identifying patterns separating either keepers or kickers into disjoint groups. An example which is very much present in all of the book, is the kickers who take the referee whistle blow as a signal to start their approach towards the eleven-meter mark and those who behave differently, taking time breathing in. Then, he counts up successes and failures in these two groups and performs over-simplified statistical testing on differences in success rates. If he finds significant differences, typically in favour of one of the groups – in this case the “taking-time-group” – he concludes that this is a favorable strategy. Let us be more precise. Jordet is really not that clear on a normative decision support approach, giving advice on how to take penalties at least not in his research. Still, the following quotation from the book on page 148 tells a different story regarding his consulting activity:

“Gianni Zuiverloon scored the last and decisive penalty with an unreachable shot down the left, and then very kindly gave me a mention in a post-match interview with Algemeen Dagblad, a Dutch newspaper:
’Geir Jordet told me that you have to take time before taking the shot. Before the run-up, it is good to rest a second and take a breath. I did that and I knew exactly where I wanted to shoot the ball.’ ”

Clearly, Jordet gives advice and that is the problem. Failing to take opponent players reactions into account may provide extremely erroneous advice. A simple example applying Jordet’s research model may clarify. Let us return to the “Panenka” discussed above. Jordet’s research model would typically then lead to observing “Panenka” penalties versus all other types. If it turns out that the goal frequency among the “Panenka” penalties is significantly higher than the rest of the penalties, Jordet’s advice should be to recommend applying “Panenka” penalties all the time. We all know (from the above) what happens – total disaster. Soon all penalties are saved, and all shootouts are lost. The point is simple: If one wants to analyze games, it is extremely hard without using game theory.

So, what else is there? Jordet spends a whole chapter (chapter 2) to discuss whether penalties are a lottery or not. Although not perfectly clear, Jordet seems to conclude that penalties are not lotteries. Taking the game theory intuition into account, clearly penalties are lotteries. This does of course not mean that neither kicker nor keeper should not try their best, but essentially, game theory tells us the answer. Clearly, a Nash equilibrium in mixed strategies defines a lottery.

Now, if much of Jordet’s advice may not be good advice, could it still work? Definitely, and it is again surprising that a researcher in psychology does not mention this at all. The placebo effect [1] may be present. After all, when a football team hires a person described by the New York Times as “[t]he leading expert on the psychology of penalties” this is a very strong signal to the players. Strong focus and resources are evidently put on improving penalty taking and this might clearly induce a placebo effect. As such, even if the advice given on how to take penalties may not be the best, the placebo effect may override the effect of bad advice and still produce success. Again, it is really surprising that a person holding a PhD in soccer and psychology does not find it worthwhile to discuss this effect in a book on penalties.

Conclusion

If you are looking for a well-written, sensation seeking book, this is the book for you. However, if you are looking for a book based on rigorous scientific method and analyses, you should look elsewhere. After all, an author who is described, on his own homepage^[4], as “The World leading expert on the psychology of penalties” should be expected to keep high scientific quality and perhaps also accept some critique.

To summarise, one of the greatest Dutch goalkeepers, Edwin van der Saar, wrote about this book that if he had known half of what’s in the book, his trophy cabinet would be much bigger. We would like to console him by ensuring that his trophy cabinet would not have been different had he known even everything that is written in this book.

Copyright © Kjetil K. Haugen & Alex Krumer

References

[1] K. Beecher. The powerful placebo. Journal of the American Medical Association, 159(17):1602–1606, 1955.

[2] A. Chiappori, S. Levitt, and T. Groseclose. Testing mixed-strategy equilibria when players are heterogenous: the case of penalty kicks in soccer. American Economic Review, 9:1138– 1151, 2002.

[3] Jordet, E. Hartman, and E. Sigmundstad. Temporal links to performing under pressure in international soccer penalty shootouts. Psychology of Sport and Exercise, 10(6):621–627, 2009.

[4] Jordet, E. Hartman, C. Visscher, and K. A. Lemmink. Kicks from the penalty mark in soccer: The roles of stress, skill and fatigue for kick outcomes. Journal if Sports Sciences, 25(2):121–129, 2007.

[5] Ignacio Palacias-Huerta. Professionals play minimax. The Review of Economic Studies, 70:395– 415, 2003.

[6] Ignacio Palacios-Huerta. Maradona plays minimax. Sports Economics Review, 1:100001, 2023.

[7] Czechoslovakia v West Germany: 1976 UEFA European Championship final highlights, 2016. From Youtube: https://www.youtube.com/watch?v=ROG4-QPIDgo.

Footnotes

[1] For the interested reader, the intervals were defined as follows: 0-0.2 seconds (n = 54), 0.3-0.4 seconds (n = 63), 0.5-0.7 seconds (n = 51), 0.8-1.0 seconds (n = 61), above 1.1 seconds (n = 53).

[2] See for instance [2].

[3] A “Panenka”-type penalty is performed by running fast towards the ball, mimicking a hard shot to one of the sides, but instead finishing with a simple loose lob typically in the middle of the goal. Given that the goalkeeper does not predict it, it is very effective. This shot is named after the Czechoslovakian player, Antonin Pankenka [7] for his decisive penalty in the 1976 EURO final shootout against West Germany.

[4] www.geirjordet.com