www.hockeythink.com
Statistical Analysis in Hockey
Copyright Iain Fyffe, 2003
Contents:
The Place of Statistics in Sports
The Purpose of Statistical Analysis
Principles of Statistical Analysis
Statistical analysis in hockey is an important and growing field within hockey research. Traditionally, hockey statistics have lagged behind the stats of other sports, particularly baseball, in terms of sophistication. In recent years, however, more emphasis has been placed on statistics in hockey, both from within the sport and without. This has been a double-edged sword of sorts; statistical analysis, when performed properly and used appropriately, can be a very powerful and informative tool. Unfortunately, all too often it is misused, or even abused. This leads to legitimate analysis being eyed suspiciously, because there are so many invalid statistical assertions made about the sport. This essay discusses statistical analysis in hockey, and attempts to identify what can and cannot be done with it. Guidelines for conducting statistical analysis within a consistent framework will also be discussed.
First we must dispose of a pervasive notion: that statistical analysis in hockey is leading (or has led) to the "baseballing" of hockey. By this, I am referring to baseball's tendency to be overanalyzed on a statistical basis. Everything in baseball is expressed in numerical terms, whether doing so is appropriate or not. Stats are kept that could not possibly have any significance. For instance, if you wanted to know how Manny Ramirez is hitting against Roger Clemens, at home, on Tuesdays, during night games, with a 3-1 count and no men on base, you could find out. Stats of this kind are ubiquitous in baseball (though not usually to this extreme), and are particularly emphasized in television broadcasts. For the most part, statistics this detailed have very limited meaning, and as such are more irritating than useful.
Fortunately, we need not fear this degree of statistical abuse in hockey. Hockey can never be "baseballed". The games are intrinsically different. Baseball can be broken down into discrete, easily-identifiable bits of action (identified as "segments" by Jeff Klein and Karl-Eric Reif) that can be well-summarized by statistics. There is some variation in what constitutes a sinlge, of course, but on the whole singles are very comparable to one another. Any action that occurs on a baseball diamond can be summarized with relative reliability. Each action is seperable and identifiable: hit, ground out, stolen base, et cetera. This is what lends baseball so easily to extreme statistical analysis.
Hockey simply cannot be broken up into these nice, identifiable segments. By its very nature, hockey is a fluid sport, constantly moving, with stoppages or breaks in the action being relatively uncommon. Hockey, by its inherent qualities, cannot be "baseballed". Statistical analysis in hockey can be inspired by the analysis of baseball, but it must be approached from a perspective unique to the sport.
For instance, there are two significant aspects of hockey that baseball researchers would never have to consider, because there are no parallels in their sport: special teams, and the relationship between offence and defence. In baseball, teams never have to take the field with only eight men due to some infraction. In hockey, odd-man situations are common, and must be considered in statistical analysis.
In baseball, offence and defence are completely independent; you cannot score a run while playing defence. In hockey, offence and defence are played simultaneously. You can score and be scored upon at any time. Historically, there is a fairly strong realtionship between goals for and goals against: teams that score a lot of goals tend to allow few. There are two explanations for this. First, good players (and good teams) tend to be good at all aspects of the game, on average. Second, there is the "indirect defence" effect of having a good offence. That is, if you have a good offence, you will be controlling the puck more often, and therefore your opponents simply have less time with the puck to score against you. This sort of consideration does not exist in baseball.
This does not mean we cannot learn from statistical research in baseball. For the greater part, the statistical methods themselves cannot be adapted to hockey, much less be applied directly. But some of the ideas involved in baseball statistics can be adapted for use in hockey research. No source of ideas should ever be ignored; one never knows what one can learn should one simply investigate.
The Place of Statistics in Sports
Statistics have their place in sports. Their role is best described as a complement to the game, recording what has happened. Stats are tools, best used to help understand the game, to reinforce or refute our perceptions. Unfortunately, all too often statistics will override the game, rising above it to become ends unto themselves. The final results of games are ignored; only individual statistics are considered. This is demonstrated by the prevalence of fantasy leagues, wherein fictional teams are assembled, and the player's statistics are added together to determine success. This assumes a player's stats are absolute, free of any dependence on circumstances such as team and teammates; this is, of course, silly.
It is especially disturbing when statistics actually affect how the game is plyaed; not by the truths revealed by statisical analysis, but by the desire for the accumulation of stats themselves. This is most frequently seen in baseball. How often have we seen a team's closer warming in the bullpen while his team enjoys a three-run lead (the maximum allowed for a one-inning save), only to be sat down when his team pushes across another run? Now that a save is no longer available (simply by the arbitrary definition of a save), the closer is not used. While you could argue that the larger margin leads the manager to save his best reliever for other situations, why does he not do this when the lead goes from two runs to three? If you watch enough baseball, it becomes clear the manager's decision is at least partly based on the arbitrary definition of a save. Statistics should be a reflection of what occurs on the field (or rink), or tools used to better understand the game; they should not be determinants of how the game is played, simply by the existence of the stats. Statistics are not the be-all and end-all of sports; they are only one small part.
The Purpose of Statistical Analysis
To paraphrase Bill James, we can define statistical analysis as the search for objective knowledge about hockey. The key word here is "objective". Statistical analysis attempts, in part, to prove or disprove accepted "truths" about hockey, as well as to uncover things never before considered or known. Hockey certainly has its share of platitudes, which need to be examined on an objective basis. Examples include ideas such as "defence wins championships" (not true), and "playoff experience is critical for playoff success" (also not true).
When an assertion is made about hockey, it is most often based upon personal observations and perceptions, and is therefore subjected to personal biases. If possible, the assertion should be tested through statistical analysis, in order to determine its validity, or lack thereof. Of course, not everything can be tested statistically. But everything that can be tested should be tested.
The purpose of statistical analysis in hockey is to determine the validity of assertions, not only to prove or only to disprove. Statistical analysis, to maintain objectivity, should be free from bias. A decision about the validity of an assertion should not be made until it is tested. Statistical analysis is not intended solely to debunk beliefs about hockey, but also to support them, if the evidence is there. It involves only objective analysis.
This is not to say that subjective perceptions and judgments cannot be valid. Some things simply cannot be examined statistically. In these cases, we can only rely on consensus. However, when something can be tested statistically, objective analysis (when properly done) should be preferred over subjective judgment.
In order to discuss the use of statistics, we should define what a statisic is. A statistic is a number that, either by definition or by mathematics, measure the occurrence or value of something. For example, goals scored is a statistic, because it measures the number of times that a player put the puck into the net. Goals per game is also a statistic, because it measures the average number of goals a player scores in a game.
The definition of a statistic is a common pitfall of pseudo-statistical analysis. Often, when a new "statistic" is created, it is simply the combination of two or more existing statistics. Combining statistics in this way does not necessarily created a valid statistic. As a simple example, take the idea of the "Intimidation Quotient" (IQ), as discussed in The Hockey News Yearbook a few years ago. This number, designed to represent how "intimidating" a player is, was defined as (note how this is not a quotient):
IQ = 3 x goals + penalty minutes
The result of this formula is inherently meaningless. There is no basis for the combination of goals (a good thing) and penalties (a bad thing), or for the goal multiplier. Just by playing with the way in which the statistics are combined, we can influence the results significantly. This creates inherent bias.
IQ, as defined above, is actually an index, not a statistic, as we will see later. However, no evidence is presented to indicate that IQ measures something valid. Is a high IQ good? It could be the result of having a ton of penalties. This was prevented by the assignment of arbitrary minimum values for goals and penalties, thereby reducing the validity of the index even further. But the real problem is that IQ attempts to quantify something that cannot be quantified. Intimidation is almost certainly a facet of hockey that cannot be examined statistically. It may show up as part of a true measure of defensive ability, for example, but we would not be able to idenfity how much should be attributed to indimidation.
There is nothing wrong with combining statistics together to form another statistic, but it must be done in an appropriate manner. It is not valud to simply assign values to statistics based on personal perceptions or whims. By not using appropriate statistical methods, it is easy to allow personal biases or agendas affect the formulation or interpretation of results. If one decides ahead of time what the results of a study should be, it is far too easy to bias the results, if only unconsciously. In order for statistical analysis in hockey to have real meaning, it must be free from bias.
Statistics can be classified in three ways: by their mathematical nature (counters, rates, metrics and indexes), by their inherent nature (straight-time or stop-time), and by their contextual nature (hard or fuzzy).
Counters: These statistics are the simplest; they count the number of times an event occurs. Goals scored, for instance, count the number of times a players puts the puck in the net. Counters are common in hockey. Unfortunately, they need the most interpretation to be understood properly, and many people try to take them at face value.
Rates: These are generally one counter divided by another. They indicate the rate at which one counter occurs relative to another. Rates help in the interpretation of counters. For instance, take two players who score the same number of goals, buy play a different number of games. The player with the higher goals-per-game average is probably the better goal-getter.
Metrics: Any statistic that combines several other statistics together (in a mathematically valid manner) is a metric. Rates are actually a subclass of metrics, but are so common that they deserve a group of their own. While common in baseball, complex metrics are still rare in hockey. Plus-minus is an example of a metric, because it is the combination of four other statistics (total goals for, power-play goals for, total goals against and power-play goals against).
Indexes: Indexes are not true statistics in that they do not have any inherent meaning. This is not to say that they have no use; indexes can be used to rank players in some way, even if the value of the index itself has no meaning. Indexes are often formulated similarly to metrics. And similar to metrics, their validity must be established before they can be used in any meaningful way.
Stop-time and straight-time statistics: The difference between stop-time and straight-time statistics is subtle, but can be important. It is another example of something a baseball researcher never has to consider. A statistic that reflects an action that causes the play to stop is a stop-time statistic. The primary examples are goals, assists and penalties. A straight-time statistic reflects an action that does not cause the play to stop. The key distinction is that, for the most part, a stop-time statistic is actually a straight-time statistic that produces a particular result. For example, goals are a stop-time statistic that are only recorded when a shot (a straight-time statistic) produces the particular result of a goal. If the shot is stopped, it is only a shot, not a shot and a goal. It is worth noting that assists have no straight-time equivalent as do goals. When using both stop-time and straight-time statistics in a study, this basic difference must be borne in mind. Extra care must be taken when comparing these statistics directly, due to their inherently different natures.
Hard and fuzzy statistics: These aren't really two different types of statistics, but rather two extremes of a scale. A hard statistic is one that is immune to contextual effects, while a fuzzy stat is useless without considering its context. Most stats are not so clearly defined in this regard, but will tend toward one end or the other.
Numbers can be manipulated in many and diverse ways. When dealing with sports statistics, we generally need not concern ourselves with the more complex mathematical methods. No disaster will be produced by results that aren't exactly right; it's just a game, after all. However, to properly perform statistical analysis, a basic understanding of some common statistical terms and methods should be had. At the very least, it will aid you in your understanding when reading a statistical study.
Deviation: The standard deviation of a data set indicates the variability around the mean (average). For instance, the sets {1,2,2,3} and {0,1,3,4} have the same mean (2.0), but the second has more variability. The coefficient of variation allows the comparison of the variability of two data sets, by dividing the standard deviation by the mean. In a normal distribution of data (a bell-shaped curve), approximately 68% of the observations (points of data) will be within one standard deviation of the mean. Approximately 95% will be within two standard deviations of the mean, and virtually all will be within three standard devaitions. The number of standard deviations an observation is from the mean is also called the z-score.
Standard error: Whenever a mathematical model is developed to reflect the real world (as in the case of many metrics), error will result; no model is perfectly accurate. The standard error can be used to determine which version of a model is the most accurate. The model with the smallest standard error "fits" the data the best, and is therefore the most accurate.
Correlation: The coefficient of correlation indicates the relationship between two matched sets of numbers. The coefficient ranges from -1 to +1. If it is positive, it means that if a number is high, its matched pair will also tend to be high, relative to their respective data sets. It the coefficient is negative it means that if a number is low, its pair will tend to be high. A correlation with an absolute value of 1 (that is, either +1 or -1) indicates a perfect relationship. A correlation with a lesser absolute value indicates a weaker relationship. A correlation of 0 indicates no relationship exists. In order to infer a strong relationship, you should have a correlation of 0.6 or greater in absolute value. But remember, correlation does not imply causation. Two sets may be highly correlated because they are both related to some third set, and the mathematical relationship between them may have nothing to do with a real relationship.
Regression: A linear regression is a statistical method used to develop models that minimize the standard error of the estimate. It involves one dependent variable (the number you are trying to estimate) and several independent variables (which are use to determine the dependent variable). It assumes a linear relationship between dependent and independent variables. Other types of regressions exist, allowing for relationships of other than a linear type.
Significance: The significance level of a statistical inference indicates how often the inference will be wrong. For example, a significance level of 5% means that 5% of the time, the conclusion drawn will be wrong. While this is a very important concept when using real-world statistics, where a high rate of incorrect conclusions can be truly hazardous, in the world of sports statistics it is not so important. Which is to say, we have the advantage of not having to be stringent in our application of significance; remember, it's just a game.
A note on decimals: When calculating an average or rate statistic, a number of decimal places must be selected for presentation purposes. For instance, one divided by two could be presented as 0.5 (one decimal) or 0.500 (three decimals), or what have you. You need only the number of decimals that will differentiate players the majority of the time. Let's look at save percentage. It is generally expressed to three decimals, such as .906 (the intial zero is left off for simplicity). This is enough to differentiate most players most of the time; expressing it as .91 would not be enough to do this (many players would be considered equal), while .9060 is redundant, very rarely being needed to break a tie. Sometimes ties do occur at three decimals. For example, for the 1999/2000 NHL season, three goaltenders finished the season with a .919 save percentage, which was highest in the league: Ed Belfour, Dominik Hasek, and Jose Theodore. However, only Belfour is credited with leading the league, because by taking the numbers out to extra decimals, his is the highest. To five decimals, Hasek's figure was .91889, Theodore's was .91923 and Belfour's was .91925. Now, you can be comfortable saying a goalie with a .919 save percentage was probably better than one with a .918 save percentage. But can we really say Belfour was better than Theodore because his save percentage was .00002 better? No. It was a statistical tie, and should be considered as such. Extending the number of decimals only when there is a tie atop the leaderboard is inconsistent and illogical. When the appropriate number of decimals produces a tie, a tie it is.
Principles of Statistical Analysis
The purpose of hockey is to win. There are two lesser purposes that contribute to this greater cause: to score goals, and to prevent your opponent from scoring goals. Therefore, for a statistic to be worthwhile, it should indicate a contribution, either direct or indirect, towards one or more of these purposes.
Furthermore, analyses of individual players should, as much as possible, reflect only a player's own contribution towards these purposes, rather than the play of his teammates or opponents, or his coach's decisions. This is not always an easy proposition when dealing with hockey statistics. The very nature of the game makes the distinction between team play and individual play blurry. But we must make the attempt. If it were easy, it wouldn't be any fun.
A further distinction must usually be drawn between team analysis and player analysis. An analysis that applies on a team level will not necessarily apply on an individual player level. This is the problem faced by such stats as Runs Created in baseball; they are formulated based on team totals, and then applied directly to individual players. This may or may not be valid, and is very difficult to prove one way or the other.
Finally, when evaluating a statistic or a satisistical inference or conclusion, we must always ask if there is a better way to measure the same thing. Redundant statistics are just that, and should be avoided. If an analysis adds nothing new to the collective knowledge of hockey, it is pointless.
Here's one final point. Statistical analysis in baseball is often called Sabermetrics, after the Society for American Baseball Research, or SABR. Some hockey researchers (myself included) believe this gives baseball reasearchers a greater degree of solidarity, as well as credibility. So it is desireable to have a name to call what we do. The most appropriate term I have yet seen suggested is simply "hockey metrics", coined by Marc Foster. So to all you hockeymetricians out there, I hope this essay has been of interest to you, and perhaps of aid as well.
Here are a few quotes to keep in mind, mostly written by people far cleverer than myself.
On Objectivity
"'I don't know.' That is the first principle." - Shunryu Suzuki-roshi
"I mean by intellectual integrity the habit of deciding vexed questions in accordance with the evidence, or of leaving them undecided where this evidence is inconclusive." - Bertrand Russell
"Every man is a creature of the age in which he lives; very few are able to raise themselves above the ideas of the times." - Voltaire
"It is the mark of an educated mind to be able to entertain a thought without accepting it." - Aristotle
"If fifty million people say a foolish thing, it is still a foolish thing." - Anatole France
"A man should never be ashamed to own he has been in the wrong, which is saying, in other words, that he is wiser today than he was yesterday." - Jonathan Swift
"Practically perfect people never permit sentiment to muddle thier thinking." - Mary Poppins
On New Ideas
"All truth passes through three stages: First, it is ridiculed; Second, it is violently opposed; and Third, it is accepted as self-evident." - Arthur Shopenhauer
"Research is to see what everybody else has seen, and to think what nobody else has thought." - Albert Szent-Gyoergi
"New opinions are always suspected, and usually opposed, without any other reason but because they are not already common." - John Locke
"Advances are made by answering questions. Discoveries are made by questioning answers." - Bernhard Haisch
"If a man is in too big a hurry to give up an error he is liable to give up some truth with it." - Wilbur Wright
"When people thought the world was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that thinking the earth was spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together." - Isaac Asimov (note: in case you don't know, the earth is an oblong spheroid, meaning it's almost spherical, but not quite)
On Human Bias
"Only two things are infinite, the universe and human stupidity, and I'm not sure about the former." - Einstein
"When did ignorance become a point of view?." - Scott Adams' Dilbert
"The eye sees only what the mind is prepared to comprehend." - Henri-Louis Bergson
"There are some people that if they don't know, you can't tell 'em." - Louis Armstrong
"There's nothing so passionate as a vested interest disguised as an intellectual conviction." - Sean O'Casey
"The fact that a believer is happier than a skeptic is no more to the point than the fact that a drunken man is happier than a sober one." - George Bernard Shaw
"Let the mind be enlarged...to the grandeur of the mysteries, and not the mysteries contracted to the narowness of the mind." - Francis Bacon
"When two opposite points of view are expressed with equal intensity, the truth does not necessarily lie exactly halfway between them. It is possible for one side to be simply wrong." - Richard Dawkins
"Men occasioanlly stumble over the truth, but most of them pick themselves up and hurry off as if nothing had happened." - Winston Churchill
"A lie can travel half way arounf the world while the truth is putting its shoes on." - Mark Twain