For the past few years, I have become intensely interested in the NHL draft: specifically, how do NHL teams make decisions on who to draft and who to pass-over? What factors do they use to determine the wheat from the Chaffe? More importantly, do these factors actually predict anything?
Projecting the future performance of 17 and 18 year old kids (based mostly on a season or two of play against their peers) involves so many variables, many of which are chaotic and unpredictable, it’s technically amazing that scouts are able to ever get it right. This is why the draft descends into straight probability after the few, obviously identifiable talents at the top of the class are gone. The future NHLers and busts are basically scattered around randomly, with the latter grossly outnumbering the former. This is due in no small part to the fact that the number of available positions in sports leagues is solidly capped: with just X roster spots on Y teams, only the best can take advantage of the few availabilities that pop up year-in, year-out. The window to take advantage of said availabilities is also very small: many organizations attached a “no prospect” label to their kids if they haven’t made the big club by the time they’re 24-years old.
However, sometimes talent alone isn’t the only reason first rounders tend to play more than later picks. In a 1995 study by Barry Staw and Ha Hoang, the authors tested the NBA draft to see if the sunk-cost bias caused draft position to predict future playing time, being traded and survival in the league independent of results or ability. From the article abstract:
Although one might logically expect that teams play and keep their most productive players, we found significant sunk-cost effects on each of these important personnel decisions. Results showed that teams granted more playing time to their most highly drafted players and retained them longer, even after controlling for players’ on-court performance, injuries, trade status, and position played.
Ori and Rom Brafman sum up the Staw and Hoang investigation in their book Sway:
[They] found that the variable most responsible for an NBA player’s time on the court – “above and beyond any effects of a player’s performance, injury, or trade status” – was his draft selection order. Even after controlling for other factors, in a given season “every increment in the draft number (e.g., getting drafted ninth instead of eighth) decreased playing time as much as 23 minutes.” Incredibly, draft order continued to predict playing time all the way through a player’s fifth year in the NBA, the final year measured in the study.”
Staw and Hoang also found that being picked later increased the chances a player would eventually be traded and have a shorter career. The perceived value of the first round pick clearly exerts influence years into the future, despite the fact that the draft number of any given prospect should be completely irrelevant in contrast to his actual performance.
This obviously suggests that a prospect’s future as a professional isn’t merely moderated by his level of skill, commitment, character or work ethic. Economic and psychological processes acting in the coaches and management of the team can hinder or promote a prospects efforts to make the leap from the minors to the majors. This is technically irrational behavior in terms of on-ice outcomes, although it can be argued that commitment escalation (or committing future activity based on sunk-costs) is sensible, for example, when a manager is trying to protect his reputation or when the outcome of the project is uncertain. Both factors are obviously at play when it comes to the NHL and developing prospects: scouts and GMs are apt to “fight” for their higher picks since both the success of the team and their subsequent reputations depends on them hitting “home runs”. In addition, predicting the ultimate development path of a prospect is highly difficult (ie; uncertain) so there’s almost always a plausible reason to continue to invest time and money in a former first round pick, perhaps ahead of or instead of other, lower selections.
This isn’t to say that draft order is the only variable that determines future playing time in the NHL. Obviously there are a number of high profile late or undrafted players who have gone on to have fruitful careers. In addition, no similar study has been carried out for the NHL so perhaps the effect is lessened or absent in hockey (possible, but unlikely).
Beyond the sunk-cost bias, the selection process itself as well as it’s resultant labels and expectations can also have an effect on the development of the prospect in question. Brafman and Brafman detailed how “diagnosis” or labeling can influence both perceptions and outcomes in the Israeli army:
105 soldiers were about to participate in a grueling fifteen-week commander training program…Before this session’s classes started, psychologist Dov Eden informed the training officers leading the program that the army had accumulated comprehensive data on each of the trainees, including…”psychological test scores, sociometric data…and ratings from previous commanders.
Based on this comprehensive information, Eden told the officers each soldier had been classified into one of three “command potential” categories: “high,” “regular,” and unknown (due to insufficient information).”
In the study, the trainees were unaware of the background data fed to the commanding officers, all of which was completely bogus and fabricated. Each trainee’s “command potential” was therefore assigned at random and had no relation to their actual abilities or intelligence.
Nevertheless, Eden found that the made-up categories for each trainee actually predicted higher test scores on combat tactics and practical skills at the end of the 15-week course:
The soldiers whom the training officers thought had a high CP score performed much better on the test (scoring an average of 79.98) than their “unknown” or “regular” counterparts (who scored 72.43 and 65.18 respectively). Simply being labeled, however arbitrarily, as having a high leadership potential translated directly into actual improved ability – improved by a staggering 22.7 percent.
The molding process becomes self-perpetuating: when we take on characteristics assigned to us, the diagnosis is reinforced and reaffirmed… When Eden informed the Israeli soldiers and officers that the command potential scores had actually been fabricated and assigned randomly, they staunchly disagreed.
As Brafman and Brafman later note, the tendency to take on positive and negative attributes or characteristics that are assigned to us by others is called the pygmalion effect and the golem effect respectively. Both the pygmalion and golem effect are two sides of the same coin: the tendency for expectation, perception and social reinforcement to drive how people interact, learn and progress.
These are just two overriding cognitive biases and behavioral tendencies that influence the draft and prospect development in sports. In an upcoming article, I will discuss the prospect interview and how it may help or hinder scouts in their quest to identify future NHLers.