One of the most interesting parts of Gare Joyce’s Future Greats and Heartbreaks was his description of prospect interviews. For those who haven’t read the book, Joyce was allowed to peek behind the curtain of the Columbus Blue Jackets for about a year in order to research the ins-and-outs of NHL scouting and prospect procurement. He was present for the team’s draft operations roughly between 2006 and 2007.

Joyce takes the reader into the prospect interview and shares some of the questions, reactions and ostensible purposes of the process. The questions seemed rather innocuous in the book, although it’s interesting some five years later to compare the impressions of the kids at the time to their success as pros now.

More recently, former Flames sixth-overall pick Daniel Tkaczuk wrote about his own experiences with NHL draft interviews. His list of questions runs the gambit from the obvious to the bizarre:

• If you shoot 100 pucks during drills in practice, how many of them actually hit the net? (Is there really a right answer to this? We all know saying 100 is a flat-out lie.)

• The Reaction Question – Some teams like to throw reactionary questions out there to see what you do. Are you a virgin? Have you ever done drugs? What do you hate about the game? Why don’t you play more like ‘X’?

• The One-on-One – This meeting is over a casual coffee and is a little more intimate.

• Who do you hate playing against?

• The Board Room – Donald Trump-style firing squad, where questions come from all different angles and you have to defend your play and show them you are determined to make the jump to the pro game.

• Solving Puzzles and small IQ tests.

• What music do you listen to? What your favorite movie of all-time?

• The Party – One team was basically having a party for scouts and offered me a beer while I was fielding their questions.

• Personality Test – Taking the Myers-Briggs test to find out just exactly what you are about. Pretty scary how accurate the results are.

That’s a sampling of what Tkaczuk discusses. As you can see, some of it seems useful (“who do you hate to play against”), but some of it is flat out ridiculous (“are you a virgin?”). Tkaczuk also mentions the intimate one-on-one interview, which is probably more in-line with what many of us non-athletes face when trying to land a job.

In my previous article, I discussed Ori and Rom Brafman’s book Sway and how things like draft order or diagnostic labels can influence player development. In the same book, the authors also discuss the typical unstructured job interview and how ineffective it is for predicting future performance:

When researchers conducted a meta-analysis…they discovered only a small correlation between (unstructured) job interviews and job performance. The marks managers give job candidates have very little to do with how well those candidates actually perform on the job.

Interviews are poor at predicting performance for a number of reasons: the sample of time with the candidate is small, the interviewee can “prep” their answers to reflect what the interviewer wants to hear, etc. Many typical interview questions ask the candidate to honestly evaluate themselves (when they won’t) to act as a historian (accurately remember past performances) or predict the future (what are you plans in five years?) which is, naturally, impossible.

Brafman and Brafman note that managers and decision-makers are nevertheless drawn to the unstructured interview:

Although everyone knows…that performance charades are going on, hiring managers are attracted to the “first date” format, thinking that a good conversation will allow their instincts to guide them to the right candidate.

The reason managers can err so easily is that, in addition to ignoring objective data, they focus on and give too much credence to irrelevant factors…It’s a trap familiar to anyone who’s ever bought a bottle of wine soley because of the attractiveness of the label.

There are many traps that lie in wait for the unwary manager – simply “liking” a candidate personally more than another because he/she reflects the managers personality or values, for example. People tend to prefer people who are like them which makes sense when we’re choosing friends, but not when the goal is to predict future performance at a specific job or task.

Another trap is the influence of non-diagnostic information to clutter or inhibit our judgements. This is known as the dilution effect. Henry Zukier studied the dilution effect by asking people which of  two hypothetical students would have a higher grade point average:

Tim Spends about 31 hours studying outside of class in an average week.

Tom spends about 31 hours studying outside of class in an average week. Tom has one brother and two sisters. He visits his grandparents about once every three months. He once went on a blind date and shoots pool about once every two months.

On average, people answered that Tim was probably the better student than Tom, even though the pertinent, diagnostic information for both students was exactly the same.

To bring this back to Future Greats and Heartbreaks, I was struck while reading the book just how poorly Phil Kessel was received by the Columbus Blue Jackets organization during his draft year; mostly because of personal foibles that seemed (to me) irrelevant or at most periphery when it came to his actual abilities as a hockey player. Kessel was chosen fifth that year by the Boston Bruins, one place ahead of the Blue Jackets (who took Derek Brassard sixth) so ultimately their grading of him was moot, but it’s instructive to see that Kessel went on to become a fairly capable NHLer despite the BJ’s doubts.