The Truth About Prospecting

It seems to me that the closer one looks into a subject, the more particular one’s taste becomes.

If you study to become a sommelier, you tend to appreciate wines other than the ones you used to drink too much of in university.  If you watch enough television, you can’t help but begin to shy away from the three camera sitcoms in favour of less traditional choices.  If you listen to a lot of pop music, you’re likely going to develop an interest in more obscure acts.

While some may attribute this to simply becoming a snob in your chosen area of expertise, I see it as a natural progression that keeps your interest piqued in a subject you’re passionate about.

After years of appreciating the game of baseball at the Major League level, I’ve slowly developed an interest in prospects and player development.  However, I fully realize I’m not clever enough to spot the difference between a blue chip prospect and merely a good one through observation alone.  And I haven’t been exposed to nearly enough Minor League Baseball to form opinions of my own when it comes to player development.

Instead, I’ve come to trust certain voices in the sport that are recognized as authorities when it comes to prospects.  In addition to providing baseball nerds with excellent statistical analysis, websites like Baseball America and Baseball Prospectus publish annual lists of the best prospects in each team’s system, and their word is generally accepted among baseball fans as being, if not Gospel, than certainly reasonable.

Nevertheless, the recent history of baseball metrics compels fans to constantly question that which is generally accepted about the game.

I first began to wonder about prospect rankings last week when Baseball America issued their top ten list for the Toronto Blue Jays organization.  Admittedly, I’m a Blue Jays fan and I know the organization and their system better than any other team’s in baseball.

Specifically, I questioned the rise of Anthony Gose to number three on the Jays list from number six on the Phillies, Deck McGuire’s position as their number two ranked prospect, the lack of Adeiny Hechevarria and the glaring omission of Brett Cecil from the projected 2014 rotation.  I also wondered how Gose could be named the organization’s fastest player, best defensive outfielder, with the best outfield arm, yet lose out to Jake Marisnick as the team’s best athlete.

I scoured the Baseball America website to learn exactly how these lists were compiled, looking for more than the explanation that the lists “are based on projections of a player’s long-term worth after discussions with scouting and player-development personnel.”

I eventually learned, through a contact from Baseball America, that the writers putting these lists together, are in fact, not scouts, and in most cases haven’t even seen the players that they’re “evaluating.”  Their lists are all second hand information.

Perhaps I’m being naive, but this was staggering to me.

While I understand that this evidence is somewhat anecdotal, to make sure I wasn’t alone in my ignorance, I asked every baseball fan I came in contact with over the next few days about Baseball America.  Not one realized that the website’s popular lists were being put together by guys who had rarely, if ever, seen the prospects they were offering opinions on.

Through my contact I learned that while each writer and editor has varying methods for compiling their list, most start with Baseball America’s Top 20 Prospect Lists for each Minor League, which has previously been compiled by amalgamating the opinions offered to them by actual scouts and league managers.

Again, I emphasize that none of this is based on Baseball America’s own evaluation, but rather a collection of what they’re hearing from their contacts in the industry.

The list compilers take that original information and then speak with several members within the player development department of the organization they’re ranking, including assistant GMs, scouting directors, pitching/hitting coordinators, pro scouts as well as a few others outside of the organization.

When I asked my contact if there was any concern over relying too much on the biased information coming from the clubs that they were supposed to be unbiasedly evaluating, he replied:

Certainly an organization will be higher on a guy because he’s in their system. The people I talk to I really trust so while I do temper their excitement some, it’s never really a lot. I haven’t had anyone tell me a guy is 95-98, when he’s actually 93-95.

You can even tell in their voice sometimes. They can tell me all they want about how much they love this guy, but then when they answer questions about specific tools you can tell that he shouldn’t be quite as high.

The flaws in this type of system are immense.  In addition to asking readers to trust the writer’s ability to discern the bias of the organization, we also must trust that his contacts are plentiful enough to get an accurate overall picture.

Considering how large their fleet of scouts is, the Toronto Blue Jays offer a perfect example of how something could fall between the cracks or appear to rise from the crevices.

With more scouts in the organization, each one sees a smaller quantity of players.  The writers at Baseball America could have the scout that recommended Anthony Gose to Alex Anthopoulos as a contact, but lack the scout who touted Adeiny Hechavarria, and we, the readers, would have no idea.  It can also explain why Kevin Ahrens was ever included in any top prospect conversation.

I also spoke with someone from Baseball Prospectus who compiles his lists in a similar fashion to Baseball America, through amalgamating the opinions of scouts and industry insiders, but he avoids speaking with members of the actual organization he’s ranking.  Instead, he goes so far as to specifically seek out scouts from other organizations in order to get a less biased opinion of a player.

When I asked my contact what benefit the scouts get from information sharing he said that it’s primarily done for social reasons.  Just as you and I like to talk about the games we’ve been to, the players we’ve seen, so do scouts.  However, there’s also an element of information that can go the other way, with writers taking advantage of their own network of contacts to exchange information with other scouts.

I wonder how much more accurate this method is.  If Baseball America is merely regurgitating the news releases of the clubs they’re supposedly evaluating, then it sounds as though Baseball Prospectus is giving a platform to their rivals.

Again, this method only works when the writer has a large array of contacts within the scouting world that he can trust, and while that may very well be the case, there are undeniably still flaws in this system.

The Toronto Blue Jays once again offer a perfect example of how things might go wrong, given the team’s new found aversion to speaking with media members.  If the same lock and key that guards the mouths of the team’s executives also governs the availability of their scouts to members of the media, are their voices simply not being heard by Baseball Prospectus?

What if one were to only get the opinions of the particularly chatty people in your office?  Would their viewpoints accurately represent the reality of that office?

I don’t mean to suggest there’s nothing of value that these two websites bring to their readership.  Baseball America and Baseball Prospectus are every day visits for me because of their ability to provide statistical and performance based analysis.  There’s no question in my mind that both publications have earned the respect they have in the baseball community.

My contact at Baseball Prospectus posed a very interesting question to me while we were discussing his methods.  Would I prefer to read the opinion of a guy who has seen a player one time or a guy who has talked to several people who’ve seen a player forty times?

I once read a quote that brought up all the issues that make the theory of evolution questionable, but at the end of the statement the author confirmed that, despite all of its drawbacks, it remains the best theory we have to explain our existence.

I’m not convinced that either Baseball America or Baseball Prospectus’ scouting methods are the best we have available, but there are no doubt flaws and biases in every method we have for prospect evaluation.  What’s most bothersome to me is the perception that exists among baseball fans that these websites are providing actual original talent evaluations, when in reality they’re relying directly on sources that in all likelihood are biased.

The perception exists that the writers for these publications are more scout than journalist, but the reality is very much different.

While I’m a firm believer that readers should take responsibility to properly weigh, judge and use critical thinking with anything they read from any source, websites also have a responsibility to provide a level of transparency in explaining how they derive at the conclusions that they publish.  When a false perception exists, even if it benefits the authoritative voice a publication is trying to create for itself, it should be corrected, lest the wrong impression grow.

As a first step, I would like to see both organizations be more forthcoming about the process for compiling their rankings and scouting reports.  Short of naming names, some sort of list could be provided relaying what organizations or what positions within an organization provided input into their rankings.

Secondly, if the writers aren’t claiming to be scouts, why are they including their own input at all?  Their performance analysis can certainly be valuable, but by mixing it with a smattering of other people’s skills analysis, they’re confusing their own role to readers, and at times, it seems, themselves.

Why not ask their contacts to give numeric scores on the players they’re ranking, and then tabulate the results?  “We spoke with fifteen scouts and six executives in charge of player development about the Toronto Blue Jays farm system, here’s how they ranked the top prospects.” It’s not going to be unbiased, but at least there’s no doubt about what role the writer is playing in ranking the players.

In the grand scheme of things, the way in which two websites rank 20 year old kids on how they play the game of baseball probably isn’t important enough to spend 1800 words writing about, but baseball can inspire people to do funny things, not least of all, get things right.