Reading about baseball prospects is a little bit like travelling by airplane or getting surgery. It requires trusting in the expertise of others to perform a duty that is wholly and completely outside of our realm of qualifications.

Because the vast majority of people interested in baseball are unable to see younger players perform on a regular basis, and even if they do, it’s unlikely that they will know exactly what it is that sets one player’s raw talent above another player’s, we rely on the written words of those that do to form our hopes and fears for our team’s future.

But how do these people that we read and trust come up with their evaluations?

I wondered about such things a couple of years ago and reached out to a contact at Baseball America who explained the publication’s evaluation methods. I learned that most of the writers putting together top ten lists aren’t scouts, and in many instances, haven’t even seen the players than they’re ranking. The lists that they produce are based on second hand information.

While each writer and editor have varying methods for compiling lists, most start with Baseball America’s Top 20 Prospect Lists for each league that comprise the Minors. These master lists are created in advance by amalgamating the opinions offered to them by actual scouts and league managers.

The list compilers take that original information and then speak with several members within the player development department of the organization they’re ranking, including assistant GMs, scouting directors, pitching/hitting coordinators, pro scouts as well as a few others outside of the organization.

When I asked my contact if there was any concern over relying too much on the biased information coming from the clubs that they were supposed to be unbiasedly evaluating, he replied:

Certainly an organization will be higher on a guy because he’s in their system. The people I talk to I really trust so while I do temper their excitement some, it’s never really a lot. I haven’t had anyone tell me a guy is 95-98, when he’s actually 93-95.

You can even tell in their voice sometimes. They can tell me all they want about how much they love this guy, but then when they answer questions about specific tools you can tell that he shouldn’t be quite as high.

The flaws in this type of system are immense.  In addition to asking readers to trust the writer’s ability to discern the bias of the organization, readers also must trust that the writer’s contacts are plentiful enough to get an accurate overall picture.

Today, on Toronto sports radio, former Major League catcher Gregg Zaun went on a bit of a rant, voicing his concern with such methods and Baseball America, specifically.

Publications like Baseball America, those are all fueled by general managers and scouting directors, they’re telling them what to write about the players. They don’t send people from Baseball America out to these Minor League towns that have any kind of experience or credibility when it comes to evaluating talent and sit in the stands for a week and a half and watch a kid play.

They’re listening to what the GMs and scouting directors are saying. So, what do you think they’re going to say? They’re going to tell you that their players are awesome and they’re great defensively and they have massive talent because they want to pump the prospect in case they have to trade him at some point. The worst thing that can happen to a prospect is for him to come to the big leagues too soon, get over exposed, fall flat on his face and now everybody in baseball knows that the kid’s not the real deal.

It’s a garbage publication in my opinion. There’s some quality writing in there, but when it comes to the evaluation of players and how they rank them and all that kind of stuff that’s all fueled by the organization. They tell them what to write. They tell them who their top five prospects are.

While Zaun portrays himself as a bit of a clown, and goes to hyperbolic lengths on occasion, I can’t help but agree with some of what he’s saying here. However, his brush may be a little bit wide. There’s a distinct difference with how Baseball America collects their information and how rival publications, like Baseball Prospectus, put together their lists of top prospects.

When I initially looked into this, I also spoke with Kevin Goldstein of Baseball Prospectus, who told me about his own methods for compiling team rankings. He also amalgamates the opinions of scouts and industry insiders, but he will purposefully avoid speaking with members of the actual organization he’s ranking.  Instead, he specifically seeks out scouts from other organizations in order to get a less biased opinion of a player.

Most of Goldstein’s information is acquired through sources in the industry that, just like any other journalist, have been built up over time. Typically, it’s just a matter of being social with a scout, who just like everyone else, likes to share his opinions from time to time. Occasionally, it can become a two way street with information on opinions of one player being exchanged for other opinions.

This seems to me to be the better method, as it’s less dependent on information coming from individuals with a vested interest in how the quality of their prospects are portrayed. Of course, it’s not perfect, and biases will leak through from time to time, just as they will in any sort of opinion based writing, and it’s equally dependent on having a wide variety of sources providing information.

In a way, the entire issue that Zaun brings up offers us an important lesson. For fans who want the most accurate information, it’s important to read a wide variety of sources and do so while thinking critically and understanding where the information being provided is coming from. This doesn’t just go for baseball prospecting, either. The wider variety of opinions you gather, the better off you’ll be in understanding any argument on any issue.

And for the record, in 1993, Baseball America named Zaun, then in the Orioles organization, the best defensive catcher in the Double A Eastern League.