Reading about baseball prospects is a little bit like travelling by airplane or getting surgery. It requires trusting in the expertise of others to perform a duty that is wholly and completely outside of our realm of qualifications.

Because the vast majority of people interested in baseball are unable to see younger players perform on a regular basis, and even if they do, it’s unlikely that they will know exactly what it is that sets one player’s raw talent above another player’s, we rely on the written words of those that do to form our hopes and fears for our team’s future.

But how do these people that we read and trust come up with their evaluations?

I wondered about such things a couple of years ago and reached out to a contact at Baseball America who explained the publication’s evaluation methods. I learned that most of the writers putting together top ten lists aren’t scouts, and in many instances, haven’t even seen the players than they’re ranking. The lists that they produce are based on second hand information.

While each writer and editor have varying methods for compiling lists, most start with Baseball America’s Top 20 Prospect Lists for each league that comprise the Minors. These master lists are created in advance by amalgamating the opinions offered to them by actual scouts and league managers.

The list compilers take that original information and then speak with several members within the player development department of the organization they’re ranking, including assistant GMs, scouting directors, pitching/hitting coordinators, pro scouts as well as a few others outside of the organization.

When I asked my contact if there was any concern over relying too much on the biased information coming from the clubs that they were supposed to be unbiasedly evaluating, he replied:

Certainly an organization will be higher on a guy because he’s in their system. The people I talk to I really trust so while I do temper their excitement some, it’s never really a lot. I haven’t had anyone tell me a guy is 95-98, when he’s actually 93-95.

You can even tell in their voice sometimes. They can tell me all they want about how much they love this guy, but then when they answer questions about specific tools you can tell that he shouldn’t be quite as high.

The flaws in this type of system are immense.  In addition to asking readers to trust the writer’s ability to discern the bias of the organization, readers also must trust that the writer’s contacts are plentiful enough to get an accurate overall picture.

Today, on Toronto sports radio, former Major League catcher Gregg Zaun went on a bit of a rant, voicing his concern with such methods and Baseball America, specifically.

Publications like Baseball America, those are all fueled by general managers and scouting directors, they’re telling them what to write about the players. They don’t send people from Baseball America out to these Minor League towns that have any kind of experience or credibility when it comes to evaluating talent and sit in the stands for a week and a half and watch a kid play.

They’re listening to what the GMs and scouting directors are saying. So, what do you think they’re going to say? They’re going to tell you that their players are awesome and they’re great defensively and they have massive talent because they want to pump the prospect in case they have to trade him at some point. The worst thing that can happen to a prospect is for him to come to the big leagues too soon, get over exposed, fall flat on his face and now everybody in baseball knows that the kid’s not the real deal.

It’s a garbage publication in my opinion. There’s some quality writing in there, but when it comes to the evaluation of players and how they rank them and all that kind of stuff that’s all fueled by the organization. They tell them what to write. They tell them who their top five prospects are.

While Zaun portrays himself as a bit of a clown, and goes to hyperbolic lengths on occasion, I can’t help but agree with some of what he’s saying here. However, his brush may be a little bit wide. There’s a distinct difference with how Baseball America collects their information and how rival publications, like Baseball Prospectus, put together their lists of top prospects.

When I initially looked into this, I also spoke with Kevin Goldstein of Baseball Prospectus, who told me about his own methods for compiling team rankings. He also amalgamates the opinions of scouts and industry insiders, but he will purposefully avoid speaking with members of the actual organization he’s ranking.  Instead, he specifically seeks out scouts from other organizations in order to get a less biased opinion of a player.

Most of Goldstein’s information is acquired through sources in the industry that, just like any other journalist, have been built up over time. Typically, it’s just a matter of being social with a scout, who just like everyone else, likes to share his opinions from time to time. Occasionally, it can become a two way street with information on opinions of one player being exchanged for other opinions.

This seems to me to be the better method, as it’s less dependent on information coming from individuals with a vested interest in how the quality of their prospects are portrayed. Of course, it’s not perfect, and biases will leak through from time to time, just as they will in any sort of opinion based writing, and it’s equally dependent on having a wide variety of sources providing information.

In a way, the entire issue that Zaun brings up offers us an important lesson. For fans who want the most accurate information, it’s important to read a wide variety of sources and do so while thinking critically and understanding where the information being provided is coming from. This doesn’t just go for baseball prospecting, either. The wider variety of opinions you gather, the better off you’ll be in understanding any argument on any issue.

And for the record, in 1993, Baseball America named Zaun, then in the Orioles organization, the best defensive catcher in the Double A Eastern League.

Comments (29)

  1. Goes to show how once again the lack of knowledge that Zaun has. Like anything. Look at stocks, each stock company pays an analyst for recommendation on a stock. If the valuation or recommendation is wrong they will lose credibility just like scouting directors with BA so the info has to be as accurate as possible. Just anytime where Zaun is bitter and trying to get some attention and pub like always. Zaun is a media attention whore

    • I disagree. A publication like Baseball America doesn’t have a lot of parallels to stock analysis. If a scouting director constantly gave bad opinions to his organization, you’d have a comparison, but the collecting of biased information and reprinting it to the public is quite different.

      • haha. parkes stick to baseball! most real hedge fund guys use sell side research as a contrary indicator that is fed to retail investors only to drum up trading and finance commissions. oh ya, we make money and retail investors lose it.

        • What does that have to do with anything? Also, define “most.”

          • parkes, i would agree that there are probably issues with how BA collects data… but let’s remember that this is an inexact science and if BA’s evaluations and rankings were consistently poorer than their competitors they would quickly lose their status as the gold standard in the field. it hasn’t happened.

          • I’d suggest that there’s such a lack of knowledge and the “evaluations” in general aren’t held up to much of a standard afterwards that there’s an incredible amount of room to avoid a reckoning for inaccurate projections.

          • @Ryan BA is in the prediction industry, many of the people who consume their product (us fans) do not look back at their previous rankings and how they turned out. Even teams might not (if they even use BA rankings). Humans want predictions and don’t usually revisit them. This is how Paul Ehrlich is still in the news predicting global collapse, even though his predictions in the 1960s were demonstrably false. Ottawa Citizen comunist Dan Gardner wrote a good book on it called Future Babble.

            @Parkes Has anyone done a statistical analysis of the prediction records of the various prospect rankings? If BA does OK, it could be that the biases cancel each other out (i.e., each team has similar incentives to over-hype their prospects, and each team is biased equally infavor of their own prospects)

          • What does it have to do with anything? Your assumption that just because Goldstein speaks with people outside a prospect’s organization, his information is less biased is not valid. Differently biased would be more like it. Also based on less observation than those within the org.

            Check out the details of Sarbanes-Oxley – a pathetic attempt to change the situation monkeyman describes in financial markets stateside.

          • it is an interesting discussion. my perception is that BA puts forth the most positive evaluations of prospects. individual scouts (like Law and KG) seem more likely to discuss a prospect’s weaknesses.

            i’ve been reading BA for many years and i like the fact that they talk to alot of sources for each individual prospect… and they have been in the business long enough to develop sources that they trust. if a GM or scouting director were trying to oversell his player i trust that this opinion would be an outlier… and they probably wouldn’t be seeking opinions from this GM or SD again.

    • There are very few examples that I’ve ever seen of a terrible recommendation coming back on an analyst. That was kind of the whole point of the 2008 recession.

  2. One of Zaun’s 3 Ways to Win prior to a game in the Philly series was “Find a way to win.”

    • +1 for mailing it in. I’m really going to miss Alan on TV.

    • for a series a couple weeks ago it was ‘keep shaking hands’ ie. keep winning the winning streak going.
      his key to winning was winning.

  3. At least he washes his hands, though. Either that or hand soap is one of his 3 Ways to Win(ning Hair).

  4. Gregg Zaun’s preferred method of ranking prospects: just sign the bottom of a blank evaluation sheet, deny all accountability for whatever ends up being written on it.

  5. I just have to say how much I like that picture, its as if it was taken by surprise after he got out of a stall except he noticed at the last second and tried to make it casual.

  6. Gregg Zaun is Baseball Don Cherry (puke!), at least his shoulders are shoulder width apart. Amirite?

  7. No sarcasm.

    Really, really, good post Parkes.
    Appreciate the insight and the writing.
    I learned something new..

  8. Fuck off you gay idiots. Zaun will unload his spunk on all your bitch asses. Eat up, fucking losers. And Parkes … come to my house, and I’ll let you smell my asshole.

  9. It must be that time of the month for Zaun

  10. The Jays just have Buck and Pat interview AA during innings to hype prospects. Same thing your hearing how great these players are from their organization.

  11. The other thing that isn’t mentioned here is that Zaun was badmouthing BA because he doesn’t think that D’Arnaud belongs in the big leagues. According to Zaun, D’Arnaud’s defence is suspect, and he’s not ready for the show. While he may be right, and Parkes, you raised some very valid points around the potential issues with BA’s approach, I don’t see how Zaun is really getting more valid information about this player, or any other. Is he on regular flights to Vegas to watch these guys? Or is he, too, relying on information being passed to him by the folks who know D’Arnaud best, that is, Blue Jays front office staff?

    Also, that was a great piece of irony about Zaun being ranked so high in 1993. Well done.

    • Zaun clearly states in other interviews, ETC he watches a ton of film. Minor league tape would be very easily excessable to somebody like Zaun. I too have read reports about Travis’s D still lagging behind a bit and that their working on his throwing etc.

      But my point is, who in the media better to critique D’arnaud then Zaun? If you don’t like his opinion, don’t agree but he’s also somebody whose fresh in the game, not like Buck who hasnt caught in 30 years and forgot what it was like 20 years ago, or Wilner or , or , or etc !

  12. Like fucking Gregg Shitbag Zaun’s analysis of prospects is measured and quantifiable. Fucking hypocrite.

    He’d gladly rant about the opposite if BA’s opinions had supported his warped entrenched opinions.

    Reason = evidence ———-> opinions, conclusions
    Gregg Zaun = opinions ————> evidence that suits them

  13. Zaun isn’t completely off base. Problem is he is about as gentle with his words as Lennie is with puppies.

  14. Excellent post, Parkes.

    I would never defend Gregg Zaun as a media personality. I just don’t enjoy his attitude or style, and like many others I tend to think he’s stuck in the past.

    But I also don’t respect people who comment as if they are experts, when their opinions are based strictly on things they’ve read. How can one refer to Zaun as an idiot when your own baseball opinions are based entirely on those of Keith Law or Kevin Goldstein or whomever your favourite analyst happens to be? Regurgitating some other expert’s opinion doesn’t make you an expert. So, don’t act as if you are smarter than anybody else if you haven’t seen the players yourself and done your own research.

    Agreeing and disagreeing are both fine. Calling someone stupid strictly because they disagree with your favourite analyst is ridiculous.

  15. Greg Zaun accurate. Great take.
    Garbage. Sells papers and hurts young players and the game itself. In a period of time where 17-22 year old potential means more than a ML’er. The reason? People don’t know.

    • “Sells papers and hurts young players and the game itself. In a period of time where 17-22 year old potential means more than a ML’er. The reason? People don’t know.”

      is this a joke?

      how exactly does it hurt ‘the game itself’ to evaluate the potential of a minor league player?

      secondly EVERYONE knows why prospects are valued. clubs hold their rights and control their salaries for 6 mlb seasons.

  16. “The wider variety of opinions you gather, the better off you’ll be in understanding any argument on any issue.” – Too simplistic. That would be true of a logic engine, but not the human mind. Cognitive dissonance & recency biases outweigh the marginal benefits of additional information at some point, leading to negative value for each additional piece of info. There is an optimum amount of information which depends on the thought processes involved and the individual.

    Gad Saad of Concordia has done great work in this space (check out: The Evolutionary Bases of consumption, if curious).

Leave a Reply

Your email address will not be published. Required fields are marked *