The Truth About Prospecting

It seems to me that the closer one looks into a subject, the more particular one’s taste becomes.

If you study to become a sommelier, you tend to appreciate wines other than the ones you used to drink too much of in university.  If you watch enough television, you can’t help but begin to shy away from the three camera sitcoms in favour of less traditional choices.  If you listen to a lot of pop music, you’re likely going to develop an interest in more obscure acts.

While some may attribute this to simply becoming a snob in your chosen area of expertise, I see it as a natural progression that keeps your interest piqued in a subject you’re passionate about.

After years of appreciating the game of baseball at the Major League level, I’ve slowly developed an interest in prospects and player development.  However, I fully realize I’m not clever enough to spot the difference between a blue chip prospect and merely a good one through observation alone.  And I haven’t been exposed to nearly enough Minor League Baseball to form opinions of my own when it comes to player development.

Instead, I’ve come to trust certain voices in the sport that are recognized as authorities when it comes to prospects.  In addition to providing baseball nerds with excellent statistical analysis, websites like Baseball America and Baseball Prospectus publish annual lists of the best prospects in each team’s system, and their word is generally accepted among baseball fans as being, if not Gospel, than certainly reasonable.

Nevertheless, the recent history of baseball metrics compels fans to constantly question that which is generally accepted about the game.

I first began to wonder about prospect rankings last week when Baseball America issued their top ten list for the Toronto Blue Jays organization.  Admittedly, I’m a Blue Jays fan and I know the organization and their system better than any other team’s in baseball.

Specifically, I questioned the rise of Anthony Gose to number three on the Jays list from number six on the Phillies, Deck McGuire’s position as their number two ranked prospect, the lack of Adeiny Hechevarria and the glaring omission of Brett Cecil from the projected 2014 rotation.  I also wondered how Gose could be named the organization’s fastest player, best defensive outfielder, with the best outfield arm, yet lose out to Jake Marisnick as the team’s best athlete.

I scoured the Baseball America website to learn exactly how these lists were compiled, looking for more than the explanation that the lists “are based on projections of a player’s long-term worth after discussions with scouting and player-development personnel.”

I eventually learned, through a contact from Baseball America, that the writers putting these lists together, are in fact, not scouts, and in most cases haven’t even seen the players that they’re “evaluating.”  Their lists are all second hand information.

Perhaps I’m being naive, but this was staggering to me.

While I understand that this evidence is somewhat anecdotal, to make sure I wasn’t alone in my ignorance, I asked every baseball fan I came in contact with over the next few days about Baseball America.  Not one realized that the website’s popular lists were being put together by guys who had rarely, if ever, seen the prospects they were offering opinions on.

Through my contact I learned that while each writer and editor has varying methods for compiling their list, most start with Baseball America’s Top 20 Prospect Lists for each Minor League, which has previously been compiled by amalgamating the opinions offered to them by actual scouts and league managers.

Again, I emphasize that none of this is based on Baseball America’s own evaluation, but rather a collection of what they’re hearing from their contacts in the industry.

The list compilers take that original information and then speak with several members within the player development department of the organization they’re ranking, including assistant GMs, scouting directors, pitching/hitting coordinators, pro scouts as well as a few others outside of the organization.

When I asked my contact if there was any concern over relying too much on the biased information coming from the clubs that they were supposed to be unbiasedly evaluating, he replied:

Certainly an organization will be higher on a guy because he’s in their system. The people I talk to I really trust so while I do temper their excitement some, it’s never really a lot. I haven’t had anyone tell me a guy is 95-98, when he’s actually 93-95.

You can even tell in their voice sometimes. They can tell me all they want about how much they love this guy, but then when they answer questions about specific tools you can tell that he shouldn’t be quite as high.

The flaws in this type of system are immense.  In addition to asking readers to trust the writer’s ability to discern the bias of the organization, we also must trust that his contacts are plentiful enough to get an accurate overall picture.

Considering how large their fleet of scouts is, the Toronto Blue Jays offer a perfect example of how something could fall between the cracks or appear to rise from the crevices.

With more scouts in the organization, each one sees a smaller quantity of players.  The writers at Baseball America could have the scout that recommended Anthony Gose to Alex Anthopoulos as a contact, but lack the scout who touted Adeiny Hechavarria, and we, the readers, would have no idea.  It can also explain why Kevin Ahrens was ever included in any top prospect conversation.

I also spoke with someone from Baseball Prospectus who compiles his lists in a similar fashion to Baseball America, through amalgamating the opinions of scouts and industry insiders, but he avoids speaking with members of the actual organization he’s ranking.  Instead, he goes so far as to specifically seek out scouts from other organizations in order to get a less biased opinion of a player.

When I asked my contact what benefit the scouts get from information sharing he said that it’s primarily done for social reasons.  Just as you and I like to talk about the games we’ve been to, the players we’ve seen, so do scouts.  However, there’s also an element of information that can go the other way, with writers taking advantage of their own network of contacts to exchange information with other scouts.

I wonder how much more accurate this method is.  If Baseball America is merely regurgitating the news releases of the clubs they’re supposedly evaluating, then it sounds as though Baseball Prospectus is giving a platform to their rivals.

Again, this method only works when the writer has a large array of contacts within the scouting world that he can trust, and while that may very well be the case, there are undeniably still flaws in this system.

The Toronto Blue Jays once again offer a perfect example of how things might go wrong, given the team’s new found aversion to speaking with media members.  If the same lock and key that guards the mouths of the team’s executives also governs the availability of their scouts to members of the media, are their voices simply not being heard by Baseball Prospectus?

What if one were to only get the opinions of the particularly chatty people in your office?  Would their viewpoints accurately represent the reality of that office?

I don’t mean to suggest there’s nothing of value that these two websites bring to their readership.  Baseball America and Baseball Prospectus are every day visits for me because of their ability to provide statistical and performance based analysis.  There’s no question in my mind that both publications have earned the respect they have in the baseball community.

My contact at Baseball Prospectus posed a very interesting question to me while we were discussing his methods.  Would I prefer to read the opinion of a guy who has seen a player one time or a guy who has talked to several people who’ve seen a player forty times?

I once read a quote that brought up all the issues that make the theory of evolution questionable, but at the end of the statement the author confirmed that, despite all of its drawbacks, it remains the best theory we have to explain our existence.

I’m not convinced that either Baseball America or Baseball Prospectus’ scouting methods are the best we have available, but there are no doubt flaws and biases in every method we have for prospect evaluation.  What’s most bothersome to me is the perception that exists among baseball fans that these websites are providing actual original talent evaluations, when in reality they’re relying directly on sources that in all likelihood are biased.

The perception exists that the writers for these publications are more scout than journalist, but the reality is very much different.

While I’m a firm believer that readers should take responsibility to properly weigh, judge and use critical thinking with anything they read from any source, websites also have a responsibility to provide a level of transparency in explaining how they derive at the conclusions that they publish.  When a false perception exists, even if it benefits the authoritative voice a publication is trying to create for itself, it should be corrected, lest the wrong impression grow.

As a first step, I would like to see both organizations be more forthcoming about the process for compiling their rankings and scouting reports.  Short of naming names, some sort of list could be provided relaying what organizations or what positions within an organization provided input into their rankings.

Secondly, if the writers aren’t claiming to be scouts, why are they including their own input at all?  Their performance analysis can certainly be valuable, but by mixing it with a smattering of other people’s skills analysis, they’re confusing their own role to readers, and at times, it seems, themselves.

Why not ask their contacts to give numeric scores on the players they’re ranking, and then tabulate the results?  “We spoke with fifteen scouts and six executives in charge of player development about the Toronto Blue Jays farm system, here’s how they ranked the top prospects.” It’s not going to be unbiased, but at least there’s no doubt about what role the writer is playing in ranking the players.

In the grand scheme of things, the way in which two websites rank 20 year old kids on how they play the game of baseball probably isn’t important enough to spend 1800 words writing about, but baseball can inspire people to do funny things, not least of all, get things right.

Comments (27)

  1. Don’t forget that both of these publications were completely wrong about JP Ricciardi’s ability to draft.

  2. What I learned from reading this is that Anthony Gose is going to be AWESOME.

    Great post, Parkes.

  3. Interesting that they hate Ricciardi’s picks when their main sources are scouts, who, let’s face it, probably hated Ricciardi considering his first order of business as Jays GM was to decimate the scouting staff.

    Anthony Gose is the best thing since sliced Wieters.

  4. fuck off Parkes! … no, wait?

    Great read Parkes.

    While I did know that BA operated in this manner and I have always looked at their content and rankings with skepticism(along with every prospecting site), I enjoyed the article and I wholeheartedly agree with your conclusion that it would be best for BA/BP to be more open about their methods and potentially present an unmodified source of information straight from the collective scouts mouth as it were.

  5. Wow, this is a fantastic post, and dare I say, great journalism. Thanks for sharing the background on Baseball America and Baseball Prospectus’ methods.

  6. Dustin -
    My name is Conor Glassey. I’m a writer for Baseball America and I’d like to answer a few of the questions you posed, as well as provide a little more insight into our process.

    • Like you touched on, the process for our organizational Top 30 lists begins with our League Top 20 lists. For example, when I do the Northwest League Top 20 list, I talk to all the managers in the league, as well as some pitching and hitting coaches in the league and scouts that saw the league as part of their professional coverage. We talk about players that are likely to make our Top 20 list, but also about other players that, while they aren’t good enough for a league top 20, are still considered prospects. After our League Top 20s are done, we pass along our team-by-team notes to the writers doing those teams’ Top 30 lists. So, for example, in the Northwest League, there’s affiliates for the Padres, Rangers, A’s, etc. So, I pass my Padres notes on to Matt Eddy, who does our Padres Top 30. I pass my Rangers notes along to Aaron Fitt for his Rangers Top 30 and I pass my A’s notes on to Jim Shonerd, who does the A’s chapter. And so on. Those notes are just a starting point for us to begin our calls. And, yes, we do talk to a bunch of people within the organization we’re covering. Obviously they know their prospects better than anyone. But, there’s a common misconception that we ONLY talk to people within the organization. That’s not true. We talk to many scouts from outside the organization that saw the players we’re writing about. There’s also a common misconception that Baseball America uses only tools and not stats when ranking players. I can tell you that’s not true, either. We certainly put a lot more weight into a player’s tools, but we do also look at stats. Our lists try to weigh consensus view of how the players stack up, oftentimes with more weight given to a player’s upside. While each list is reported and written by one individual writer, they’re all a collaborative process around the office. But, at the end of the day, if it’s your name going next to the list and there are going to be some biases that come into play, which is natural. This is true of any ranking that is based purely on opinions. Scouting is more art than science and scouts are paid for their opinion. Can someone’s opinion be wrong? You may disagree with an opinion, but that doesn’t mean you are right and the other person is wrong, it simply means you have a difference of opinion.

    • Although four former writers from Baseball America have gone on to become full-time scouts (including Josh Boyd, who is now the director of pro scouting for the Rangers) and one intern went on to become an associate scout, it’s true that none of our current writers are scouts. We do our lists journalistically, like I outlined above, by talking to a lot of people who have much more baseball knowledge and experience in player evaluation. I’m not sure what kind of method you would propose that you think would be better. Essentially, anyone ranking prospects that doesn’t work for an MLB team is going to be using “second hand information,” as you call it.

    • You make it seem like Baseball America writers never go to baseball games. Again, not true. We are based in Durham, N.C., which is a fantastic area for fans of prospects. Within an hour or two, we have access to the International League (AAA), Southern League (AA), Carolina League (Hi A), South Atlantic League (Lo A) and the Appalachian League (Rookie), plus ACC college baseball and a great area for high school prospects. We also have the Team USA National Training Complex 10 minutes away from our office, so we get one of the best summer high school showcases, the collegiate national team and the professional team (which, this year, included guys like Mike Trout, Brett Jackson, Mike Moustakas, Eric Hosmer, Mike Montgomery, etc) right in our backyard. We go to a lot of games and, while we don’t write based on our own scouting opinion, it gives us the opportunity to talk to players and coaches, as well as meet scouts at the games. My main focus is on the draft—although we all do a little bit of everything at BA—but I’ve gone to more than 100 games a year since I’ve been out here.

    • Finally, you mentioned that you’ve only recently become interested in prospect rankings. Are you aware that Kevin Goldstein, the prospect writer at Baseball Prospectus, used to work at Baseball America?

    I hope that provides some more insight and transparency into Baseball America’s process. Please let me know if you have any other questions.

  7. Hi Conor, thanks for the reply. I don’t see how any of this is different from what I wrote. And your response completely dodges what I’m taking issue with.

    I’m not saying that scouting is a scientific process or that BA doesn’t see enough games. I’m saying that the impression exists that BA and BP are more scouts than journalists, and that simply isn’t true. You guys are trusting sources, not creating your own opinions. And a good chunk of the sources you’re trusting, are heavily invested in how you report on them.

    It’s simply not a good method.

  8. 1) It is different from what you posted because instead of trashing our method, I’m standing behind it. I gave you a more in-depth look at our process and emphasized that Baseball America talks to people both within and outside of the organization we’re covering. We’re not just mouthpieces for the organizations we cover.
    2) You did seem to want to make the process of ranking prospects more scientific when you said, “Why not ask their contacts to give numeric scores on the players they’re ranking, and then tabulate the results?”
    3) I don’t know why anyone would think that the writers at Baseball America are scouts instead of journalists. And I disagree that the majority of our readers believe that.
    4) Not a good method, huh? I’d love to hear what you think a good method would be. I enjoy watching players as much as the next guy, but I think I’ll stick to trusting the experts when it comes to evaluating and ranking.

  9. You’re coming across as either a bad scout or a naive journalist. Neither option makes BA’s prospect rankings something I put much stock in.

    The first person I spoke with at BA, whom I quoted directly also said that he spoke with “several people in the organization’s player development department. People like assistant GMs, scouting directors, pitching/hitting coordinators, pro scouts as well as some others outside of the organization.”

    Where do you think the emphasis is there?

    I would prefer to read something more objective than subjective. That doesn’t mean scientific. What other journalist is going to get away with merely saying he’s maybe or maybe not spoken with some of these people, and based on what they may or may not have said, I’ve decided to rank players thusly?

    If things were numerically rated, which they usually are in the world of scouting (where else did the 20 80 scale come from) it would give readers a far more objective view from the sources that are being collected.

    I realize it’s anecdotal, but every baseball fan I spoke with was shocked over the difference between how BA portrays itself and the actual method it employs.

    I already explained in my piece exactly what a better method would be.

  10. I enjoyed the post, and I’m enjoying the back and forth here in the comments.

    for what it’s worth and that probably isn’t much, I always kind of assumed that publications like BA and BP had some more inside knowledge going into their rankings than just talking to some guys. Not trying to trivialize what these people do because there definitely is value in their work, but Parkes isn’t totally wrong when he calls the methods more subjective than objective, at least the way they’re explained here.

  11. As a BA reader for many years I have to say that I’ve never once been lead to believe that BA’s writers were scouts. And I have no idea why any baseball fan, after reading what BA writes, would get an impression otherwise. To say that BA protrays itself as such is completely untrue and unfair.

    I also disagree that “it’s simply not a good method”. Clearly no system is perfect, but it’s very easy to throw out a comment like that without suggesting a better way. The guys at BA and BP are pretty smart guys and have a lot of experience putting together these lists. If you say their methods are not good, then it’s only fair to ask you what a better method would be.

    • I’ve already explained in the piece exactly what would be a better way, so I’m not merely throwing out a comment like that without suggesting a better way.

      Perhaps you’re far more aware than myself and everyone I’ve spoken with about how BA does their rankings, but it’s not untrue or unfair to suggest that they portray themselves as more of an authority than they truly are. As I said earlier, their method, which I’ve explained in the article and has been confirmed by someone (disagreeing with me) in the comments section, makes them either bad scouts or lousy journalists. I don’t know how you can see that method explained and think otherwise.

      Again, as I’ve said throughout the piece and in the comments section, I believe that BA and BP have a ton to offer in other areas. Their prospect rankings though, make absolutely no sense to me whatsoever, and how anyone could see them as objective is completely beyond me.

  12. Dustin – I think the point you’re failing to understand is that ranking prospects can never be objective. The rankings are based off of people’s opinions of the prospects, so they’ll always—by definition—be subjective.

    While we don’t come out and say (for example) “13 of the 15 people we talked to thought Drabek was the Blue Jays best prospect,” we definitely run our list by the people we talk to and get feedback on where they think guys should rank.

    But the process can’t be 100% transparent to the public. Because team executives and scouts are paid for their opinion, they’re oftentimes not supposed to share that opinion with the media. We’re fortunate that many do share with us because we have a strong reputation within the industry after being a magazine for 29 years.

  13. I understand all those things. But this is my point: You can’t say you’re not scouts, but then follow none of the rules of journalism; and you can’t say you’re not journalists, but then rely on other people’s opinions as your source for “scouting reports.”

    You say that the process can’t be 100% transparent, but it’s simply not transparent at all. Readers have no idea where you’re getting your ideas on these players. You’re asking for a lot of trust from your readership, and I think that it’s a blind trust that many (Dave not included) have been misled into following.

  14. In Fairness to Aaron and B.A., this preamble goes at the top of each top 10:

    “Baseball America’s Top 10 Prospects lists are based on projections of a player’s long-term worth after discussions with scouting and player-development personnel. All players who haven’t exceeded the major league rookie standards of 130 at-bats or 50 innings pitched (without regard to service time) are eligible. Ages are as of April 1, 2009.”

    Seems that they’re pretty clear about where they get their information, and the reason for not revealing their specific sources is of course reasonable.

    Also from my recollections, most of the time in their chats, the BA guys will be honest about not seeing a guy in person if they haven’t, stating ‘scouts tell them so-and-so’.

    Big fan of BA, have been using them in my deep keeper league for a decade and last year drafted Mike Trout, Brett Jackson and Zach Britton thanks in large part to their reports.

  15. Oops, I meant Conor, not Aaron obviously (Aaron Fitt did the list I looked at).

  16. Dustin – We DO say we’re not scouts…because we’re not scouts. You won’t find anything written on our site where we claim to be scouts. Using anonymous quotes is allowed in journalism and it shouldn’t be that hard to understand why we protect the identities of our sources.

    Also, when you say “…you can’t say you’re not journalists, but then rely on other people’s opinions as your source for ‘scouting reports.’ ” That’s nonsense and makes me question if you even know what scouting reports are. Scouting reports ARE opinions. There are very few facts in scouting. If a guy has a 95 mph fastball or if he runs a 6.56 60-yard dash, that data can be presented as fact. But when scouts put grades on players using the scouting scale or project what kind of player a high school kid will turn out to be, it’s all opinion. They’re educated opinions, but they’re still opinions. Scouting is very subjective. Scouts are paid for their opinion.

  17. We seem to be arguing in circles.

    Once again, I understand that scouting is based on opinion, but there are grades that are used, the 20/80 scale, to assign numeric values to the tools being judged. While subjective, quoting these would be a hell of a lot more objective than picking and choosing the info you’re gathering from your sources.

    Would you at least agree that the style in which BA rankings are written is not your typical journalism? No journalist in the world would get away with the liberties the rankings take with sourcing their information. Again, I understand that sources can’t be identified, but there are ways to give the reader information on whom you’ve spoken with while not outing them.

    I think our largest disagreement is over the voice of authority that BA has. Whether you think so or not, prospect rankings are being read by baseball fans as though it’s the writer’s opinion, which is based on their own ability to scout the players, when in fact he’s arbitrarily accumulating the opinion of others (a lot of which are biased). This is what I take issue with, and that’s what I’ve written about here.

  18. The fact that BA’s disclaimer says that all ages are as of April 2009 should give you all the evidence you need as to how seriously that’s taken. Frankly, it’s not good enough to just say “we may have spoken with these type of people” to derive at the opinions that are being published, when it the great many instances, they’re talking to people how directly impacted by how the prospects are ranked.

  19. Yeah, I’m tired of arguing in circles too. We obviously write about players’ tools in our scouting reports. But, instead of just showing number grades, we use words. Average is 50, fringy is 45, below average is 40, above average is 60, well above average is 70, etc.

    I disagree that our approach isn’t typical journalism. Obviously it’s apples and oranges when you’re comparing baseball prospect rankings to covering a political race or the war in Iraq. But the method we use—talking to numerous experts about the players we’re writing about—is the journalistic way for ranking prospects.

    Obviously each writer has some input over where they rank the players they’re ranking—but it’s based off of the conversations we’ve had, and not just arbitrarily put together. And, again, I disagree with your assertion that most of our readers believe we’re the ones scouting the players. It’s also funny that—in a post ripping BA for not citing specific sources—you back up your argument by talking to “every baseball fan I came in contact with over the next few days.”

  20. Dustin – I enjoy your work, but I completely disagree with your opinion here. I also disagreed with it when Kevin Goldstein discussed it in last week’s (I think) podcast.
    I much prefer a disseminated opinion from many points of view, than from one guy who can only see so much before his opinion is too diluted to be of value.
    Furthermore, Kevin, nor anyone from BA have ever claimed to be scouts, nor the AUTHORITY on scouting. As another commenter pointed out, it’s written pretty clearly in their guides. If they were more explicit regarding who they talked to, we’d likely find that less people talk to them and what they can provide us is of less value.
    Even GMs are limited in their ability to form cogent opinions on players. They relay on networks of scouts to get that opinion to them. GMs most often only actually scout the upper tier of potential draftees. Are their opinions untrustworthy?

  21. I don’t usually comment but people are ganging up on Parkes in this thread and it’s complete bullshit. This piece didn’t trash BA or BP at all. I think it merely hit a nerve and instead of arguing the real issues that Parkes brings up, Goldstein and the group thinkers change the argument into whether or not they add value. That’s not the argument. It’s that the sources that are being used to compile rankings are directly impacted by the rankings that come out. Added to this relationship is the fact that the writers using these sources don’t make any mention of who they’ve specifically spoken with. This BA guy is trying to call himself a journalist, but no respected newspaper, magazine or website would publish his “findings” in the style that his website publishes it. It would be thrown out by fact checkers immediately as being too subjective and without direct quotes or references to who is saying what. In other words, it is completely worthless as objective analysis. Parkes also makes mention, and prefaces it by saying he knows it anecdotal, that everyone he’s talked to was surprised that BA and BP rankers weren’t actually scouts writing their own opinion.

    If I was really into wine, would I read about what other people said about wine based on what other experts said, or would I read about other people’s opinions on wine who have actually tasted it? Which is the more valuable analysis?

    Parkes has uncovered something foul here.

  22. So I guess Woodward and Bernstein were full of shit too?

    I get wanting more info on the sources, but they do need to protect the identities and Baseball is a small world. To get into positions instead of names would be the same as naming them as I’m sure most teams don’t have multiple scouts looking at players in the Eastern League.

    If people want to look at movie reviews, people go to Rotten Tomatoes and Metacritic. Their data is more transparent because the info isn’t from ‘insiders’. Prospect analysis does come from insiders so they have to protect their sources. Personally think both do a decent job of not being mouthpieces, unlike some well respected reporters who regularly get ‘scoops’ and ‘exclusives’ which tend to be wrong more often then they’re right.

  23. Parkes has pointed out a flaw in a sacred cow and now all the poor schmucks that use BA and BP for their own opinions are scrambling to discredit him.

    Someone raised a good point previously when they said that if BA and BP want to be considered journalists, they should write and credit their sources like journalists. Read any of the rankings on BA and they come across their own reports, not like they’re using sources at all.

    The fact that these guys source out the organization that they’re supposed to be judging for information makes them a bigger joke than I ever imagined.

  24. At the very least they should tell you who they talked to or provide direct quotes to “an unnamed source” but they don’t even do that.

    Comparing W&B to BA or BP is like comparing a steak to a McDonald’s hamburger.

  25. Obviously Woodward and Bernstein is hyperbole but it’s a valid point. Why are BA and BP being held to standards that much of the journalistic community are not, specifically guys like Gammons and Heyman(and a lot of the sports community) who will regularly use the phrase ‘Sources tell me’ and get themselves all sorts of exposure for a story that will 9 times out of 10 turn out to be bull shit.

    Call them my sacred cow all you want, I’ve been reading these sources for over a decade and have stuck with them for the solid information they’ve given me over the years. If they sucked at what they do, do you think I’d still be using them for my deep fantasy league (24 team, 44 man(minimum 8 minor leaguers), full keeper).

Leave a Reply

Your email address will not be published. Required fields are marked *