While some people view sabermetric bloggers (with some justification) as “arrogant,” the truth is that much of sabermetrics is about what we still don’t know about baseball — that is, most of what there might be to know. I believe it was Bill James who called this “measuring the fog.” Sometimes “The Fog” is misused.

Debates between saberists often follow a similar script: one person holds a position, another person disagrees but lacks evidence why, introducing “The Fog” in order to dismiss uncomfortable results. That is a topic for another time. It is enough to remember that sabemetrics is as much about what we do not know as much as it is about what we think we know.

Keeping this in mind, we see that statistical approximations of the true talent of players, usually called “projections,” are not arrogant predictions, but rather are humble attempts to get a general estimates that get most players basically right, or as “right” as possible. Regression to the mean, weighted averages, and general age adjustments are thus not methods of divining the future, but rather rational and pragmatic means of reducing the error of projections.

However, we bloggers also like to talk about general managers as being “good,” “bad,” “a genius,” or when we are feeling particularly cranky, “incompetent.” Part of this is the nature of blogging (as opposed to simply doing sabermetric research). it is still a developing form that grows not out of a journalistic or scholarly ethos, but rather simple individual personal expression, in this case, of mostly rabid baseball fans. So when we want to express ourselves, we do so using rather, well, “non-neutral” terminology.

I have been as bad as any “fan blogger” out there, so I am not going to give a stern lecture on humility and what we do or do not know. After all, I have no objection to bloggers playing GM. Everyone makes evaluations in all areas of their lives based on evidence that may not be terribly sufficient out of simple pragmatic necessity. However, as the season is upon us and people are about to get very angry and/or happy about their own teams moves, I simply want to go through some reminders about the limits of any sort of objective estimation of the true talent of general managers.

Let me note that I use “general manager” as a shorthand for the front office as a whole. Baseball teams have never really been run by a single executive, at least not since the minor leagues were mostly subjugated to the majors (Branch Rickey seems to have been some sort of visionary, ever heard of him?). General managers cannot be everywhere at once, they have relied on a bevy of scouts and cross-checkers for decades.

The influx of more objective and statistical information to be organized and analyzed has not, in an of itself, created the need for larger front (and back) offices, only added to it and expanded it into different areas. To simply say that a particularly general manager is good or bad at his (and hopefully one day, her) job has to be taken as a comment on not only the general manager, but the team’s executives as a group. The general manager is probably the single-most important person in that group, given that he usually sets the tone and hire the top assistants who organize things down below, so it is fair in most ways to focus on that person, but it should always be read as shorthand.

Back to the issues at hand — estimating the “true talent” of front offices. There are a number of things that go into player projections. How would this work with general managers?

The most obvious method might be to see what teams have won the most games in recent years. However, most people would reject that right away. For one thing, vastly differing payrolls in baseball mean that teams are not playing on a level ground. This is usually overemphasized by casual fans and grandstanding journalists (it is especially fun when they try to make the contest between teams owned by bajillionaires sound like some sort of social justice issue. However, it is relevant here. Most people understand this, which is why, e.g., Billy Beane is (was?) still lionized by the sabermetric set years after the A’s stopped winning — the financial context in which he operates meant another adjustment was due.

Finances is one big issue, but it is just an example of the sort of context that needs to be taken into account. But even that is not what I want to focus on. We know that finances matter, I want to talk about the areas of “fog” around out evaluations of GMs. There are many, but in the interest of length, I will just mention three more.

First, there is the problem of separating process from result. This goes to the issue of “who has won most recently,” too. Does this mean that the Cardinals and Rangers are the two best-run teams in the league? Actually, one could make a case for each of those organizations being exceptionally well-run, but that is not the point. How about the 2010 Giants? Well, does anyone really think that Brian Sabean has put together the best-run team in baseball? At least that year? Now, the Giants have obviously done a lot of things right, and I do not want to get into a bunch of big debates about it, but I do not think many close observers would say that the team is one of the best.

The “result” the 2010 Giants produced was obviously good, but how about the process by which it came about? Does anyone thing that they really knew that Aubrey Huff and Pat Burrell were going to have big years in 2010? The Giants clearly did not, since they offered Adam LaRoche bigger money first, and rather than trading the Rays for Burrell, they picked him up off of waivers. This is not to discount it all as luck, but obviously there was some at play there. We regress players to the mean to avoid random variation, so in principle, we would like to do the same with front offices, but exactly how that would work in practice is a difficult to work out.

Second, there is the “anonymous club official” issue. Secrecy is a big part of any competitive business in which information is crucial, and baseball is no different. I enjoy making jokes about the use of “anonymous scouts,” but I do not want any journalists reading this to get the wrong idea — that is the was the baseball reporting business has to be. However, one must also acknowledge that is makes the information gleaned in such a manner highly questionable, as there is basically zero accountability at work. An “anonymous club official” can say anything, basically, since if he or she turns out to be dead wrong, they cannot be personally mocked [Side note: remember this next time you hear about bloggers not being “accountable enough.”] for, say, telling a reporter in this middle of 2008 that Zack Greinke would be better used in the bullpen.

I do not want to belabor that particular point. It is just meant as an illustration of the necessarily secretive nature of baseball operations. Teams are spending more time and money on analysis than ever before, they are not going to give their secret formulas away. This goes back to an earlier point — we want to see how the process connects with results, but we really know very little about the specific processes teams use, only generalities, and in some cases (e.g., the Rays), not even that.

Finally, there is the possibility that GMs change over time. This is also something we deal with when projecting players, which is why we use weighted averages of the most recent seasons as well as age adjustments when estimating their true talent. One would think we should do the same with GMs. However, while players do change their true talent, it seems that in principle they are much more limited by their physical constitution than front offices are by their outlook. A player can talk about changing his swing or whatever, but ultimately, he is limited by his bodily makeup as to what sort of bat speed he can attain, for example.

With front offices executives, they might have limits to their intelligence (an executive cannot simply decide to add 20 IQ points over the summer), and people cannot go to special center to change their personality (an aspect of running an organization we nerds tend to underrate) overnite. However, starting a department simply to look at Hit f/x, for example, and being willing to take what comes out of that seriously, can make a big difference. That sort of radical shift is potentially significant to the extent that is it hard to imagine.

One more obvious shift that has already taken was in which GMs in general have changed is in prospect valuation — think of the Mariners under Bill Bavasi trading a 1B/DH platoon– Ben Broussard and Eduardo Perez — to Cleveland for in two separate trades two prospects: Asdrubal Cabrera and Shin-Soo Choo… during the same season. It is difficult to imagine even one of those trades taking place between any two teams in baseball now.

Part of that is Bill Bavasi being fired, but that is also the sort of trade Brian Sabean would have made five years ago. In 2010, though, the Giants won the World Series largely because of a home-grown pitching staff that Sabean did not trade away. I am not sure where to rank the Giants’ front office as I write this, but something radical clearly changed in their overall thinking, even if they still seem to be going out of their way to bury Brandon Belt.

For all that we can infer from observed processes and results, there is so much that we do not know. Four years ago, Dayton Moore and the Royals signed Jose Guillen to a three-year, $36 million contract that ended just as most thought it would — in disaster. Prior to 2011, Moore signed Melky Cabrera and Jeff Francoeur to small stopgaps deals, and each responded to very good years. Did this represent better thinking on Moore`s part? Was there a change in organizational philosophy from big, long-term deals for veterans to smaller, low-risk deals? Did the scouting improve? Did the team finally figure out what “building for the future” really involves? Or was it simply that ownership held the reigns more tightly after Moore’s “wilderness years?”

I am not going to stop evaluating GMs. It would be easy for me to look at my past mistakes and now take on an air of detached superiority and tell people they simply “do not know” enough to say one way or another if a front office is good or bad. We make decisions and judgments every day based on tainted and vague evidence. That is simply that pragmatic nature of the human condition. But it is still good to acknowledge those limits from time to time.