There are few things more frustrating than watching the lead that your team built through seven or eight innings of baseball wilt away at the end of a game due to the ineffectiveness of a pitcher whose sole responsibility is not to allow that to happen. It’s as maddening as it is memorable.

Unfortunately, it happens to even the best teams and the best relievers. Since the beginning of the 21st Century, only three teams have finished a season with less than double digits in blown saves: 2003 Los Angeles Dodgers, 2004 Los Angeles Dodgers and 2008 New York Yankees. Last month, Drew Fairservice compared the number of blown saves that the Toronto Blue Jays have accumulated this year to the Atlanta Braves, who have the best bullpen in baseball, and the number was staggeringly similar.

As much as fans may want to look at the number of blown saves and add that number to their team’s win total while imagining what could’ve been, they also have to add the blown saves to the win totals of every other team in their division, and account for the times that their team came back to win a game after a blown save and deduct the amount of times that a blown save has occurred twice in the same game. It’s just not as simple as many would like to believe.

As Mr. Fairservice outlined in his post, blown saves aren’t the best way to calculate a bullpen’s worth, as there are several other methods to measure the successes and failures of a reliever based on the leverage of the situation in which they’re pitching. Back to the Blue Jays as an example, despite 19 blown saves this season, their bullpen is fairly middle of the road when it comes to just about every other measurement: WAR, FIP, Clutch, WPA and shutdown to meltdown ratio. In other words, the blown saves have more to do with bad luck surrounding the timing and sequence of hits and runs that the team is giving up rather than any individual’s poor performance levels. There’s also a large gap between the worst pitchers in high leverage situations this season (Frank Francisco and Jon Rauch) and the best (Casey Janssen and Shawn Camp).

The levels of frustration from the blown saves tend to blind us from the more successful outcomes, and generally cause a fan base to work itself into a tizzy demanding the addition of a “proven closer” type to anchor the bullpen and ensure that late game collapses don’t happen. Throughout the season, I’ve claimed that this type of player addition is unnecessary, especially for a team in the Blue Jays situation that isn’t expecting to compete this year. In the same breath, I’ll often reduce the closer’s role to being an overrated part of a baseball club.

When I say or write that, I don’t mean to suggest that it’s never worthwhile to pay a lot of money for a reliever. Several teams have members of their bullpen that will contribute enough value to their team this season to justify a $10 million payout based on WAR. This is assuming that the team signing a reliever to that type of contract has done its homework and properly considered sample size in its evaluation, as well as the higher replacement level that exists for relievers.

Unfortunately, teams don’t often do this.

A couple of years ago, Dave Cameron, writing for the Wall St. Journal, took a look at how highly paid relievers were performing relative to the leverage of the situation in which they were appearing. He examined how the “proven closer” types, meant for high leverage situations, actually pitched compared to the non-proven closer type. His findings:

Teams that are paying high prices for proven closers are not getting any better production overall than teams that are paying peanuts for their relief ace.

Similar results are also found from looking at this season. Of the top ten most effective relievers pitching in high leverage situations, this is their salary for 2011:

  1. Alfredo Aceves, BOS: $0.65 million
  2. Chad Durbin, CLE: $0.8 million
  3. Jose Valverde, DET: $7 million
  4. Tyler Clippard, WAS: $0.443 million
  5. Daniel McCutchen, PIT: $0.4525 million
  6. John Axford, MIL: $0.4425 million
  7. Jason Frasor, CHW: $3.5 million
  8. Mike MacDougal, LAD: $0.5 million
  9. Joe Hanrahan, PIT: $1.4 million
  10. Drew Storen, WAS: $0.418 million

Of the top ten, three were signed to free agent contracts under a million dollars (close to typical replacement level), one was signed to a free agent contract under $1.5 million, four have yet to reach arbitration, one was signed to a two year contract worth $14 million in total, and one avoided arbitration by signing a one year deal for $3.5 million (plus an option). None were the result of free agent contracts for more than two years or for an annual salary of more than $10 million that one would think a “proven closer” would demand.

This past offseason, these relief pitchers all signed deals as free agents that guaranteed them at least $10 million:

  • Rafael Soriano, NYY: 3 years/$35 million
  • Mariano Rivera, NYY: 2 years/$30 million
  • Joaquin Benoit, DET: 3 years/$16 million
  • Scott Downs, LAA: 3 years/$15 million
  • Jesse Crain, CHW: 3 years/$13 million
  • Matt Guerrier, LAD: 3 years/$12 million
  • Bobby Jenks, BOS: 2 years/$12 million
  • Brian Fuentes, OAK: 2 years/$10.5 million
  • Kevin Gregg, BAL: 2 years/$10 million

None of these pitchers, not even Rivera, rank in the top 50 (out of the 137 qualifying) for clutch performances (measured by performances in high leverage situations) this season, and two of them haven’t even pitched enough innings as a reliever this year to qualify. Of the nine, only three (Rivera, Downs and Crain) are being used in enough high leverage situations to rank in the top 20 among relievers for highest average leverage when entering the game.

In other words, teams that have spent a lot of money on relievers, aren’t getting the high leverage performances that they presumably expected, and in most cases, aren’t even using those pitchers in the highest leverage situations. So much for the proven closer thing.

How does this happen? It’s the two factors that I mentioned a few paragraphs ago: lack of sample size consideration and high replacement value. When people talk about the volatility of relievers, I have a hunch that it has a lot to do with relievers simply pitching less innings and facing fewer batters in a single season than starters, meaning that their performances from year to year are more volatile, not their actual talent level.

That’s why homework into signing relievers is so important, and why so often, a Triple A player or a Minor League contract guy can perform just as well over a single season as the highest paid reliever available. And that’s why my own preference is for avoiding long term/big dollar contracts for relievers in most situations (not all), even when it might seem as though the largely fictional “proven closer” type is exactly what an organization is missing.