There are few things more frustrating than watching the lead that your team built through seven or eight innings of baseball wilt away at the end of a game due to the ineffectiveness of a pitcher whose sole responsibility is not to allow that to happen. It’s as maddening as it is memorable.

Unfortunately, it happens to even the best teams and the best relievers. Since the beginning of the 21st Century, only three teams have finished a season with less than double digits in blown saves: 2003 Los Angeles Dodgers, 2004 Los Angeles Dodgers and 2008 New York Yankees. Last month, Drew Fairservice compared the number of blown saves that the Toronto Blue Jays have accumulated this year to the Atlanta Braves, who have the best bullpen in baseball, and the number was staggeringly similar.

As much as fans may want to look at the number of blown saves and add that number to their team’s win total while imagining what could’ve been, they also have to add the blown saves to the win totals of every other team in their division, and account for the times that their team came back to win a game after a blown save and deduct the amount of times that a blown save has occurred twice in the same game. It’s just not as simple as many would like to believe.

As Mr. Fairservice outlined in his post, blown saves aren’t the best way to calculate a bullpen’s worth, as there are several other methods to measure the successes and failures of a reliever based on the leverage of the situation in which they’re pitching. Back to the Blue Jays as an example, despite 19 blown saves this season, their bullpen is fairly middle of the road when it comes to just about every other measurement: WAR, FIP, Clutch, WPA and shutdown to meltdown ratio. In other words, the blown saves have more to do with bad luck surrounding the timing and sequence of hits and runs that the team is giving up rather than any individual’s poor performance levels. There’s also a large gap between the worst pitchers in high leverage situations this season (Frank Francisco and Jon Rauch) and the best (Casey Janssen and Shawn Camp).

The levels of frustration from the blown saves tend to blind us from the more successful outcomes, and generally cause a fan base to work itself into a tizzy demanding the addition of a “proven closer” type to anchor the bullpen and ensure that late game collapses don’t happen. Throughout the season, I’ve claimed that this type of player addition is unnecessary, especially for a team in the Blue Jays situation that isn’t expecting to compete this year. In the same breath, I’ll often reduce the closer’s role to being an overrated part of a baseball club.

When I say or write that, I don’t mean to suggest that it’s never worthwhile to pay a lot of money for a reliever. Several teams have members of their bullpen that will contribute enough value to their team this season to justify a $10 million payout based on WAR. This is assuming that the team signing a reliever to that type of contract has done its homework and properly considered sample size in its evaluation, as well as the higher replacement level that exists for relievers.

Unfortunately, teams don’t often do this.

A couple of years ago, Dave Cameron, writing for the Wall St. Journal, took a look at how highly paid relievers were performing relative to the leverage of the situation in which they were appearing. He examined how the “proven closer” types, meant for high leverage situations, actually pitched compared to the non-proven closer type. His findings:

Teams that are paying high prices for proven closers are not getting any better production overall than teams that are paying peanuts for their relief ace.

Similar results are also found from looking at this season. Of the top ten most effective relievers pitching in high leverage situations, this is their salary for 2011:

  1. Alfredo Aceves, BOS: $0.65 million
  2. Chad Durbin, CLE: $0.8 million
  3. Jose Valverde, DET: $7 million
  4. Tyler Clippard, WAS: $0.443 million
  5. Daniel McCutchen, PIT: $0.4525 million
  6. John Axford, MIL: $0.4425 million
  7. Jason Frasor, CHW: $3.5 million
  8. Mike MacDougal, LAD: $0.5 million
  9. Joe Hanrahan, PIT: $1.4 million
  10. Drew Storen, WAS: $0.418 million

Of the top ten, three were signed to free agent contracts under a million dollars (close to typical replacement level), one was signed to a free agent contract under $1.5 million, four have yet to reach arbitration, one was signed to a two year contract worth $14 million in total, and one avoided arbitration by signing a one year deal for $3.5 million (plus an option). None were the result of free agent contracts for more than two years or for an annual salary of more than $10 million that one would think a “proven closer” would demand.

This past offseason, these relief pitchers all signed deals as free agents that guaranteed them at least $10 million:

  • Rafael Soriano, NYY: 3 years/$35 million
  • Mariano Rivera, NYY: 2 years/$30 million
  • Joaquin Benoit, DET: 3 years/$16 million
  • Scott Downs, LAA: 3 years/$15 million
  • Jesse Crain, CHW: 3 years/$13 million
  • Matt Guerrier, LAD: 3 years/$12 million
  • Bobby Jenks, BOS: 2 years/$12 million
  • Brian Fuentes, OAK: 2 years/$10.5 million
  • Kevin Gregg, BAL: 2 years/$10 million

None of these pitchers, not even Rivera, rank in the top 50 (out of the 137 qualifying) for clutch performances (measured by performances in high leverage situations) this season, and two of them haven’t even pitched enough innings as a reliever this year to qualify. Of the nine, only three (Rivera, Downs and Crain) are being used in enough high leverage situations to rank in the top 20 among relievers for highest average leverage when entering the game.

In other words, teams that have spent a lot of money on relievers, aren’t getting the high leverage performances that they presumably expected, and in most cases, aren’t even using those pitchers in the highest leverage situations. So much for the proven closer thing.

How does this happen? It’s the two factors that I mentioned a few paragraphs ago: lack of sample size consideration and high replacement value. When people talk about the volatility of relievers, I have a hunch that it has a lot to do with relievers simply pitching less innings and facing fewer batters in a single season than starters, meaning that their performances from year to year are more volatile, not their actual talent level.

That’s why homework into signing relievers is so important, and why so often, a Triple A player or a Minor League contract guy can perform just as well over a single season as the highest paid reliever available. And that’s why my own preference is for avoiding long term/big dollar contracts for relievers in most situations (not all), even when it might seem as though the largely fictional “proven closer” type is exactly what an organization is missing.

Comments (29)

  1. Brilliant. Please forward this link to every sports talk radio host (minus Wilner, who would actually understand this), and MSM writer. I love Bob McCown, but I’m getting sick of his recent rants about a ‘proven closer’ and Snider being ‘Quad-A’.

    How soon we forget the clusterf*ck that was BJ Ryan.

  2. Great post Dustin.

    I’ve been having this argument with a few people since the Tampa Bay game last week. Another way to look at it:

    Yankees Bullpen (best in AL): 5.4 WAR
    Bluejays Bullpen: 1.5 WAR
    Diff: 3.9 WAR

    In other words, assuming it was possible to replicate the Yankees bullpen through a series of offseason deals – which would involve at the very least a BJ Ryan style signing or two – we could have had 4 more wins this year! (Yes I know WAR isn’t perfect here but its a hell of a lot better than saves/blown saves).

  3. But you also have to remember that their big free agent signing in Rafael Soriano isn’t adding anything to that difference.

  4. How about we swap bullpens with the Yankees. Add 4 wins to our total, take away 4 wins from their total, and we are 3 back in the win column for the wild card…

  5. Great post Dustin.

    Please also forward this to Scott Carson who the other day suggested the Jays should trade Snider for a closer. Astounding amount of stupidity! What an idiot.

  6. Where can I find data for clutch performances (measured by performances in high leverage situations)

    Is there a simple website that tracks this? I can’t imagine someone went into pitching logs and validated scores etc…

  7. Great read. You and Stoeten’s live streaming is enertaining, but you are both much better writers.

    While I agree with you, I would like for you to give your opinion on the course of action the Jays should take in developing the bullpen next year. There will be a few holes. Though you are correct, it seems to me relievers like having set roles. While the stats may not back it up, the most successful teams (generally) have a designated closer. I don’t care where this guy comes from; but how would you set up the bullpen next year.

    And are you in the group that thinks Litsch would make an effective reliever, or thinks he should be back in rotation. I would like for him to be in bullpen with a FA SP sign/trade or go with our younger SP’s.

  8. Dustin – great analysis although I think your use of the top ten relievers in high leverage situations to justify your point was poor. While those guys are unquestionably excellent relievers, it speaks to the stupidity of MLB managers who utilize these guys in their highest leverage situations rather than the best arm in their bullpen (aka their closer). While their closer may not be their best pitcher, often times that is the case. Yet that arm is saved for the ninth inning far too often to face the 6-7-8 hitters in a lineup.

    Guys like Aceves and Clippard are used more often than not by their managers to get them out of high leverage situations where perhaps the more logical choice would be to use their best arm. Rivera is never used outside of the 9th and thus is that much more likely to be guaranteed to be in lower leverage situations.

  9. @Dave Mac: FanGraphs has a whole slew of WPA-oriented stats for pitchers and hitters.

    The other thing is that Farrell is going to continue to use Rauch and Francisco in save situations to bolster the Elias rankings for both pitchers. The important thing for them here is the draft picks.

  10. best way to build a bullpen is to have a surplus amount of arms for the rotation and begin moving guys to the pen. Free Agent signings for relievers are simply too expensive same with trades..

  11. @Dustin re: Soriano… which makes it all the more ridiculous for people to be advocating signing Type A relievers. The point is if they all pan out then you might buy yourself an extra win or two.. but thats a big if and the cost of them not panning out is high. Given how unpredictable relievers are I think you have virtually the same chance of getting those extra wins by filling the bullpen through a combination of minor leaguers and cheap type B relievers.

  12. Of course we should go out and sign a proven closer/back end of the bullpen pitcher. Look at how well Lamp, Caudill, Myers, and Ryan worked out versus the likes of Henke, Ward, Timlin, Eichhorn, Frasor, and Janssen. Silly Dustin. ;)

  13. the more I read dustin, the more I understand why millions of people invested with madoff.

  14. @Chris Did you even read the next part of the post that talks about being put in leverage situations? You’re missing the point.

  15. But, Simon, it’s hardly a reliever’s fault regarding what point of the ballgame they’re brought in to. Yes, Aceves is the highest leveraged guy in the Sox bullpen. That’s because Bard/Papelbon are your 8th and 9th inning guys. They allow you to play that 7 inning game where you know that the other team will be hard pressed to score. Plus, with a team like Boston, if they DO score off Aceves in the 6th or 7th, there’s a high possiblity the Sox will just score more runs.

    This debate is not as simplistic as it looks on this blog, far too often. You can’t just pull out stats and say a closer is worth X amount of wins, or he pitches in high leverage, therefore he’s better. You have to take into account the effect on the team as a whole to have seven guys who can come in from the bullpen and be effective. This isn’t saying that buying a closer is the best course (that does not always work out), but neither is leaving it to chance, as I feel is the solution pushed forward here. Saying a bullpen of Francisco, Rauch, Dotel could be as potentially effective as a bullpen of any other three guys is just disingenuous and using a set of numbers to make yourself seem smarter.

  16. LMAO you guys are so dumb. Lets see how we can switch and twist stats to try and prove a point. Talk about over analyzing something to death.

    Its pretty simple, you hand the ball to a guy in the 9th inning to close out the game, he either gets the job done or he doesn’t. 19 times the Jays have blown this opportunity, that needs to change. How do you change it? by getting better players, how do you get better players by either drafting, trading for or paying for them in free agency. Any way you wanna try to spin it, if you want better players you have to pay for them.

    • I’ll ignore Mackenzie’s comment because he seems to be a bit out of his element, or 12 years old.

      The point I was making is that spending money on a “proven closer” is silly when the most important situations aren’t even being given to “proven closers.” And when they are (even in the case of Rivera), they’re not performing optimally. I’m not suggesting that these guys are shitty pitchers. Even if they’re having shitty years. In fact I even justify their shittiness by suggesting that the supposed volatility of relievers has more to do with smaller sample sizes than some unique characteristic among bullpens. That to me seems like justification for not spending on big ticket relievers very often.

  17. Why are you talking about how high paid relievers and how teams are better off with the guys on your list, do you not comprehend that when those guys become free agents that they will get more money? Tell you what, you build a team over a 5 year period with a cap of 1.5 million per reliever. Let me know how that works out for you lol

    • @Ned – Do me a quick favor, let me know how many of the Braves bullpen arms makes more than that much money. The answer is 1 – a guy the White Sox signed four years ago and had to eat 60% of his deal to trade him.

      But thanks for playing! And for your use of lol, it and LMAO do a great job of conveying your relative level of intelligence.

    • @Ned. Wow. Are you aware of the salary structures in baseball? Did you even read this post?

  18. Wow, Mac, that’s just a blinders on post. As was pointed out earlier the Jays haven’t blown the lead 19 times in the 9th inning or later. Many of those blown saves came in the 6th, 7th, or 8th. Every team has blown saves in the double digits. While I agree there is some stat twisting to make a point, the same can be said of you by using the blown save stat only.

  19. The Jays pen this year is ample proof that Last Comic Standing is indeed having a season this summer.

  20. But, there’s no actual solution given. If spending isn’t the solution (and often it isn’t, not just with regards to the bullpen), what is? None of the stats are any good, because, as you admit, the high leverage situations may not always go to the same people week in, week out, let alone year in, year out. You can’t just sign anybody because then you end up with a bullpen of Tallets, Millers, etc. So… what do you do?

    • A solution is given. It’s careful consideration of the prospective player addition that takes into account small sample sizes and the relatively high replacement value of relievers.

  21. AA is too smart to trade away the future of the franchise for some bloated hack with a decreasing K% just because he is getting luckier in the ninth inning than other “closers”.

    Thank God he doesn’t care about appeasing the idiot masses that feel like our whole season is ruined because we have less meaningless stats than other teams.

    Excellent article Dustin.

  22. Alright, so what you’re basically saying is to take into account past performance, on the basis that it is small sample size.

    Here’s an interesting article idea. Build yourself a seven man bullpen based off of who you would have chosen at the end of LAST YEAR with last year (or prior)’s stats using only last year’s free agents. Set a dollar amount (say $15 million) that you can spend on these guys. Use what they signed for this offseason on a per annum basis, and tell me who you would have signed.

    Be the GM. What do you come up with?

  23. Take a look at why I think there is a significant misconception about the bullpen.

  24. Don’t you love when a reporter/blogger doesn’t have any real talent so he simply makes posts, insulting the other people who try to give another point of view? What a joke just another reason i don’t pay attention to the score, they might as well be all of the fat kids trying to bully the other kids into taking their lunch.

Leave a Reply

Your email address will not be published. Required fields are marked *