"Replacement level" players are ones who are "of a caliber so low that they are always available in the minor leagues" (per Baseball Prospectus), or "freely-available talent" (per Wikipedia). But just how "always available" or "freely-available" are replacement level players?
In a previous Fanpost, I characterized Kevin Correia as "reliably mediocre". As of this writing, he has put up exactly 0.0 bWAR (4 RAR); his average game score is 48.5 (with 50 being average), his variance is low, and he's only had two games with game scores below 35. He's basically been a replacement level player.
If replacement level players are freely-available, there's no upside to keeping Correia. The team can take a flyer on another pitcher, safe in the knowledge that it can replace Correia with a comparable pitcher at any time.
But are they?
Between the 2007 and 2012 seasons MLB teams signed a total of 652 free agents from other teams (excluding free agents re-signed from their own teams)*. Of these, 306 either signed a minor-league contract or a major-league contract for less than $1M (roughly twice the league minimum). These 306 free agents are a representative sample of "freely-available" talent.
126 of these players never played in the major leagues for the teams that signed them. 85 of those who played in the majors put up negative WAR. Thus, only 105 of 306 players (34%) signed to what could be considered "replacement level" contracts actually put up "replacement level" or better performance.
Even those who signed major league contracts (of less than $1M) weren't locks to provide at least replacement-level performance. There were 81 players who signed major league contracts of less than $1M; 14 of these never played in the majors for the team that signed them, and another 24 put up negative WAR, leaving only 44, or 54%, who actually produced at replacement level or better.
But is the potential upside worth the gamble? In a word, no. Of the 105 players who provided "replacement level or better" performance, only four put up more than 2 WAR/year: Casey Kotchman (3.4 WAR in 2011 for Tampa Bay), Bartolo Colon (2.8 chemical-aided WAR for Oakland), Juan Uribe (4.7 WAR for SF in 2009-2010) and Joaquin Benout (2.2 WAR for Tampa Bay in 2010). Only 20 even put up 1.0 WAR/year. In other words, by signing one of these players, you were about ten times more likely to get one who never played in the majors or put up negative WAR than one who would put up at least 1 WAR/year.
The performance of free agents on major league contracts of less than $1M isn't much better - out of 81, six put up at least 1 WAR/year; 38 never made the majors or put up negative WAR.
So: "Freely-available" players are about twice as likely to bust as to provide replacement-level performance. And they're less than a 1 in 10 shot of giving even 1 WAR/year, with about a 1 in 100 shot of giving 2 WAR/year. Getting a "replacement level" player is a whole lot harder than picking up a street free agent or picking up an AAA player for cash considerations or a PTBNL; a player on your roster who can be reasonably expected to perform at replacement level is a worthwhile asset.
* This only counts free agents signed in the offseason; I don't have a reliable list of players signed during the regular season, but figure that this set is a pretty reasonable sample of "freely-available" players. Additionally, I didn't want to confuse the data set with partial-season data, except for where the partial season was due to a player being released or dumped - in which case I felt the partial-season was justified.