As with many historic events, this one resulted from a remarkable convergence of factors--spectacularly incompetent management; tight-fisted, complacent ownership; a generally pretty ambivalent fanbase; a tiny payroll compared to most teams; Collective Bargaining Agreements that rewarded the Pirates for maintaining their tiny payroll; and so on.
Perhaps the most important factor in preventing the Pirates from having a random .500 season was that the losing streak happened while baseball front offices were undergoing major changes, as general managers embraced new principles and methods and became much smarter. What follows is a broad outline of this trend and how the Pirates ended up under its wheels. (The paragraphs that follow contain a bunch of generalizations that don't apply to every GM and every situation.)
Billy Beane's A's were generally among the first to change, and with the help of a very good core of young players (some inherited, some not), the A's were able to pull off a long streak of excellent seasons despite a relatively small payroll.
Oakland's success was thus also the result of a convergence. The A's were among the first teams to fully embrace new philosophies that gave them an advantage (at least until they paid the price for relying on those new philosophies, which were not nearly as helpful in judging amateur players, to produce talent from their farm system), and they already had a good core that also gave them a leg up.
The Athletics' advantage disappeared partly because it turned out that sabermetrics didn't yet help them very much with the draft, but also because other GMs either altered their approaches to embrace sabermetric principles (Kevin Towers, Doug Melvin) or, more often, were replaced by new GMs who used sabermetric ideas as well as Beane did and, in some cases, had access to a bunch more money than he did. These new GMs neutralized Beane, making him just one smart GM among many.
(Here's the mother of all asides, because I don't want to come off as some sort of stathead partisan: I'm using the phrase "sabermetric principles" very loosely here, to include stuff like understanding the value of 30-something free agents, or not letting a 22-year-old pitcher throw 130 pitches in a start, or recognizing the importance of replacement value, at least in a general way. I don't mean that a good GM has to use WARP3 or anything like that. In fact, you don't need to know any specifics about sabermetrics, or even care about sabermetrics, to understand these principles. Nonetheless, many of them are closely connected to sabermetric research and to GMs who , for one reason or another, have been described as stat-friendly. Again, this is a generalization, and one that's not entirely fair. The example of the Atlanta Braves shows pretty clearly that you don't have to embrace sabermetrics to run a highly successful organization. But I'm going to go with it for now, because the connections between sabermetrics, Beane, and the flood of very smart GMs who've taken over teams in the past five to ten years seems pretty strong to me. In any case, I don't mean to use the word "sabermetrics" as a sword, just as a convenient way to sum up changes in GM behavior to which it seems to me to be closely connected.)
It was significant that many GMs who embraced these new philosophies were new faces, not veteran GMs who changed. What kinds of teams get new GMs? Although several of the better old GMs retired or moved to other front office positions within their organizations, several of the teams who got these new-school GMs were the teams who needed them most.
Some of these new-school GMs reformed some of the worst organizations in baseball. Thus, after ten years of terrible mismanagement under Chuck LaMar, the Tampa Bay Rays hired Andrew Friedman, who was able to scrape some of LaMar's toolsy draft picks together with some inspired general managing of his own to lead the Rays to their first winning season in 2008, in their eleventh year in the American League. Melvin, not a new GM himself but a veteran GM who had embraced new ways of thinking, took over the Milwaukee Brewers, who had twelve consecutive losing seasons before he led them to a .500 record in 2005. And the Arizona Diamondbacks were badly bloated, pretending it was still their World Series-winning year of 2001, when Josh Byrnes took over in 2005. With a core of young players he mostly inherited, some inspired decisions, and some good luck, he led them to a 90-win season in 2007, breaking their streak of three straight losing years.
The influx of new GMs thus helped prevent some teams from racking up extremely long streaks of losing seasons, because some of the best GMs wound up paired with what had been some of the worst organizations.
The Pirates slipped through the cracks. They were casualties mostly of the incompetence, complacency and greed of their ownership group, but they may have also missed out because of poor timing. When Cam Bonifay was fired in 2001, Beane's "stathead" approach was still very controversial. Who knows what Kevin McClatchy's real reasons for hiring Dave Littlefield were, but I don't recall them considering anyone with a "stathead" pedigree (someone who remembers the situation better than I do, please correct me if I'm wrong). (UPDATE: In the comments, Vlad finds this article, which speculated that the Bucs might interview former Beane assistant J.P. Ricciardi, who at least looked like a new-school guy at the time, but I can't find any evidence the interview took place.) If the Pirates had hired a very young, new-school general manager in the mold of Friedman, Byrnes or Theo Epstein, the fanbase would have been apoplectic. Four years later, that might not have been the case, at least not to the same degree.
So instead the Pirates hired Littlefield, who occasionally paid lip service to statistics but had not learned any of the lessons from them that have helped so many smart teams.
It also seemed that the Pirates were comfortable making a halfhearted run for .500 each year rather than doing the gutsy thing and rebuilding the franchise. In the end, this is almost certainly (and ironically) the biggest reason the Pirates' losing streak lasted so long. But it did not help that they were also victims of history.
Littlefield stuck out like a sore thumb throughout his tenure, but if he had been a GM ten years before... well, he still would have been one of the dumber ones, but there would have been a lot more other really dumb ones to help prevent him from messing up too badly. And eventually he quite possibly would have soaked one or two of the other dumb GMs in trades, simply through dumb luck, and the Pirates might have had a .500 season (though surely not a string of competitive seasons) completely by accident.
Instead, Littlefield was one of the last really awful GMs in an era where many were quite good. Add that fact to the terrible ownership and the ten previous years of losing that Littlefield wasn't responsible for, and you've got a recipe for exactly what happened: the Pirates have now tied the 1933-1948 Philadelphia Phillies for the longest losing streak in the history of professional sports.
If we think of the mid-1990s to the present as an era in the history of baseball when general managers changed substantially in their approach to their jobs, the Oakland A's were an outlier at the front end, as they were one of the first organizations to make a critical change. The Pirates were an outlier at the back end, an organization that changed far too late.
McClatchy moved out of the spotlight last year, and Littlefield was soon fired. They were replaced, as you know, by Frank Coonelly and Neal Huntington. Time will tell whether those two have what it takes to end this losing streak, but the early signs are positive: they've made some tough decisions, taken the long view by spending real money on amateur talent, and have made moves that are consistent with a long-term plan of rebuilding the organization from top to bottom. My advice? Do your best to forget these 16 years have happened, and concentrate on the future, because at least there appears to be light at the end of the tunnel.