"I think it is time to shake things up within the Democratic Caucus. I think we should look for some new leadership," Matheson told the Salt Lake Tribune. "I won't be voting for Nancy Pelosi."

The centrist Democrat represents the most Republican Democrat-held district in the country, and is fresh off surviving a tough challenge after redistricting, winning with 49 percent of the vote.

No Democrat has said they'll run against Pelosi, who announced her intentions to stay on as minorit leader on Wednesday, and Matheson ruled himself out as a candidate. He was one of a number of other red-district Democrats who voted against her in 2010 as well — more than a quarter of the Democratic caucus backed other candidates in a closed-door vote, and 19 voted against her on the House floor.

But she easily prevailed as Democratic leader because many of those Dems from conservative districts who would have opposed her had just lost reelection. Redistricting has further depleted the ranks of moderates on both sides of the aisle in the House, making it less likely she'll face an intra-party challenge.

The Blue Dog Democrats, a centrist Democratic group Matheson is heavily involved in, saw their numbers further dwindle due to retirements and defeats of some members, even as he and Rep. John BarrowJohn Jenkins BarrowOur democracy can’t afford to cut legal aid services from the budget Dem files Ethics complaint on Benghazi panel Barrow thanks staff in farewell speech MORE (D-Ga.) held on despite long odds. Rep. Mike McIntyre (D-N.C.), another Blue Dog who was a top GOP target, has a narrow lead in a race that may be heading to a recount.