LaTourette said both the Republican and Democratic parties have become more polarized over the years, which has squeezed out members who don't toe the line.
"I'm a RINO, I'm a squish, I'm whatever they want to call me," he said, referring to the "Republican in Name Only" tag that some have assigned to centrist Republicans.
"But when you say that I have to believe in X, Y and Z to be a good Republican, that isn't what the Republican Party stands for. And the same thing's happening on the Democratic side."
He said service in Congress has become tougher because "you have people in your own party shooting at you."
"I'm not running, because this place has gotten a little too toxic for me," he said. "Things that were no-brainers … like the farm bill, the highway bill, the student loan interest rate discussions. We have to fight about everything, and it's taken its toll. It's time to move on."
LaTourette warned that it will be tough for both sides to hammer out a deal to avoid the fiscal cliff if either party mistakenly reads the November election results as a mandate to ignore the other side. He said the election showed people want results from the federal government.
"It's going to be a problem if people misread the election results," he said. "If the president and his party say they got a mandate to do whatever they want, or if the Republicans in the House say we have a mandate to do whatever we want, that isn't what the election was about."
On the specifics of the sequester, he said tax reform is the best way to generate more tax revenue, rather than raising rates. "If you broaden the base of taxpayers, and actually get rid of all the underbrush, you can raise that money," he said.
He said Democratic demands for higher tax rates are likely to backfire.
"The people in this country who are rich, they didn't get rich by being stupid," he said. "And so you really run the risk of causing them to say, well, OK, I'm not to going pay the personal income tax rate anymore, I'm going to look at some of these loopholes and deductions."