Centrist Sen. Ben Nelson (D-Neb.) said Wednesday that GOP senators' fear of the Tea Party has made it difficult for them to reach a deal with Democrats on the expiring Bush tax cuts. 

Speaking at the conservative Heritage Foundation in Washington, Nelson said that Republicans and Democrats have to put aside some of their differences to develop a plan for what to do with the tax cuts that expire at the end of 2010.

"The willingness to come across the aisle has to be there before we talk about the arguments," he said, adding later that "there is always a political risk of working with someone else in this environment."

Nelson said it could be hard for Republicans to enter those bipartisan talks, especially those "who are up in '12 or '14" and are wary of attracting a Tea Party opponent in the primary. 

Republicans and Democrats have been stuck at an impasse over what to do with the expiring tax breaks. Democratic leaders want to extend the cuts for the middle class, but let the cuts for upper-income earners come to an end.

Some centrist Democrats, including Nelson, and all Republican lawmakers want to extend all of the cuts at least temporarily until the job market improves.

Nelson harped on the Tea Party wave in the 2010 midterm cycle in which two GOP incumbent senators were unseated by Tea Party-backed primary challengers in primary contests.

The Nebraska senator — who could face a tough reelection campaign in 2012 — named Sen. Bob Bennett (R-Utah) as one lawmaker who was targeted by the Tea Party for his work with Democrats, specifically his negotiations with Sen. Ron WydenRonald (Ron) Lee WydenScrutiny ramps up over Commerce secretary's stock moves Hillicon Valley: Justices require warrants for cellphone location data | Amazon employees protest facial recognition tech sales | Uber driver in fatal crash was streaming Hulu | SpaceX gets contract to launch spy satellite On The Money — Sponsored by Prudential — Supreme Court allows states to collect sales taxes from online retailers | Judge finds consumer bureau structure unconstitutional | Banks clear Fed stress tests MORE (D-Ore.) on healthcare legislation.

Several Republican senators who are up for reelection in 2012 could face primary challengers from the right: Olympia Snowe (Maine), Richard Lugar (Ind.), Bob CorkerRobert (Bob) Phillips CorkerCorker: 'I think there's a jailbreak brewing' in opposition to Trump tariffs GOP scrambles to regain fiscal credibility with House budget On The Money — Sponsored by Prudential — Senators hammers Ross on Trump tariffs | EU levies tariffs on US goods | Senate rejects Trump plan to claw back spending MORE (Tenn.), Scott Brown (Mass.) and Orrin HatchOrrin Grant HatchRaces to watch in Tuesday’s primaries Senate panel to hold hearing next week for Trump IRS nominee On The Money — Sponsored by Prudential — Senators hammers Ross on Trump tariffs | EU levies tariffs on US goods | Senate rejects Trump plan to claw back spending MORE (Utah).

Sen. Lindsey GrahamLindsey Olin GrahamSenate panel advances three spending bills Trump says he will sign executive order to end family separations Trump backs narrow bill halting family separations: official MORE (S.C.), who is up in 2014, is also a potential Tea Party target. 

Nelson said that bipartisan negotiations that are limited in scope to the tax hikes could prove to be very helpful, similar to the Gang of 14 that in 2005 negotiated a compromise on President Bush's judicial nominees.

"We [need to] have the willingness to come together to sit in several negotiating sessions without staff and talk through these things together," he said. "That willingness, I think, could be harder from the standpoint right now from both sides."