A former Republican lawmaker who nearly defeated Arlen Specter in a bruising 2004 primary said on Monday that the Pennsylvania senator faces a tougher road to reelection in 2010.

Former Rep. Pat Toomey (R-Pa.), whom Specter defeated by less than two percentage points in 2004, said he hasn't ruled out a 2010 rematch, but has no timetable for a decision.

“He’s significantly more vulnerable now than he was in 2004,” Toomey said in an interview.

He argued that Specter’s core constituency in the GOP, which he called “liberal and moderate Republicans,” have since left the party and will be unable to vote in Pennsylvania’s closed Republican primary. That will make it more difficult for Specter to prevail against a conservative opponent, Toomey said.

In 2004, President Bush and then-Sen. Rick Santorum (R-Pa.) supported Specter over Toomey, who said that was a key to Specter's triumph.

Toomey also said Specter’s age and health will be factors this time around.

Specter will turn 80 in February 2010, and has battled cancer in recent years. After getting a clean bill of health in 2005, Specter suffered a recurrence of Hodgkin's disease earlier this year.

The ex-House legislator claimed his success in 2004 bodes well for any Republican challenger to Specter in 2010. Toomey won 49.18 percent of the vote to Specter’s 50.82 percent.

“The fact that I was able to get as close as I did proves to voters that it is possible,” he said. “If there’s a credible, well-financed conservative challenger, he will be seen as a potential winner.”

Democrats are also eyeing Specter’s seat in 2010. As a centrist Republican from the Northeast, Specter is something of a dying breed, and the speculation in Democratic circles in recent weeks has centered on talk-show host Chris Matthews of NBC’s “Hardball” entering the race.

Specter predicted on Sunday he would have a tough opponent in the GOP primary and in the general election.

“Well, I’m going to have an opponent,” he told CNN’s Wolf Blitzer on “Late Edition”. “In fact, I'm going to have two opponents, Wolf: one in the primary where I always have a tough race, and again in the general.”

He said he would be prepared “whoever my opponents are.”

Specter’s Senate office referred The Hill to his statements on “Late Edition” in response to a request for a comment for this article.

Defeating Specter will not be easy. He had nearly $5.4 million in cash on hand at the end of September, and a recent poll by Quinnipiac University found Specter enjoying a job approval rating of 62 percent.

But Specter may be more vulnerable from the right than the left. In 2004, Specter easily defeated then-Rep. Joe Hoeffel (D-Pa.), 53-42.

In the Quinnipiac poll, Specter held a comfortable lead of 45 percent to 33 over Matthews in a hypothetical match-up. Sixty percent of those polled said they did not know enough about Matthews to form an opinion.

Other Democrats who have been mentioned as possible challengers to Specter include Reps. Patrick Murphy, Allyson Schwartz and Joe Sestak.

Specter's centrist record has made him a frequent target of the right. In 2004, some conservatives did not want him to take the gavel of the Judiciary Committee, but he won that battle as well.

Conservative critics of Specter will be closely watching the senator's positions on President Obama's judicial nominations. Specter is now the ranking member of the Judiciary panel.

Specter helped block President Reagan's appointment of Robert Bork to the Supreme Court in 1987 but strongly supported the nomination of Justice Clarence Thomas.