Rep. Paul Ryan made the rounds on Capitol Hill on Wednesday, huddling with conservative groups whose support he’ll need to be elected Speaker of the House.

But for a second day, the Wisconsin Republican made clear he wouldn’t kowtow to conservatives’ demands and would only serve as Speaker if his own lengthy list of conditions are met.

{mosads}When a handful of vocal conservative colleagues hammered Ryan for declaring he wouldn’t be like other modern Speakers and fundraise on weekends, the 45-year-old married father of three young children aggressively pushed back.

“Hey, look, I’m here four days a week as it is,” Ryan said Wednesday as he walked out of the Capitol building. “I’m not going to spend the other three days a week running around America.

“I guess that’s just Washington for you,” he said of the criticism.

At meetings Ryan held with the conservative House Freedom Caucus and Republican Study Committee on Wednesday, it became clear some of his demands are not going down smoothly.

At both sessions, Ryan fielded tough questions about his proposal to water down an arcane House procedure that allows rank-and-file members to oust a sitting Speaker — something that had been used as a weapon against Speaker John Boehner (R-Ohio).

During the RSC meeting in the Capitol basement, Rep. Raúl Labrador (R-Idaho), a Freedom Caucus leader, and Rep. Joe Barton (R-Texas) were among those who pressed Ryan to drop his plans to change the so-called motion to vacate the chair.

Barton advised Ryan to “back off the issue,” said a lawmaker who was in the room.

Ryan told his colleagues he has no desire to eliminate the procedure; he simply wants to raise the voting threshold for members to use the motion against a Speaker.

Winning over the Freedom Caucus is proving to be a huge mathematical challenge for Ryan.

Many in the group of roughly 40 Tea Party insurgents have said they’re standing by Rep. Daniel Webster (R-Fla.), whom the caucus endorsed for Speaker earlier this month. For weeks, Webster has been calling for an overhaul of internal rules and procedures, popular among Freedom lawmakers, that would spread out power to rank-and file members.

For the group to switch allegiances, four-fifths of their members would need to drop Webster and back Ryan, the Ways and Means Committee chairman and 2012 vice presidential nominee.

“It would be quite challenging” for Ryan to win the endorsement, said Rep. Mo Brooks (R-Ala.), a Freedom member and ardent Webster backer. “Daniel Webster wants to decentralize the power of the House of Representatives, while Paul Ryan seems to be still wanting to consolidate it in the Speaker’s position.”

Boehner on Wednesday scheduled an internal election to nominate his successor on Oct. 28, with a formal floor vote tentatively set for Oct. 29. Whoever is elected Speaker would serve until after the 2016 elections, when new contests for the House leadership would be held. Ryan has given no indication that he would be a short-term Speaker and would be expected to seek to lead the GOP beyond next year.

The outgoing Speaker said he was confidant Ryan would wrap up support from the trio of caucuses. Ryan also must win over the centrist Tuesday Group — which is expected to be a comparative breeze to the other two. He’ll meet with those lawmakers at 10 a.m. Thursday.

Boehner, who’s in the middle of his third term as Speaker, still intends to resign from Congress on Oct. 30.

If the Freedom Caucus pushes Ryan out of the race, it would represent the group’s third scalp. A motion to vacate threat by Rep. Mark Meadows (R-N.C.) forced Boehner into early retirement, and the group successfully pressured Majority Leader Kevin McCarthy (R-Calif.) to drop his bid for Speaker earlier this month.

Ryan told his colleagues in a closed-door meeting Tuesday night there was no way he’d be the “third log on the bonfire,” sources said. Ryan allies, meanwhile, are gearing up for open warfare if the Freedom group scuttles his bid for Speaker.

“If they don’t get behind him and they sink him, they’re going to have hell to pay,” said Rep. Adam Kinzinger (R-Ill.), one of three leaders of the Tuesday Group.

But conservatives are also pointing to Ryan’s record on immigration reform as another knock against him. Republicans quizzed him about the issue during his meeting with the Freedom Caucus on Wednesday.

In 2010, for instance, he voted against the Dream Act, which would have halted deportations for qualified undocumented immigrants brought to the country as children — a move conservatives deride as “amnesty.” But he’s also supported several proposals, favored by advocates, to legalize illegal immigrant farmworkers and allow undocumented students to get in-state tuition benefits.

Conservatives are eying Ryan’s pro-reform history carefully and seem to want assurances that, as Speaker, he won’t try to bring anything resembling comprehensive immigration reform to the floor.

Rep. Tim Huelskamp (R-Kansas) said immigration was the only policy reference Ryan made in his 10-minute speech to the conference Tuesday night — a mention that’s left some conservatives questioning his intentions.

Ryan “left the door open” to bringing an “amnesty” plan to the floor, Huelskamp said. “These are the types of questions we would like answered.”

Cristina Marcos contributed.