House Financial Services Chairman Barney Frank (D-Mass.) said Wednesday that if bailed-out banks do not like conditions such as pay caps and restrictions on corporate extravagances placed on their companies, they can give the money right back.

"I'm prepared to say this: if they really feel this way now, and they can't live with the restrictions, and they feel, 'Oh, how dare you ask me how I spent the money,' then give it back," Frank told CNBC during an interview ahead of hearings featuring the CEOs of major banks.

"If they want to pay it back right away, I'd accept it," the chairman, a principal author of the bailouts, added, noting that most companies are too involved with the funds at this point.

Frank insisted he shares Americans' anger over the way the first half of the Wall Street bailout Frank helped author in Congress was administered by the Bush administration.

"I think a lot of people in this country, probably an overwhelming majority, would like to scrap the current system...and start a new one," Frank told CNBC in an interview ahead of today's hearings in which leaders of large banks will be summoned to testify. "The goal is to figure out what to do going forward."

Frank said that the reality is that Congress will likely have to commit more resources to the system, but that he did not like it: "That makes people angry, and I share that anger."

Watch a video of Frank's appearance here: