Geithner: Dodd-Frank rollback efforts waning

He added that the banking industry is also suffering from reduced influence on Capitol Hill since the financial crisis, leaving the difficulties in enticing talented people to come work for the government as the biggest challenge going forward.

President Obama has long had a rocky relationship with Wall Street, as many financial donors to his first campaigns switched horses in his reelection bid, backing GOP challenger Mitt Romney. Obama's rhetoric toward the financial sector, and in particular his remark about "fat-cat bankers," has often been cited as a driving force behind that switch.

But Geithner struggles to understand that perspective, saying the president's remarks were "very moderate relative to the populist rage sweeping across the country."

"I never quite understood why the financial community took such offense at what was such moderate rhetoric," he said.

He also reflected on what his legacy will be — the handling of the financial meltdown, first as head of the Federal Reserve Bank of New York and then as Treasury secretary.

The Troubled Asset Relief Program (TARP) stabilized the financial system and largely recouped its costs, but remains unpopular with the public. Geithner argued that it is "very hard, if not impossible" to craft a financial rescue package that does the job while still addressing "the completely understandable public desire for justice and accountability."

"Those things were in direct and tragic tension, never resolvable at that time," he added.

On fiscal issues, Geithner struck an optimistic tone on the ability of both sides to come together and craft a workable agreement. Washington has been obsessed with failed efforts to strike a "grand bargain." But Geithner contended what needs to be done is "pretty modest" in order to get the nation's debt on a sustainable path, and that the two sides are not nearly as far apart as it seems.

"A very modest amount of additional adjustment is required to achieve a pretty meaningful level of sustainability," he said. "The political divisions you read about in the country, about the nature of the safety net and the role of the government in the economy, are way, way exaggerated.

"In some ways, that’s what’s so frustrating about the current context, because it should be relatively easy to reach an agreement," he added.