The United States may not be nationalizing banks, but they are certainly taking control, Senate Banking Committee Ranking Member Richard ShelbyRichard Craig ShelbyMcConnell gets GOP wake-up call The Hill's Morning Report - Presented by Facebook - After high drama, Senate lifts debt limit Here are the 11 GOP senators who helped advance the debt extension MORE (R-Ala.) said Wednesday.

Shelby, appearing on "Morning Joe" ahead of more hearings on financial regulation, said that when the government takes a 40 percent stake in banks, as it's expected to do with Citibank, "it's not nationalization, but it's control."

Shelby argued that some banks, who are sometimes alleged to be "too large to fail," should be allowed to do just that.

"They need to close some of these banks," Shelby added. "Some banks were too large to exist, so you can't be too big to fail."

The Banking committeeman said the most important thing to do now is to reform the financial regulatory system, adding that the Federal Reserve fell short in its oversight role leading up to the financial crisis.