The United States may not be nationalizing banks, but they are certainly taking control, Senate Banking Committee Ranking Member Richard ShelbyRichard Craig ShelbySenate defense bill at standstill over amendment fight Woman arrested for laughing in Sessions hearing heading back to trial Senate Dems slam Banking chairman on nominee backlog MORE (R-Ala.) said Wednesday.

Shelby, appearing on "Morning Joe" ahead of more hearings on financial regulation, said that when the government takes a 40 percent stake in banks, as it's expected to do with Citibank, "it's not nationalization, but it's control."

Shelby argued that some banks, who are sometimes alleged to be "too large to fail," should be allowed to do just that.

"They need to close some of these banks," Shelby added. "Some banks were too large to exist, so you can't be too big to fail."

The Banking committeeman said the most important thing to do now is to reform the financial regulatory system, adding that the Federal Reserve fell short in its oversight role leading up to the financial crisis.