The United States may not be nationalizing banks, but they are certainly taking control, Senate Banking Committee Ranking Member Richard ShelbyRichard Craig ShelbyCornyn: We'll need at least one more stopgap funding bill Moore supporters fire back at Richard Shelby Disaster aid becomes hostage to funding fight MORE (R-Ala.) said Wednesday.

Shelby, appearing on "Morning Joe" ahead of more hearings on financial regulation, said that when the government takes a 40 percent stake in banks, as it's expected to do with Citibank, "it's not nationalization, but it's control."

Shelby argued that some banks, who are sometimes alleged to be "too large to fail," should be allowed to do just that.

"They need to close some of these banks," Shelby added. "Some banks were too large to exist, so you can't be too big to fail."

The Banking committeeman said the most important thing to do now is to reform the financial regulatory system, adding that the Federal Reserve fell short in its oversight role leading up to the financial crisis.