Burton said education is supposed to be administered and controlled by state and local governments. He accused the Department of Education of "gobbling up a lot of taxpayer dollars" without providing benefits.

Burton also said the quality of education in the U.S. has decreased under the Department of Education's supervision.

ADVERTISEMENT

"The quality of education has gone down, down, down," he said. "We're one of the least educated, as far as the quality of education is concerned, least educated countries in the Western world, in the industrial world."

Burton didn't limit his criticisms to the Department of Education, saying that a host of agencies need to be eliminated because the government doesn't have the money to pay for them. He also criticized President Obama for "throwing money" at the country's economic woes and for governing with a "socialistic" approach.

"He believes that government ought to control healthcare, he believes that government ought to control the energy sector, and if we pass what was called cap-and-trade, which would deal with energy and the emission of CO2 into the atmosphere, it would raise the cost of electricity, and everything else that we use, dramatically," Burton said.