Gingrich was speaking at an education forum in New York attended by a number of Republican candidates.
Michele Bachmann, who is also running for the presidency, also criticized the plan at the forum, saying it created a "moral hazard."
"There is a morality in keeping our financial promises, and I don't think we should push that off onto the taxpayer," Bachmann said. "The individual needs to repay and be responsible for repaying their student loan debt."
The Minnesota congresswoman also echoed comments by Speaker John Boehner (R-Ohio), who said Thursday he was concerned about the constitutionality of the administration's decision to change rules without congressional approval.
"I believe it is abuse of power from the executive to impose via an executive order a wholesale change in the student loan," Bachmann said.