Goolsbee left the White House in June and currently serves as professor of economics at the University of Chicago Booth School of Business.
He has also been haunted by an old story about a road trip when the Romney family strapped their dog — in a carrier — to the top of their car. President Obama's chief political consultant David Axelrod has also hit Romney over the incident.
Obama's campaign manager, Jim Messina, also jabbed at Romney via Twitter on Wednesday, tweeting: "So much for 'we're all in this together.' Romney today: 'I'm not concerned about the very poor.' "
Romney said his comments were taken out of context.
"Of course I'm concerned about all Americans — poor, wealthy, middle class — but the focus of my effort will be on middle income families who I think have been most hurt by the Obama economy," he told reporters traveling with him between campaign stops on Wednesday, according to reports.
—This post was updated at 3:56 p.m.