Rubio: Obama 'doesn't have clean hands' in IRS scandal

"What he should have said yesterday is, 'In addition to it being outrageous we want the Justice Department to investigate fully, and if a crime has been done here, people will be prosecuted no matter what level of the IRS they're in,'" Rubio said.

Attorney General Eric Holder announced earlier this week that the Justice Department would open a probe into the targeting.

Rubio went on to accuse the Obama administration of fostering a tone that drove the IRS to focus attention on Tea Party groups.

“It’s his campaign; it’s his White House. It’s basically an attempt to muscle anyone who is their political opponent, and to use whatever power they have at their disposal to intimidate people who they don’t agree with,” he said.

In a statement from the East Room on Wednesday, Obama said he was "angry" to learn of the IRS's actions.

"I will not tolerate this kind of behavior in any agency, but especially in the IRS, given the power that it has and the reach that it has into all of our lives," Obama said.