"The bottom line is that regardless of one’s political beliefs, conservatives and liberals alike must be able to exercise their First Amendment rights without fear that the federal government will abuse them for having differing opinions," they say.
Special prosecutors are generally appointed by the attorney general or Congress to investigate a case that could be politically manipulated or have a conflict of interest.
On Monday, President Obama said he would hold the IRS "fully accountable" for the political targeting, if reports were true.
"If, in fact, IRS personnel engaged in the kind of practices that have been reported on and were intentionally targeting conservative groups, then that's outrageous. And there's no place for it," Obama said.
The House Ways and Means Committee will launch a hearing to investigate the brewing scandal on Friday.