Rep. Charles Rangel (D-N.Y.) on Sunday said that he feared the Internal Revenue Service scandal over political targeting of Tea Party groups would unfairly stigmatize other workers at the agency.

“The biggest problem is that tens of thousands of IRS workers that work hard with all of this stigma being tax collectors, are getting a bad shake out of this,” said Rangel during a panel discussion on ABC’s “This Week.”

The IRS acknowledgement that agents used higher scrutiny against conservative groups seeking tax exempt status has already claimed the job of acting director Steven Miller and has the White House and Congress both vowing thorough investigations.

Rangel said the problem lay with the law governing tax-exempt status, which he said was prone to abuse.

“I don't think that gang in Cincinnati had the slightest clue as to find out whether or not people making contributions were involved in politics or whether they were involved in social welfare. Under the law, it's not just an exclusive thing. And people were abusing this law,” said Rangel.

But he said he disagreed with the agents’ decision to target those groups that had the words “Tea Party” or “patriot” in their names.

“Whether they should have used this buzz words for conservatives or liberal, it is wrong and outrageous when people are penalized for their political beliefs,” he added.

“But the law has been abused ever since it's been there.”