Former Bush administration solicitor general Ted Olson, a part of the legal team who fought to overturn California's gay-marriage ban, said Sunday the decision by a federal judge ruling Prop. 8 unconstitutional was not an example of judicial activism.

"That's why we have judges and that's why we have an independent judiciary," Olson said on "Fox News Sunday." 

"It is not judicial activism, it is judicial responsibility in its most classic sense," he said. 

Back in January, Olson made waves in conservative circles by signing onto the lawsuit to overturn the ban and penning an op-ed he labeled "the conservative case for gay marriage."

Pressed on whether a single judge can overrule the will of some seven million California voters, Olson called it the proper role of the federal judiciary to "make sure that when we vote for something we're not denying minorities their constitutional rights." 

Without separation of powers and the Bill of Rights, Olson said citizens in California or any other state would be empowered to "take away my rights or your rights." 

Olson continued: "We do not put the Bill of Rights to a vote." He noted that 41 states outlawed interracial marriage at one time. Had that been allowed to stand, "the president's parents could not have been married," he said.

Olson said he fully expects the case to reach the U.S. Supreme Court and is "reasonably confident" the Court will side with Prop. 8 opponents, noting the "overwhelming evidence" his side put forth over the course of the Prop. 8 trial.