Texas judge confronts Obama administration over 'judicial restraint' statement

"Judge Smith did say a lot of his colleagues in the judiciary weren't very pleased with the public comments they heard and they wanted them clarified," the lawyer said. "He was saying, 'two can play at that game.' "

Smith's office did not return a call for comment.

Obama's remarks caused a firestorm of criticism on the right. On Tuesday, the president tempered his remarks and said he would respect whatever decision the high court hands down.