"[It is] “kinda ironic given that his family came from a polygamy commune in Mexico, but then he’d have to talk about his family coming from a polygamy commune in Mexico, given the gender discrepancy," Schweitzer said.

The Montana governor went on to say the issue of polygamy could hurt Romney with female voters.

"[Women are] not great fans of polygamy, 86 percent were not great fans of polygamy. I am not alleging by any stretch that Romney is a polygamist and approves of [the] polygamy lifestyle, but his father was born into [a] polygamy commune in Mexico," Schweitzer said.

Romney's father, former Michigan Gov. George Romney, was not a polygamist, although he was born in the Mormon colonies in northern Mexico. Those colonies were founded primarily by Mormons who fled the United States once the government began cracking gown on the practice of plural marriage. According to the Mormon Church's Family Search service, Romney's great-grandfather, Miles Park Romney, had five wives. The practice of polygamy was repudiated by the LDS Church in 1890.

Romney has been mostly quiet about his Mormon faith throughout the Republican primary. On Monday, Romney told ABC News he'd be "happy to respond" to questions about his experiences in the church, but didn't see it as a major component of the campaign.

"I'm not running for pastor in chief. I'm running for commander in chief," Romney said.

But Romney has previously repudiated the practice of polygamy as not in keeping with the practices of his Church — and even joked about it.

“I believe marriage should be between a man and a woman … and a woman … and a woman,” Romney said in 2005, according to the Boston Globe.

The Obama campaign looked to distance itself from Schweitzer's comments Thursday.

“Attacking a candidate's religion is out of bounds, and our campaign will not engage in it, and we don’t think others should either," Obama spokeswoman Lis Smith told the Daily Beast.