Conservative organizations in Tennessee are trying to build a coalition of interest groups that will ask Rep. Scott DesJarlais (R-Tenn.) to resign from Congress.

These discussions are a reaction to news that DesJarlais, a doctor who opposes abortion rights, had a sexual relationship with one of his patients and then pressured her to get an abortion in 2000.

The Chattanooga Times Free Press reported Monday that Tennessee Conservative Union Chairman Lloyd Daugherty is talking with other conservative groups in the state to see who else would support a call for DesJarlais to resign.

"We're very upset that he's broken his medical creed and the trust of the citizens of his district," Daugherty told the Times Free Press. The paper also noted that in 2010, Daugherty endorsed Rep. Lincoln Davis (D-Tenn.) over DesJarlais, even though Daugherty's group "usually backs Republicans."

DesJarlais defended his record and told the paper that conservatives should favor him over his Democratic opponent, Eric Stewart.

"I'll stand on my conservative record of lower taxes, reduced deficits and repealing ObamaCare," DesJarlais said. "Mr. Daugherty supported Lincoln Davis last election cycle, so Eric Stewart would be a consistent choice for him — but not for true conservatives."

DesJarlais's affair with a patient of his prompted Citizens for Responsibility and Ethics in Washington (CREW) to file an ethics complaint on Monday with the Tennessee Department of Health. The basis of CREW's complaint is a Tennessee law that bars doctors from sexual contact with patients.

The Times Free Press reported that last week, DesJarlais said he encouraged the woman to seek an abortion because he was convinced that she was not pregnant. DesJarlais also said he pressed her to get an ultrasound, and noted that no abortion procedure was ever performed.