As Herman Cain's presidential campaign works to regroup from allegations of sexually inappropriate behavior by the candidate, they can take solace in the fact that the voters who are most likely to turn from Cain after hearing the news — women — already seem to have broken for rival Mitt Romney.

Two women who worked at the National Restaurant Association in the 1990s — when Cain headed the group — left the organization after reporting sexually inappropriate behavior by Cain, Politico reported Sunday evening.

Cain's campaign fired back aggressively, calling the story an attack by "inside the Beltway media."

The former Godfather's Pizza CEO has skyrocketed in the polls over the past month, but women voters, who are mostly likely to shy away from Cain in the aftermath of allegations, have consistently leaned toward Romney.

In a Fox News poll released last week, 23 percent of women Republicans had already broken for Romney, versus only 17 percent for Cain — numbers sharpened by the fact that Cain led the field 24 percent to 20 percent overall, based primarily on capturing nearly three out of every 10 male Republican voters. Alternatively, Romney was supported only by 18 percent of men — the majority of voters supporting the former Massachusetts governor are women.

Women were also far more likely than men to support Cain's nearest competitors, with 12 percent of women supporting Ron Paul and 11 percent choosing Rick Perry. Neither candidate cracked double digits with men.

Not only do women view Cain as an unlikely choice for president, they do not have a particularly positive view of the candidate. Only 27 percent of all women surveyed said they had a favorable opinion of Cain. Four percent said they would be enthusiastic if the former Godfather's Pizza CEO were elected, while 20 percent said they would be "scared."

A CNN/ORC poll conducted earlier last week backs up this view of Cain's base, showing that 60 percent of his supporters are men — a higher proportion than any candidate, other than former House Speaker Newt Gingrich (R-Ga.), who has confessed to extramarital affairs.

The CNN/ORC poll again found that a majority of Romney's support came from women, and that Romney had captured more than a quarter of women Republican voters.

But while Cain's current base might be a primarily men, the candidate can hardly afford to alienate woman Republicans — nor can he count on men not being troubled by the allegations. More than half of men and more than two-thirds of women in the CNN/ORC poll said they still might change their minds about who to support in the upcoming primary, meaning that while Cain might be popular now, his support within the Republican base is on a shaky foundation.

Moreover, the Fox News poll found that two-thirds of Republican women said they are interested in the 2012 campaign, and that women are more likely than men to choose their candidate based on social issues.

These signs point to a troubling road for Cain if he is unable to quickly put to rest the allegations — especially coming off a week where his campaign was playing defense on issues of abortion and foreign policy experience. The story also provides an opening for Romney to consolidate his strong support with women and gain momentum as the better prepared and vetted GOP contender.