The 40 percent of the people who said their situation is worse now were choosing Republican House candidates.
Nearly one-third of voters said their family suffered a job loss in the past two years. Those setbacks didn't give their votes a clear direction — the group divided over which party to support in House races.
Only about a quarter of voters blamed Obama for the nation's economic troubles. Voters overall were more likely to point the finger at Wall Street bankers.
Women, who voiced as much concern about the economy as men, didn't lean Democratic as they have done in previous elections, exit polls say.
Voters in other, smaller demographic groups essential to the Democrats stuck by them, including blacks and young people. Hispanics favored the Democrats over Republicans by about 2-to-1.
Those who called themselves Tea Party supporters overwhelmingly voted Republican. Almost all of them want Congress to repeal the new healthcare law. They also were focused on reducing the budget deficit and cutting taxes.
Meanwhile, voters who cast ballots for Obama in 2008 mostly stuck by the Democrats and still back the president on health care and the economic stimulus package.