The survey's gauge of current economic conditions fell to 91 from 98.6, the largest drop in three years.
Expectations for six months down the road fell to 72.9 from 76.5 last month.
Fed Chairman Ben Bernanke has emphasized that while the central bank could begin to wind down bond purchases, it does not plan to let short-term interest rates rise until the economy is back on solid footing, which could be in 2015.
There is growing confidence among economists that the Fed will announce at its meeting next month that the pullback will start, possibly by about $10 billion a month.
With the rise in rates, mortgages have become a bit more expensive, although many housing market experts don't expect that it will curtail years of pent-up demand.
On Friday, the 10-year Treasury yield hit 2.8 percent, the highest level in two years.