"Whatever one thinks of government oversight, it is quite clear that supervising only part of a market, while leaving the rest of it to operate largely in the shadows, cannot be expected to work to anyone’s satisfaction," he said.
The young financial watchdog, created by the Dodd-Frank Act to protect consumers from unfair practices, has so far issued prominent rules for qualified mortgages and international money transfers, though it has also issued revisions and amendments to the rules.
In April, the CFPB relaxed its February 2012 rule on the transfers, called remittances.
"Although we were reluctant to revise a rule so quickly, far better that we should get things right on the second try than stand fast out of a misguided sense of our infallibility," Cordray said on Wednesday.
Additional changes to rules on mortgages will also be proposed "shortly," he added, and "there may be more adjustments at the margins over the coming months."
Cordray's remarks about the CFPB's rulemaking were delivered at the Exchequer Club, a Washington organization that hosts discussions on financial and economic issues.
Cordray maintained in his speech that edits to rules are done to make compliance easier for financial institutions and to best implement the law.
He also said that regulatory procedures can be hampered by poor judgment and lack of information and imagination, traps that the CFPB has aimed to avoid by gathering a wide set of perspectives.