"Frankly they failed, and it had tragic consequences," Cole told CNN.
A September attack on the compound left four Americans, including Ambassador Christopher Stevens, dead.
Cole, however, dismissed the report's claim that budget cuts imposed by Congress created an atmosphere where some officials stressed savings over additional security measures.
The Oklahoma lawmaker placed blame for the incident squarely at the feet of officials in the State department, saying "someone didn't get the job done" in anticipating and preparing for a possible terrorist attack.
"There's clearly a lot of blame to go around here, but at the end of the day, if there was an insistence, there was a recognition of the danger — there clearly wasn't — if there was a specific request for Libya, it probably would have been entertained. No such request came to Congress," Cole said. "The administration is responsible here, that's what the executive branch does."
Cole did say that he agreed with the report's findings that no single individual had "engaged in misconduct or willfully ignored his or her responsibilities."
"I have no doubt nobody intended for this to happen," Cole said. "But to say nobody is responsible is something else. It's negligence."
Cole said the individuals with "line item responsibility for security" — and their bosses in the State Department — should be held accountable.
"When you don't say somebody was responsible … that just doesn't square with the facts," Cole said. "Someone didn't get the job done, didn't send the word up the chain of the command or what have you."