Both men "expressed the desire to implement the terms of the agreement" and committed to a smooth transition of the prison to Afghan control, he said during a briefing at the Pentagon.
The move was heralded as the first step toward transitioning all security operations in Afghanistan to local forces by 2014, when the Obama administration plans to withdraw all U.S. forces from the country.
However, Panetta and Karzai did attempt to resolve issues surrounding the decision by American commanders to retain custody of roughly 50 detainees currently incarcerated at Parwan.
Prior to the handover, U.S. officials suspended the transfer of new detainees to Afghan forces due to long-standing disagreements between Washington and Kabul regarding the circumstances of their capture.
Karzai's administration has long been opposed to U.S. detention practices where terror suspects are held in U.S. custody for indefinite amounts of time, often without being formally charged.
To that point, Parwan will be run as a civilian prison under Afghan control and all "innocent detainees will be released," according to an official statement by the Karzai administration.
The statement did not elaborate on how Afghan officials planned to define which of the thousands of prisoners housed at Parwan would be considered innocent.
For their part, several U.S. lawmakers have expressed concerns that Afghan forces are simply not up to the task of managing a military prison as large as the Parwan facility.
“I've seen the caliber of an Afghan corrections officer soldier ... [and] I've got to be honest with you, I have deep concerns,” Sen. Scott Brown (R-Mass.) said during a Senate Armed Services Committee hearing in March.
Little could not confirm the total number of terror detainees still under American control at the Parwan facility or when they will be handed over to Afghan authorities. Parwan houses roughly 3,000 insurgents captured by U.S. and coalition forces.
That said, the Defense Department hopes to "reach a resolution very quickly" regarding those detainees still in U.S. custody and the overall transfer of the detention facility to Afghan authorities, according to Little.
Initially designed as a temporary holding point for suspected terrorists, the Parwan facility now holds more prisoners than the U.S. detention center in Guantánamo Bay, Cuba.
It was also the facility where U.S. servicemembers inadvertently burned several Qurans belonging to detainees, touching off a wave of violence across Afghanistan earlier this year.
The public uproar ended with the death of six U.S. soldiers and more than 30 Afghan civilians, prompting Karzai to demand all American troops leave the country ahead of the 2014 deadline.