The investigation centers on a joint operation in April between NATO troops and members of the Afghan National Security Forces (ANSF) in Zabul province.
Gen. David Hurley, head of the Australian Defence Force, said the ISAF troops involved in the April mission belonged to a Special Operations Task Group fighting in the southern part of the country.
The joint team was tasked with killing or capturing a local Taliban commander who was coordinating operations against allied forces in Zabul, according to Hurley.
"An incident of potential misconduct during the operation has been raised through the internal national command chain," he said in a statement to local media outlets.
Afghan and Australian special forces killed four insurgents during the operation, according to Hurley, who declined to provide further details until the NATO investigation was complete.
That said, U.S. forces in country will pursue "all allegations of misconduct by our personnel very seriously," Gen. Joseph Dunford, the top American commander in Afghanistan, said in a statement Wednesday.
Dunford pledged to "fully investigate the incident and keep the Afghan government informed.
The incident sparked violent public protests in Kandahar province in southern Afghanistan on Wednesday.
During one protest near Maiwand district in Kandahar, Afghan police opened fire into the crowd, killing eight and wounding 10 others, the AP reports.
The NATO investigation is only the latest instance of U.S. or NATO special forces being accused of misconduct in the country, less than a year until U.S. forces withdraw from Afghanistan in 2014.
In February, Afghan President Hamid Karzai ordered U.S. special forces to leave Wardak, alleging American troops had committed torture and abducted civilians during their time in the province.
Last month, Karzai demanded the CIA rein in its network of paramilitary units in the country.
Trained and equipped by the CIA and Afghanistan's National Directorate of Security (NDS), the units have been operating in some of the most restive provinces in Afghanistan in preparation for the American drawdown in 2014.
Karzai pushed for the drawdown of the CIA-NDS units after a gun battle in Kunar province ended with the deaths of 10 Afghan children, killed when NATO warplanes carried out an airstrike against a senior Taliban commander during the firefight.