Private and government defense and intelligence agencies must work together

Private and government defense and intelligence agencies must work together
© Getty Images

To address today’s national security threats, the Central Intelligence Agency, Department of Defense, and broader national security community needs to hasten the use of 21st century technologies like machine learning and data analytics. But the current big proposals aren’t going to provide serious innovation at the scale the U.S. Government needs.

Highly respected innovators, including Eric Schmidt of Alphabet and astrophysicist Neil deGrasse Tyson, recently rolled out a new series of recommendations to increase innovation across the Department of Defense (DoD).

Having led the implementation of similar programs at CIA, their recommendations sounded eerily familiar and I fear they won’t break the DoD out of its technology rut any more than they have at the CIA.


Schmidt and deGrasse Tyson’s newest recommendations include: Building an accelerator, establishing a system to elevate new ideas, creating a new data science and innovation career field and launching a training program for technology adoption.

These all sound like steps in the right direction. But they’re not enough.

I left the agency last year to begin addressing the biggest problem the defense and intelligence agencies have: lack of 21st century tools that genuinely meet the needs of the warfighter, the operator and the analyst who keep America safe.

With tremendous help from great colleagues and senior leaders, I successfully led the creation of the CIA’s Data Science career service in 2012-2013. It was the first such career structure in the Intelligence Community (IC) and we learned a lot about innovation inside a high-security environment. We did within the IC much of what has been proposed above. While these actions may be necessary, they are not sufficient to turbo-charge innovation in a large government bureaucracy.

While the government deeply understands the mission, the private sector holds the keys to state of the art tools and rapid development. But when it comes to technology, the cross-fertilization between government analysts/operators and corporate innovators is minimal, so we sit in two different worlds and neither side truly understands the other. On one side, the government is notoriously bad defining what it needs.

The same traps spring up every time:

  1. Design by committee ensures no one is thrilled with the final product
  1. Poorly understood requirements create mismatches from initial expectation to final delivery
  1. Departure of key government people in the middle of the project slows everything down
  1. Insistence that your agency’s requirements are unique and therefore your tools must be custom over inflates cost and schedule which exacerbates 2 and 3 above

On the other side, the commercial sector is bad at imagining what the national security personnel need, forcing the government to specify in slow and excruciating detail exactly what it wants. This approach lowers the risk for vendors but eliminates the kind of bold and creative product development one sees in the commercial sector.

There are some concrete steps the national security agencies, commercial sector and Congress can take to improve the speed and quality of innovation, including:

Group similar thoughts and insights 

One of the main pieces of feedback editors give analysts is to combine like with like, meaning group similar thoughts and insights together on the page. The more the government departments can share IT requirements and platforms, the cheaper the software gets for any one Agency because costs are shared across the group.

A small step in the right direction is DIA’s Needipedia,  a simple but effective tool for collecting and disseminating one Agency’s innovation needs to the private sector.

Streamline and speed up the flow of data and tools to the people who need it the most

The Intelligence community does not do its work on the Internet. Solution providers who only deploy their software as a service, where they host applications on their own servers, fail to understand the unique needs of national security personnel to work on isolated network behind extensive firewalls. On the government side, the IC needs to make the process of moving new software and data from the internet to a secure network as fast and easily repeatable as possible.

Encourage data governance practices that are efficient, repeatable, and built for the long term

It is not unusual for data sharing agreements to take up to 80 percent of a project’s scheduled duration. Think months and even years to secure programmatic access to data that must be integrated into new analytic tools. On the commercial side, software platforms that lock data up in proprietary formats create crippling integration problems down the road.

We often talk about finding needles in haystacks. But finding a needle in a haystack is easy. What analysts and operators are really trying to do is find a few specific needles in a stack of needles, while a massive conveyor belt continues to dump more needles on the pile.

We can and must make it easier for our analysts to make sense of the data they receive — to find the right needle in the stack needles. Fostering innovation is not about incubators and specialized career fields, it’s about making it easier for everyone to do what they are best at.

If the commercial sector can find a way to engineer with empathy for the national security professional and the government can relax its bureaucratic grip and trust their commercial partners, only then will we have a shot.

Noel Calhoun was the CTO for the CIA's Directorate of Support and is the co-founder of Koto, the national security division of Kensho Technologies.