Why the Cloud? Processing, Exploitation and Dissemination

So why is the intelligence community so interested in cloud computing? Three letters: PED (Processing, Exploitation, Dissemination). Take these two real life examples from the publishing industry.

Jim Staten of Forrester Research provided an example of how the New York Times leverage the cloud. The Times wanted to makes its historic archives available for online access. They needed to process 11 million articles and turn them into .pdf files. Initial estimates outlined that hundreds of servers and about 4 Tb of storage would be necessary. The IT organization at the Times estimated a months-long delay before beginning, the need for a significant budget and highlighted the difficulty of locating the computing resources. The project manager give Amazon Web Services a try and kicked off 100 EC2 instances and 4 terabytes of S3 storage. The job was finished the next day with a total cost of $240.

Another hard example comes from the Washington Post. Peter Harkins, a Senior Engineer at The Washington Post, used the Amazon Elastic Compute Cloud (Amazon EC2) to launch 200 server instances to process 17,481 pages of non-searchable PDF images into a searchable online library. With a processing speed of approximately 60 seconds per page, job was completed within nine hours and provided web portal access to the public 26 hours later. Harkins ruminates, “EC2 made it possible for this project to happen at the speed of breaking news. I used 1,407 hours of virtual machine time for a final expense of $144.62. The database of Hillary Clinton’s 1993-2001 Schedule is publicly available at: http://projects.washingtonpost.com/2008/clinton-schedule/.

Examples like this show how cloud computing techniques can be used to revolutionize PED processes. By increasing the use of automation and focusing our analyst on higher level exploitation tasks, near-real time exploitation and dissemination of critical intelligence products may be enabled in the very near term with cloud computing.