May 1999

Subscribe

eMail the Editor
 

In From The Cold

In a new era of openness, the U.S. intelligence community learns to balance knowledge sharing and knowledge security

Last year, Seattle-based Boeing Co. found itself an unwitting pioneer at a fuzzy frontier of the Knowledge Age. Boeing employees working on Sea Launch, an international joint-venture satellite project, were accused by the U.S. Justice Department of improperly sharing sensitive rocketry data with their colleagues from Ukraine and Russia. 

Boeing eventually agreed to pay $10 million to settle civil allegations that it broke U.S. export control laws. Its problems are not over, however: A federal grand jury in Seattle is investigating whether the company violated criminal laws. Among the allegations is the possibility that some of the foreign technicians may have been intelligence agents for their countries. 

The Sea Launch embarrassment highlights the increasingly problematic contradiction between the benefits of knowledge sharing and the requirements of knowledge security. Whether in the domain of defense contracting, weapons research or intelligence gathering, organizations face new opportunities and new burdens in fulfilling their missions in a global, Internet-linked knowledge economy. As Tim Dolan, a Boeing spokesman, pointed out, "there were no precedents for handling many of the export control issues" besetting Sea Launch. Meanwhile, the stakes are rising as corporate security increasingly impacts with national security. 

Nowhere is this sharing-versus-security quandary more sharply drawn than in the U.S. intelligence community. The "need-to-know" doctrine that guided information sharing for decades is eroding, without any clear edict to replace it. Intelink, a secure intranet that began linking the intelligence agencies in 1994, is sparking unprecedented levels of knowledge-sharing both between agencies and across security levels, from unclassified to top secret. Grassroots posting and collaboration over Intelink could soon encompass 300,000 users at all levels of clearance, before adequate security measures are fully developed.

"The technology is irresistible," observed Bruce Berkowitz, a consultant in Alexandria, Va., and co-author of the forthcoming Best Truth: Intelligence and Security in the Information Age (Yale University Press). "People email and collaborate very freely over Intelink. The odd thing is that it's still inconsistent with official doctrine. And, ironically, a lot of this technology that's plaguing the agencies was developed by them." 

Figuring out how to balance knowledge sharing with national security is just one of the challenges faced by the agencies in the wake of the Cold War. When the Soviet Union collapsed, taking with it the era of bipolarism, no analyst could have forecast that the intelligence scene 10 years on would look so much like a competitive marketplace. Among the forces: competing political, economic and military agendas proliferating almost as fast as Internet IPOs; an explosion of open sources with wildly varying levels of reliability; and end user demands for more customized information products delivered against ever-tighter deadlines. Just as in the private sector, technology is both creating these problems and offering awesome new capabilities for resolving them. 

"For the most part, we face the same challenges reported by industry: huge competition for time and attention, cultural resistance to sharing knowledge and the difficulty of transferring tacit knowledge, to name a few," said Bill Spencer, an in-house knowledge consultant at the National Security Agency (NSA) and technical director for NSA's Operations Directorate Business Results Center. 

For companies, this growing symbiosis between government and corporate concerns is a windfall. The intelligence community needs to revamp itself—fast—for the Knowledge Age. As it embarks on this metamorphosis, it is breeding a new generation of tools and methods tailor-made for KM-enabled companies. 

Indeed, a lot of smart people inside and outside the intelligence world are arguing that the community is a priceless but overlooked trove of tools and techniques for corporate KM—not just for competitive intelligence (see "Competitive Intelligence," April KMM), but for the full span of practices and technologies. 

A cornucopia of high-end KM tools developed under the agencies' aegis is spilling into the commercial marketplace. In the next year, corporate users will see powerful new natural-language and multimedia search engines; visualization and classification tools for advanced document management and knowledge creation; and turnkey competitive intelligence and KM systems that incorporate workflow. To KM practices, intelligence lends a crucial future focus, new models for collaboration and rigorous approaches to external knowledge collection, analysis and dissemination. 
Virtuous circle

Of course, the 13 U.S. intelligence agencies have been seedbeds of innovation since the modern spy organization was born 50-odd years ago. In the 1960s, "we helped set the agenda for companies like IBM," stated Fredrick Thomas Martin, a former deputy director at NSA and author of Top Secret Intranet: How U.S. Intelligence Built Intelink—The World's Largest, Most Secure Network (Prentice Hall, 1999). 

But that preeminence is past. Since the 1980s, the dynamic has gradually reversed as the information technology (IT) industry has outstripped the intelligence community in size and capability. In 1998, Director of Central Intelligence George Tenet revealed the community's collective budget to be $26.7 billion—roughly equivalent to that of one IT company: Microsoft. "Now the likes of Bill Gates seldom talks to us," laughed Martin, vice president and director of intelligence community programs at High Performance Technologies in Reston, Va. "We're small potatoes compared to the private sector's total IT budget." 

Thus, going into the 21st century the community urgently needs to reconsider the value-added it provides its consumers, the nation's policymakers; and to redefine its relationship to business. 

The community is rising to these challenges, and in the process defining a new role for itself—one that appears to be even more valuable to industry than the one-way tech transfers of yore. The community-corporate dynamic is mutating into a "virtuous circle" in which the agencies borrow KM tools and techniques from the corporate sector and extend them in new directions, then encourage their commercialization. In this scenario, the intelligence community's key role vis-a-vis industry is to "develop very high-risk technologies, where you don't know when you're going to get a return on investment," Berkowitz explained, "and things the commercial sector lacks incentives for, like basic research or massively parallel processing."

Turning tradition on its head, the agencies are embracing commercial off-the-shelf (COTS) solutions with the zeal of converts. Rather than awarding multimillion-dollar contracts for custom systems, they look to industry before setting their own R&D agendas. "To a greater extent than I can remember, we're capitalizing on and benefiting from commercial advances," said John Young, chief information officer at the CIA. 
 

The all Informed Person

While serving as the CIA's deputy director for scienceand technology, Ruth Davidpromoted a vision of the agile intelligence enterprise.

Where COTS stops, the community steps in. The agencies push the technology envelope in two ways. First, they rely on systems integrators to help them customize and combine COTS tools in unique ways—advances that the integrators later return to the private sector as proprietary products and services. 

Second, the community remains an important funder of R&D at companies and universities. Now such arrangements usually carry the explicit aim that the best solutions developed with agency funding will be commercialized. Once the new products reach market, the agencies are often their biggest customers. 

The same dynamics apply to concepts and methods. The notoriously rigid intelligence bureaucracy is advancing in fits and starts toward a KM-like vision—in turn extending KM in new directions. 

It's a win-win formula. "We gain by having viable technologies absorbed into the commercial sector. Industry benefits by the introduction of new techniques or research capabilities," summed up NSA's Bill Spencer.
The agile enterprise

In a quiet corner office in Arlington, Va., facing the Pentagon's mud-brown walls and endless parking lots, sits one of the key designers of this new vision. 

Ruth David, president and CEO of the national security think tank ANSER, has become something of an icon in intelligence circles. An award-winning engineer and administrator whose soft-spoken graciousness tends to belie her formidable intellect, David recently ended a three-year stint as deputy director of the CIA's science and technology directorate (the largest of the agency's four directorates). There she pioneered application of KM-like ideas to the U.S. intelligence community. Her model of the Agile Intelligence Enterprise (AIE) helped jumpstart the community toward a technology-enabled vision of a virtual, global intelligence enterprise in which self-organizing, multidisciplinary teams muster on short notice to meet urgent security needs. 

"A lot of people had these kinds of ideas, inside and outside government," David remarked. "What we did was paint a picture that people could grasp and use as a goal." 

According to many observers, the AIE and related schemes were stimulated largely by the intelligence shortfalls of the Persian Gulf War, which harshly spotlighted the clash between old intelligence models and new realities. Although more intelligence was being generated than in any previous war, interoperability problems prevented much of it from getting to the battlefield. Gen. Norman Schwarzkopf complained on international TV that imagery intelligence wasn't reaching him in time. 

Accordingly, David's AIE is just one of several comprehensive reform programs emerging from the mid-1990s. Others include the Pentagon's Joint Intelligence Virtual Architecture (JIVA) and NSA's self-reinvention as a "Knowledge-Based Business." 

The AIE, however, most clearly embodies a striking commonality of these schemes—one that companies should study closely. Instead of promoting bureaucratic reorganization, they all use technology to bypass organizational constraints. 

"When I came to the CIA I observed that any kind of organizational restructuring was, first, going to take a great deal of time; and second, you might not get it right," David explained. "The notion of agility attracted me because it's about leveraging technology to put the right talent on the right job immediately—about working collaboratively not only with your colleagues and peers but also with the consumers of your information, knowing full well that your problem set keeps changing." 

The AIE has three key elements that could translate seamlessly to any global knowledge-based enterprise, Martin noted: an infrastructure of interoperable networks; self-organizing multidisciplinary virtual teams, enabled by intelligent agents and collaboration technologies; and shared data. Intelink could serve as the germ of the AIE, David and others have posited. The community has several efforts underway to refine and realize the potentials of Intelink and the AIE. 

JIVA, too, uses technology to slice across organizational boundaries. The world's most ambitious collaboration project to date, JIVA will give the military intelligence agencies (eight of the 13) and their customers a global common operating environment, linking intelligence officers at headquarters with commanders in the field. Warfighters will participate in intelligence production, receive real-time, Web-based decision support from experts and resources anywhere in the world and tap into a new, unified knowledge base. Launched last January, JIVA could eventually embrace nearly 100,000 users, said Randy Brooks, director of command-and-control applications for GTE Government Systems Corp.'s Electronic Systems Division. 

GTE's InfoWorkSpace provides the Web-based framework for JIVA, with interactive whiteboard, chat, televideo and other collaboration tools. "This common perception allows for parallel execution of very complex processes," noted Brooks.

In David's vision, virtual teams like those in JIVA and AIE could be largely self-selecting, with experts around the world monitoring agency networks and volunteering their services as needed. Self-selection could be augmented by intelligent networks that alert relevant experts when their skills might be useful. 
A revolution in progress

As chief information officers for the CIA and the overall intelligence community, respectively, John Young and John Dahms (top) balance the needs to share knowledge while keeping  it secure.

Just past the security checkpoint at the CIA's campus in McLean, Va., a long drive down a forested avenue dead-ends at the Headquarters Building: an elegant white marble cube rimmed by more woods. Visitors have to scan the public relations materials to learn that behind this apparently solitary edifice stretches a vast complex of office towers and parking lots, modeled on a college campus by Director Allen Dulles in the mid-1950s.

Knowledge management has always been the CIA's chief mission, according to John Young, CIO for the CIA, and John Dahms, CIO for the entire intelligence community. Relaxing in an Oriental-carpeted conference room deep in the internal maze of the HQ Building, Dahms said, "Our entire business is information and knowledge, so we've been struggling for a very long time with many of the issues KM is now bringing to light."

So where exactly does the intelligence community stand on these challenges? 

"Ruth David's vision is still five years out," admitted Dahms. "By being five years out it's helped us understand we have to move away from incremental advances and undertake fundamental change. She made a compelling argument that in terms of both opportunity—the emerging technologies—and the risks of failing to embrace that opportunity, it was the right time to leap forward." 

Dahms' title as community-wide CIO, a brand-new one, reflects the high priority the community now gives to collaboration. "My job is to figure out how to get these 13 agencies to work together even though they have different bosses, different IT infrastructures, different missions and all the rest. It's a problem I think more and more businesses will face." 

Dahms is also the individual chiefly responsible for figuring out how to balance knowledge-sharing with protecting each agency's family jewels. How does he intend to do it? 

The problem is not trivial. Intelink (which, remarkably, was built entirely with open standards, COTS products and installed systems) poses the world's toughest data security problem. Its spectacular rise directly confronts the community's traditional segregation into functional stovepipes whose walls are pierced only on a need-to-know basis. 

"The stovepipes remain both a blessing and a curse," Martin explained. While they make it possible to manage the incredibly complex, multibillion-dollar enterprises of the CIA, NSA and other agencies, he noted, they also generate redundancy by the spadeful. And the data they produce "tends to be less broadly focused, whereas intelligence consumers want integrated, all-source, fused-intelligence data tailored to their interests. Moving toward this may be the intelligence community's biggest challenge." 

Sound like your company? Here's how the world's largest KM enterprise is addressing the challenge: According to Dahms, the community is undertaking a major shift from system-level security to information-level security. Rather than hiding an entire knowledge base behind a firewall, or trying to segregate knowledge functionally, the agencies are organizing knowledge-sharing by communities of interest and using strong authentication to control who sees what. (Interestingly, although Intelink's managers have experimented with NSA's Multilevel Information Systems Security Initiative, they have turned mostly to commercial security products and X.509 digital certificates for authentication, Dahms added.) 

Security also is being enhanced by the use of metadata—tags that describe individual pieces of information. Metadata imposes a structure search engines can readily see, allowing collectors and analysts to "link different types of data, structured and unstructured, to a single problem," explained Dahms, while permanently encoding each document with its clearance level. 

In one of the world's biggest data harmonization projects, the intelligence community has launched a groundbreaking experiment with metadata to enhance security, as well as searching, over Intelink. To leverage metadata's potential, Intelink search tools have been specially modified to read the tags. Companies with significant database investments should consider following suit, suggested Martin. "Companies should consider developing their own custom metadata collections. It allows quick recognition of structure and content, no matter what the medium," he noted. 

The Intelink community also has invested heavily in the Web- publishing standards SGML and XML (a subset of SGML optimized for the Web). XML is a key enabler of metadata, said Dahms. "We're experimenting with XML for what I call document understanding—parsing a document into component pieces according to their level of classification, and tagging those pieces so we can customize delivery to users with different levels of clearance. If you tag the pieces during knowledge creation, you can use them almost like objects," he explained. 

Security on Intelink currently leaves a lot to be desired. Collaboration between people with different clearance levels can be difficult. Ultimately, "human judgement comes into play," said Dahms."I don't want to underemphasize how hard it is to strike the right balance" between sharing and security. "Since it's very hard to measure the productivity benefits of sharing, we have to weigh it against the risks—which are also a judgement call."
Internet gumshoes

Knowledge-sharing is just one of the transformations rippling through U.S. intelligence. The classic intelligence cycle has four stages—collection, processing, analysis/production and dissemination—which closely mirror KM's four-step cycle of capture, transformation, communication and utilization. (Some agencies, like the CIA, include planning as a fifth, initial step.) Now the community is using technology to streamline each stage, as well as to harmonize them with each other, said the CIA's Young. The goal is to create a seamless link between intelligence consumers and producers. 

Priorities among the stages are also shifting. Analysts are overwhelmed by the daily onslaught of open source information, and tools and techniques for every stage of the intelligence cycle are oriented more and more to supporting analysis. 

In the collection phase, for example, sorting has eclipsed sleuthing. Here, the intelligence community is undeniably in the forefront—especially when it comes to search-and-retrieval of unstructured data, which streams daily into agency offices in all possible media and languages. 

Thanks largely to agency-funded R&D, "tools that can access data across internal and external, structured and unstructured sources and then filter, rank and display it visually—in effect thinking along with you—will become the norm," posited Elizabeth Liddy, a professor at Syracuse University and president of TextWise (Syracuse, N.Y.), an award-winning R&D firm that creates advanced search-and-retrieval technologies under government contract. 

TextWise was a key player in Tipster, a major program spearheaded by the Defense Advanced Research Projects Administration (DARPA) during most of the '90s to exploit natural language processing (NLP) for intelligence analysis. Tipster "was the first contemporary, large funding of information retrieval," said Liddy. "The agencies were pretty advanced in their thinking, but they had big KM problems: The people who brief the administration were inundated with information." 

DARPA ran Tipster the way it runs most of its R&D programs: Competing teams of corporate and academic researchers worked in parallel, then presented their results in front of each other and borrowed from each other's successes. "This methodology really fosters quick advances," Liddy commented. The solution devised by Liddy's team was so successful that it startled the intelligence analysts who tested it, she said. "It does a very rich semantic representation of [a document's] contents, then does the same thing to the query," Liddy explained. 

TextWise commercialized the concept as DR-LINK, a general-purpose IR system. A questioner can pose a freestyle question like, "How is the euro affecting marketing possibilities in Eastern Europe?" The results can be amazingly on-target, and are visually mapped so users can drill down to documents of interest. 

TextWise is extending DR-LINK's capabilities in evolving agent (EVA) technology, under a contract from the National Imagery and Mapping Agency—the Pentagon agency formed after the Gulf War to expedite dissemination and exploitation of imagery intelligence. EVAs are based on so-called genetic algorithms, mimicking natural selection by terminating poor performers and combining successful ones to produce new offspring. Applied to the same question about Eastern Europe posed above, an EVA might go beyond searching to connect the questioner with relevant experts. 

Other information retrieval breakthroughs are in the pipeline. Excalibur Technologies, whose products reportedly are widely used at the CIA and other intelligence agencies, is taking its flagship RetrievalWare suite of adaptive pattern recognition tools to new levels. "The holy grail is any data, anywhere," noted Mark Demers, director of marketing. Within a year or so, Demers said, Excalibur will "combine all our search technologies to enable full multimedia searching. You'll be able to start with just a photo as your query and find text, video and paper documents relating to the same topic."

On the horizon of collection technologies is the intelligence community's drive to know what it doesn't know, "to answer open, unsolved questions or gaps in the existing data," said Ruth David. 

Such tools are about five years off, she guessed. "Identifying gaps is very, very difficult. But I think these technologies will mature to a point where you could get a 60-75 percent solution to your problem." If one of these search engines came up empty-handed, or found only a fragment of the answer, it would alert a human or automated collection agent. For open sources the solution could be fully automated, David suggested: "If you launched a query against your known archives, it might spawn another query against the entire universe of open sources, then bring back either a pointer or the information itself."

Processing, too, has become a foot soldier in the mission to ease the analyst's task. Where the agencies are really pushing the state of the art is in automated archiving and knowledge extraction. These programs work with unstructured text, extracting relevant nuggets and placing them in a database where analysts can skim them before deciding whether to read the source material. 

Knowledge extraction tools were instrumental just after the Persian Gulf War, when an experimental visualization technology called Spyr was used to analyze patterns in Iraqi message traffic—allowing U.S. intelligence officers to determine that Saddam Hussein's threat to re-enter Kuwait was a ruse. Developed at Batelle Memorial Institute for the CIA, Spyr evolved into a commercial product called ThemeScape, recently launched by Cartia (Bellevue, Wash.).ThemeScape's major innovation is to comprehend and map the actual content of documents, not just topics as most knowledge mapping tools do. ThemeScape harvests themes from unstructured text and maps them into an interactive topographical landscape with "hills" representing particular themes and pop-up summaries of each document. 
The analysis gap

Analysis is the axis of the intelligence cycle. "Aldrich Ames wreaked so much damage precisely because he was an analyst," noted Peter McKenney, managing director of Cipher Systems, a KM consulting and solutions development firm. 

Considering the importance of analysis, the number of IT products dedicated to enabling it is surprisingly small. The agencies have always needed such tools, but not until the mid-1990s have desktop power and the volume of digital materials reached enough of a critical mass to make them viable, David hypothesized. Another challenge for vendors is that analysts (and their customers) are demanding higher value-added than ever before.

"The analyst is the bottleneck," said McKenney, a former naval intelligence officer. "The government's efforts increasingly will be targeted to helping analysts visualize, understand implications and decide what the options are for the company or the country."

PC-based analytical tools will blossom in the next few years. JIVA, for instance, will give a high priority to developing analytic aids like predictive/decision support tools, full-motion video and graphics, interactive data manipulation and advanced search and retrieval, according to Martin. 

Commercial harbingers of the new genre include Wincite by Wincite Systems, KnowledgeX Analyst from IBM and grapeVINE from grapeVINE Technologies. 

A new breed of turnkey solutions for supporting analysis is coming onstream. One example, Knowledge.Works by Cipher Systems, integrates data warehouse, automation and workflow tools in a groupware product designed to organize, create and use intellectual capital enterprise-wide. Knowledge.Works allows human insight to be interleaved with technical functions throughout the intelligence cycle, as competitive intelligence professionals field queries to subject experts, monitor the results and synthesize a set of recommendations for the questioner. Lotus and Microsoft both have bundled Knowledge.Works into their KM offerings. 

Predicted McKenney, "We think KM will come to see that the best value IT can lend to knowledge creation is workflow, not automation." 

Human analysts don't need to worry about their jobs just yet: Computers still can't compete with the human brain when it comes to validating information, discerning people's motives or inferring implications. But the new tools are forcing fundamental changes in the analyst's job, said the CIA's Young. 

"In the past we've tried to run faster and smarter within the same basic business processes. Now the pressures for speed and accuracy are such that if we don't exploit tools that augment the power of the analyst, it could be detrimental to national security. Humans will have to become a lot smarter in how to use these tools," Young said.
Symbiotic future

Action Item GTE Government Systems www.gte.com (781) 449-2000  TextWise www.textwise.com (315) 443-1989  Excalibur Technologies www.excalib.com (703) 761-3700  Cartia Inc. www.cartia.com (425) 468-9020  Cipher Systems www.cipher.com (503) 617-7447  Wincite Systems www.wincite.com (312) 424-6420  grapeVINE Technologies www.grapevine.com (248) 680-6653  Lotus Development www.lotus.com (617) 577-8500  Microsoft www.microsoft.com (425) 882-8080  Commerce Business Daily cbd.savvy.com/ (516) 255-0504

Knowledge-enabled companies can glean enormous value from these trends, but they need to learn to monitor the intelligence community on an ongoing basis. 

Until recently, the body of lessons a company could glean from the agencies was relatively static. Now "what the private sector can learn from the intelligence community is constantly changing—if the community adapts as it should," said Berkowitz. 

Businesses can monitor organizational and technical innovations in publications like Commerce Business Daily, now online. Some of the agencies' public Web sites, especially the CIA's, are rich sources of white papers and other materials that can be adapted for competitive intelligence. Each agency also maintains a technology transfer office. Most important, companies should develop partners inside and outside the community—in private intelligence organizations, in IT companies that work with the government or among the consultants and authors who study the community—to help them keep tabs on important developments.

IT and the knowledge economy are bringing government and industry to new levels of symbiosis. Each sector needs to learn from and teach the other, leveraging each other's unique capabilities in a rising spiral of mutual benefit.

The risk of security breaches is nothing new for the intelligence agencies.Despite incidents such as the Sea Launch leak or the transfer of nuclear technology from Los Alamos, the agencies remain challenged to improve their methods for transforming information into knowledge. Perhaps the solution to the intelligence community's sharing versus security quandry is quite simply more of both.

Kristin K. Nauth, a Washington, D.C.-based specialist in intelligence/counterintelligence reporting and future studies, has contributed articles to numerous business, technology and policy publications.