The STARLIGHT Information Visualization System

The STARLIGHT Information Visualization System JS Risch, DB Rex, ST Dowson, TB Walters, RA May, BD MoonPacific Northwest National LaboratoryRichland, Washington USA{js_risch, db_rex, st_dowson, tb_walters, ra_may, bd_moon}@AbstractSTARLIGHT is an example of a new class of information system expressly designed around a visualization-oriented user-interface. Incorporating more traditional information storage and retrieval technologies into its design, the Starlight system also enables the integrated use of multiple, concurrent visualization techniques to support comparison of content and interrelationship information at several levels of abstraction simultaneously. This approach enables powerful new forms of information analysis, while at the same time easing cognitive workloads by providing a visual context for the information under study. Originally developed for intelligence analysis applications, the Starlight software is intended to support the rapid, concurrent analysis of complex multimedia information, including structured and unstructured text, geographic information, and digital imagery. The system uses novel 3-D visualization techniques that interactively generate easily understandable representations of explicit and implicit relationships contained in information collections of various types. This paper describes the general theory behind our approach, and the design and features of a Windows NT-based operational system.1 Introduction and Problem DescriptionLike many information analysts, members of the U.S. Intelligence Community (IC) face the daunting task of making sense out of enormous quantities of multiformat, multimedia information. Unlike most other fields, however, intelligence analysts may also face the need to make life-or-death decisions under extreme time pressures that require them to perform these tasks quickly while maintaining exceptionally high quality standards. Unfortunately, conventional analysis tools and approaches are failing to keep pace with either the enhanced capabilities of modern information collection systems or the evolving analytical requirements of contemporary intelligence problems.The intelligence analysis process is principally driven by the need to filter, distill, and correlate large quantities of structured and unstructured textual information. There is an increasing need, however, to correlate textual information with non-text information, such as data from relational databases, and with spatial information, such as maps, images, and schematic information. Another critical requirement is the need to automatically derive and clearly represent complex associations, including temporal interrelationships, among the elements of such multimedia information collections. Finally, the rapid tempo of contemporary intelligence analysis environments demands that information analysis systems preserve, as much as possible, the conceptual continuity of the analyst during analysis. We feel these requirements make the field of intelligence analysis a good candidate for the application of Information Visualization (IV) technologies.2 Background“Information Visualization” is the process of transforming abstract, typically non-numerical information (such as text) into more easily understandable, graphical forms. Information Visualization approaches commonly involve converting non-numerical information into a numerical form prior to the application of statistical graphics or other visualization techniques commonly used by the scientific and business communities. In this way, inherently non-spatial information can be “spatialized,” or converted to a spatial form, for easier analysis and assimilation.Much previous IV research has concentrated on the development of visualization techniques individually applicable to relatively limited information domains (e.g., hierarchical data structures [1,2], semantic relationships [3], relative query results [4], relational data [5,6], many others). In contrast, Starlight represents an attempt to address the pressing problems of the U.S. IC by developing a visualization system that is capable of supporting the integrated analysis of a wide range of information types and structures. To accomplish this, Starlight incorporates multiple established and newly developed visualization techniques into its design, alongwith a comprehensive information preprocessing, storage and retrieval system.The unique demands of intelligence analysis problems have driven our design of Starlight. The IC’s heavy reliance on textual information required that an effective text visualization approach be a key component. The need to correlate text contents with information in relational databases led us to experiment with some of the approaches being developed by the data mining community. The requirement for integrating free- and structured-text information with map and image data led us to develop an approach for displaying this information concurrently with graphical representations of textual information. And the need to simultaneously depict complex interrelationships among elements of information collections led us to incorporate a system for dynamically constructing associative network diagrams into our design.In addition to effective visual presentation capabilities, it was clear that a practical information analysis tool must also provide a complete information system, with the capability to ingest, store, and retrieve information dynamically during analysis. This was one of the key limitations of our earlier prototype [7]. Further, the realities of dynamic, often chaotic, intelligence analysis environments require that such systems employ flexible information structures that can rapidly accommodate new information and quickly generate the “view” of a particular data set appropriate to the immediate task. Finally, while high-end, dedicated graphics computers provide many powerful, advanced capabilities, their expense and complexity makes them impractical in most analysis domains. The current version of Starlight, hosted on a high-performance Intel platform operating under Windows NT, effectively addresses these requirements and represents one of the first attempts to bring emerging IV technologies to the desktops of intelligence analysts.3 General ApproachConceptually, we have divided the information analysis process into two main components and have organized the software to provide integrated support for both types. These two main components are content analysis and interrelationship analysis. By content, we mean both the semantic content (“meaning”) of individual pieces of information, as well as their individual structure or form. An analysis of the occurrence and internal distribution of words in an individual document is an example of content analysis. By interrelationships, we mean the explicit and implicit associations present among information objects in a collection. Note that such associations can be considered at many different levels of abstraction. Examples of implicit interrelationships include general conceptual similarities among free-text documents. Examples of explicit interrelationships include direct references among documents (e.g., references in technical papers, URLs) or the co-occurance of important terms (e.g., person or place names).Our approach, then, is to provide exploratory tools that will enable analysts to quickly develop and test hypotheses related to informational content and associative interrelationships. These tools take the form of a variety of text-based and graphical querying functions that operate on metadata about the information that are derived during a preprocessing phase. The results of these operations are presented in both graphical and textual forms.To achieve our goals of accelerating and improving understanding of this information, we use a variety of proven and new 2-D and 3-D visualization methods in an integrated fashion. We integrate multimedia information, such as imagery and maps, with graphical representations of text collections to enable their concurrent analysis. This approach enables us to generate displays with very high information densities, providing users with the ability to compare information presented at several levels of abstraction simultaneously. Display complexity is managed through the application of proven visualization techniques such as layering, and through the employment of interactive information filtering tools. A variety of more traditional analysis tools provide support for Starlight’s visual analysis tools. These include several integrated query engines and an automated mapping system.Clearly, enabling the types of analyses discussed here requires the development of a comprehensive information system that includes components for information loading, storage, retrieval, and presentation, in addition to information analysis tools. Due to their visual orientation, we have termed such systems Visual Information Systems (VIS). In this sense, Geographic Information Systems (GIS) can be considered a type of VIS that deals specifically with the management, analysis, and display of geographic information.4 Functional DesignFunctionally, Starlight has two key components: 1) an information preprocessing and modeling system for characterizing information content, modeling associations among input data elements, and storing the modeled information for later retrieval and analysis, and 2) a visualization system/user interface used during exploratory analysis of the information model.4.1 Information Preprocessing and ModelingDuring the preprocessing phase, input text data isfirst reformatted to a standard form. During this process structured information (e.g., header information) is tagged and hierarchically related to emphasize semantic dependencies among the input information fields; special field types (e.g., date/time fields) are identified; and free-text portions of the data are flagged. Multimedia input data is handled similarly, through the use of text data descriptors that contain information pertaining to both the data as a whole (e.g., acquisition date, image file location) and specific features within the data (e.g., the locations and names of people in a photograph). Once reformatting has been completed, the text data is fed into a Starlight database construction utility. At this point, the structured and free-text components of the input data are separated and treated independently, however reciprocal references between them are maintained.Free-text items are routed to a series of statistical and natural language processing utilities for both individual and group characterization. The first step in the statistical processing is the removal of words with low semantic content, along with words that occur so frequently within the text collection that they would serve as poor discriminators. At this point a clustering algorithm can be optionally applied to divide large input data sets into smaller, conceptually related subgroups. Finally, a variation of Latent Semantic Indexing (LSI) [8] is applied to the individual subsets. The results of this processing is a “semantic index” that is used to support “concept”queries in the analysis portion of the software. Additionally, the singular vectors resulting from the singular-value decomposition of the term-by-document matrix can be plotted to enable visualization of conceptual interrelationships among elements of the document sets.In addition to the statistical processing, a variety of natural-language processing algorithms are applied to the free-text data to further characterize it. In particular, term-extraction algorithms are used to extract terms-of-interest from the data. These algorithms automatically identify and flag terms, such as the names of people, places, and organizations, found within the text. Place names extracted from the text are treated as a special case, and are run through a geo-referencing engine that associates the names with geographic coordinates for later use during the visualization and analysis process. All extracted terms are subsequently appended to the structured component of the original input data element.Input and derived information is modeled in an object-oriented fashion, and is stored in two parallel, but interconnected, data structures: one for storing and retrieving the actual content of the input information, and the other for deriving, storing, and retrieving associative information developed from the structured components of the data.Information content is stored in an object-oriented DBMS, along with the data indices developed during preprocessing. Currently Starlight includes three different content data stores, one for textual information, one for image information, and one for geographic information. Each of these data stores includes one or more dedicated data retrieval engines.Structured information (both input and derived) is modeled using Leenstra, et. al.’s Contiguous Connection Model (CCM) [9], which is conceptually similar to the Semantic Database Model (SDM) of Hammer and McCleod [10]. With the CCM approach, a hierarchical schema is imposed on input (typically “flat”) structured data in order to emphasize semantic dependencies among the input field values. Then, through an inversion process, an inverted index is constructed that relates multiple occurrences of the same field values in different records to one another. A query for information associated with a particular field/value pair returns a tree structure, called an Array Set, containing all information that is hierarchically related to the query.The CCM supports dynamic data structuring and restructuring based on the particular “view” of the data required at any particular instant. For example, CCM-based modeling enables the object-attribute duality commonly found among real-world entities to be captured and effectively exploited. For example, an author “has-a”book, but the book also has an author. In the first case, a book title can be considered an attribute of the author, while in the second case, the author can be considered an attribute of the book. In reality, there is only one instance of a particular author and book title. In this case, queries against a CCM-based model of this relationship could be initiated from the point of view of the book or of the author, depending upon the current interest of the user. Further, through a capacity known as “role exchange,”sets of attributes can be independently organized into logical groupings. For example, a person’s role as an author may involve an entirely different set of attributes (publisher, publications, etc.) from that person’s role as a researcher. The CCM enables attributes associated with different entity roles to be treated independently, or jointly, as required.4.2 Information Visualization and AnalysisDatabases generated by the data loading and preprocessing systems can be interactively explored using a variety of integrated visualization tools. Starlight’s key visualization “venue” is a 3-D workspace that contains graphical representations of entire databases, along with ancillary information resources, such as maps and images [Fig. 1]. This workspace is intended to foster relatively high-level comparisons among elements of an information collection as a whole, providing a sense of the overall semantic “shape” of the collection. Theworkspace can be interactively navigated, providing users the ability to alternately examine large-scale patterns in the information from a distance, or move in to inspect local informational structures in detail. Additionally, multiple views of the workspace can be open at the same time, providing the capability to simultaneously monitor local and global informational patterns during analysis.In the workspace, 3-D icons represent individual database elements (e.g., documents). The 3-D spatial distribution of the icons conveys information about their contents. Starlight currently uses two principal spatial distribution methods, one for collections of textual information, and another for representing collections of purely structured information.In the case of collections of free-text documents, the results of the semantic analysis of the documents are used to generate a 3-D scatterplot that shows general conceptual or topical interrelationships among them. In this display, which we call a Similarity Plot, the proximity of any two icons to one another is an indication of how conceptually similar they are to one another. Documents that are closely related to one another (i.e., that use the same words in similar combinations) will appear clustered close together, while less similar documents will appear distant from one another. This “starfield” display approach is similar to that used by several systems, including the Bead [11] and SemNet [3] systems. The goal of this approach is to provide a sort of automatically generated table-of-contents for the document collection.For purely structured information, a different organizing principal must be used to generate meaningful spatial distributions of icons. We are currently using the straightforward approach of enabling users to dynamically “sort the information into piles” distributed over the surface of a sphere (called a Data Sphere), based on the values associated with any given (categorical) field [Fig. 2]. For example, a user may elect to spatially redistribute a collection of personnel records based on the values in the “Employer” field. The system would first sort the records into groups based on the Employer field values, then evenly distribute the icon groups over thesurface of the sphererepresenting that collection.Note that free-textinformation also typically hasa structured component to it.Consequently, users can choseto convert free-text SimilarityPlot representations to DataSphere representations, andback, should they wish toexamine the structuredattributes of the text using thismethod.In addition to spatialdistribution, icon color andshape can be used tosimultaneously conveyadditional dimensions ofinformation. Users can electto apply color and/or shapeencodings to icons based onvalues associated withparticular fields of thestructured data. In the case ofcategorical field types, eithercolor or shape can be used toconvey information about aparticular data element’s field value. Color alone can be used for ranged numerical field types. Note that icon shape only becomes apparent when a workspace viewpoint is in close proximity to an icon. Consequently, color is typically used for the most important fields, while shape encodings are reserved for fields of secondary or tertiary importance.The combined use of meaningful spatial distributions of icons, along with icon color and shape encodings, provides three simultaneous visualization channels for conveying information about the elements of Starlight databases. A fourth channel is provided through the use of text displays of the contents of text fields associated with the data. Users can elect to display text in one of twoFigure 1. Synoptic view of a Starlight workspace, showing representations of two free-text data sets, a collection of structuredinformation, and several supporting information resources.ways: statically, in which an entire text field is displayed adjacent to an icon, or via a novel 3-D text display technique that we call text streaming . With text streaming, individual words in a text field are consecutively displayed adjacent to their corresponding icon, one word at a time. Users can adjust the rate of the text display to suit their preferred reading speed. Perhaps surprisingly, most users adapt readily to this method after a relatively short acclimation period. The advantage of this approach, of course, is that large amounts of textual information can be rapidly conveyed using only very small amount of screen real estate. In fact, an entire document can be streamed to a user in the screen space required for the display of only a single word. Multiple text fields associated with multiple database elements can be streamed simultaneously, providing users with the ability to quickly absorb and compare large quantities of textual information simply by shifting the focus of their gaze.In addition to the interactive visualization techniques described above, a number of powerful data retrieval tools are available to Starlight users. Two general classes of query operations are possible, corresponding to the two general forms of analysis operations previously described.Content Queries operate on data element contents and return sets of elements that satisfy the query. Association Queries operate on the interrelationship model generated during preprocessing, and return maps of interrelationships among data element attributes.Association Queries enable a form of “link analysis” in which users can visualize and interactively navigate the complex network of associations that typically exists among the elements of any information collection.Three types of Content Queries are possible: Boolean Pattern Queries for searching for occurrences of specific patterns of words or numbers,Concept Queries , for retrieving text describing concepts similar to those specified, and Field Queries ,for searching the structured attribute information associated with database elements. The different types of content queries can be combined with one another in various ways, providing a powerful and highly flexibledata retrieval capability. The results of a content query are presented to the user as a ranked list of the database elements that satisfy the query. Simultaneously, icons representing database elements satisfying the query are automatically highlighted in the 3-D workspace view.This provides users with the ability to visually compare query results with other dimensions of the data, such as the spatial distribution of text elements describing particular concepts within the conceptual distributions of the database as a whole. Users can examine the full content of the information by selecting individual items from the results list for inspection, or by selecting highlighted icons in the workspace display.Association Queries provide the capability to visualize and interactively navigate the structure of connections between elements of a database developed during preprocessing. They generate a “snapshot” of the CCM database from a particular point of view. An Association Query returns a hierarchical data structurecalled an Array Set that describes the hierarchical interrelationships among database element attributes from the point of view of a particular attribute. For example, an Association Query for information related to a particular author might return an Array Set describing the dates of publication, publishers, titles, and abstracts, of all the publications by that author in the database. The hierarchical structure of the Array Set conveys the exact nature of the relationships among those attributes. NoteFigure 2. Close-up view of a Data Sphereshowing structured data setelements grouped according to the values in a given field.that an Association Query for information related to a particular publisher would generate a completely different “view” of the associations model.An Array Set resulting from an Association Query is presented to the user in a split view window [Fig. 3]. A tree representation of the Array Set is displayed in the left side of the window. The tree can be interactively explored by opening particular branches by clicking on them with the mouse cursor. The right side of the split view shows a 3-D graphical construct of the same information called a Link Net . The Link Net can be considered a hybrid visualization technique that incorporates features of the cone tree [1] and parallel coordinates [12] techniques.Like cone trees, Link Nets are representations of hierarchical information. The nodes in a Link Net correspond to particular field/value pairs. The query field/value pair used to generate an Array Set forms the root node of the tree, and is located at the top of the corresponding Link Net.Hierarchically related information is located beneath the root node on planes corresponding to the related fields. The planes are arrayed vertically on levels corresponding to the level in the Array Set hierarchy in which they occur. The planes are populated with icons representing the unique values associated with that field that have been returned by the query. Direct hierarchical relationships between field/value pairs are shown by lines that connect related icons.Note that associations described by Array Sets may include multiple, redundant occurrences of the same value for aparticular field in separatebranches of the tree. Theseoccurrences are coalesced into asingle icon representation,simplifying the displayconsiderably with no loss of information.Clearly, the information content of an Array Setcould be effectively communicated using a 2-D graphicalapproach. The use of the third dimension, however,provides an additional degree of spatial freedom withwhich to convey further information about the displayedinterrelationships. For example, consider the valuesassociated with a Link Net plane representing geographiccoordinates. Icons representing those coordinate values can be geographically positioned relative to one another to provide additional important information about theother attributes with which they are associated. Further,supporting information resources, in the form of maps and images, can be applied to the field planes to provide a visual context for the information presented in Link Nets.In the case of a geographic coordinate plane, links passing through coordinate value icons will automatically be routed to the correct point on the map, providing a geographic context for the coordinates. In a similar fashion, an organizational chart might be applied to a plane containing organization names. Links will automatically pass through the appropriate points on the chart. Likewise, icons representing temporal values can be positioned along a timeline to effectively convey temporal interrelationships.The Link Net geometry provides a detailed, yetcompact representation of the relationships described byits associated Array Set data structure. Note that LinkNets always have a plane containing icons representingthe data elements (e.g., documents) whose attributes areinterrelated within the context of the Array Set. Thisprovides fast access to the full detail of the original data atany time. Note also that color and/or shape encodings thatFigure 3. Array Set and corresponding Link Net visualization of information related to a particular transmitting call sign from a simulated intelligence database. Note the temporal distribution of the transmissions to receiving call sign Varic One, and the single geographic position of those transmissions.have been applied to the data are reflected in the data element icons shown in the Link Net display.Link Nets can be graphically queried and interactively filtered to highlight single, or sets of, linkages among the attribute fields. Clicking on an individual icon toggles the set of linkages associated with that particular field value. For example, clicking on a document icon will display the attributes associated with that document; clicking on the root node will toggle on or off all the links in the Link Net. If a Link Net contains a temporal field, a slider can be used to interactively filter links by the time values they pass through. If a map has been applied to a plane, the links will be “geographically filtered;” only those links that pass through the geographic window described by the map will be displayed. These filtering techniques provide the user with a high degree of control over display complexity, helping ensure that the information presented can be easily assimilated.A final important feature of the link analysis system is the ability of users to dynamically “hop” from the current Link Net display to one generated using any one of its field/value nodes. Performing the hop operation on a node results in the generation of a new Association Query using the selected field/value pair. The resulting Link Net is displayed in the same window, providing users with the ability to quickly navigate the complex network of associations among data elements.At the same time that a new Link Net is generated, another, more abstract, representation of it is also constructed in the 3-D workspace [Fig. 4]. This graphical analogue of the Link Net is intended to provide users with the ability to compare multiple sets of associations with one another, at a higher, more abstract level, as well as with large-scale informational features displayed in the workspace. This feature takes the form of a single icon representing the query field/value pair, called a Tie-Node. The Tie-Node is automatically located at the average 3-D coordinate of the data elements referenced in the Array Set, and is connected to them by rays. If an Ancillary Information Resource (AIR, e.g., a map) displayed in the 3-D Workspace has also been applied to one of the Link Net planes, then links connecting the Tie-Node to the appropriate locations on the AIR will also be displayed. Interactions with a Link Net that result in changes to its displayed links (e.g., filtering operations) are also reflected in the Tie-Node display. Tie-Nodes can be “expanded” as needed, generating direct connections between, for example, data element icons and AIRs, or “collapsed” back to their iconic form to simplify and better organize the display.The associative Link Net and Tie-Node displays, in combination with the other visualization techniques employed by Starlight (spatial icon distributions, color and shape encodings, text displays), enable the interactive generation of extremely rich information displays (Fig. 4). Such displays help convey, in an intelligible form, much more of the complexity inherent in multimedia information collections than is possible with traditional information analysis systems. Coupled with effective information preprocessing, storage, and retrieval systems, the visualization components of Starlight enable powerful forms of information analysis that hold the potential for greatly accelerating and improving the information analysis process.5 SummaryStarlight represents a new class of information system, one that has been purpose-designed for the visual analysis of complex, multimedia information. The software couples a variety of established and novel visualization techniques with a comprehensive data storage and retrieval system, in effect extending and enhancing the capabilities of more traditional information retrieval systems. At the same time, Starlight enables new forms of information analysis by promoting visual comparisons of information presented at many levels of abstraction simultaneously. This includes the capability to concurrently analyze multimedia information, such as image and map information, along with textual information. If display complexity can be effectively managed, the very high information densities possible in Starlight displays are expected to accelerate the information analysis process while simultaneously reducing the stress levels of analysts. The effectiveness of this approach will be tested in the coming months in a number of intelligence and commercial information analysis domains.6 AcknowledgementsThe PNNL Starlight Project has benefited from the efforts of a great many talented people. We would especially like to recognize our colleagues in the Boeing Information and Support Services, Natural Language Processing Group, Andrew Booker, Mark Greaves, Fred Holt, Anne Kao, Dan Pierce, and Steve Poteet, for contributing the semantic text indexing and display components of Starlight, and our colleagues at Applied Technical Systems, Inc., Ed Marquardt, Lance Otis, and Tom Bougan, for the customization and use of their Contiguous Connection Model software. This work is supported by the U.S. Army under Contract 26970, and the U.S. Department of Energy under Contract DE-AC06-76RLO 1830. The Pacific Northwest National Laboratory is operated for the U.S. Department of Energy by the Battelle Memorial Institute.。