MULTIDIMENSIONAL HYPERPRESENTATIONS IN A DISTRIBUTED MULTIMEDIA SYSTEM

ITAD-730-CP-93-189 3rd Australian Multi-Media Communications, Applications, and Technology WorkshopWollongong, New South Wales, 14-15 July 1993 MULTIDIMENSIONAL HYPERPRESENTATIONS IN A DISTRIBUTED MULTIMEDIA SYSTEMEarl Rennison, Kate FinnInformation, Telecommunications, and Automation DivisionSRI InternationalMenlo Park, California, U.S.A.{rennison,finn}@DooHyun Kim, Young-Hwan LimElectronics and Telecommunications Research Institute, Daejeon, KoreaAbstractWe have defined a model for multidimensional hyperpresentations, in which time-varying multimedia presentations are dynamically linked together. In our model a hyperlink’s context can change over time. Thus, at any time during a continuous presentation, a user can interactively access context-specific information. How the hyperlinks are traversed depends upon user interaction and/or context-sensitive triggers. This generalized model can be applied to solve many issues associated with multimedia information representation and access.1.INTRODUCTIONVideo can be a very powerful tool for communicating information and ideas. Unfortunately, video is a serial medium, so that searching for information entails cumbersome navigation among nonsequential segments. Digital interactive video has made the navigation process easier, but has not solved the problem of finding information of interest, such as a particular video segment. In contrast, hypermedia systems provide effective techniques for precisely navigating through and accessing information spaces, thus enabling users to quickly find information of interest via a more natural interface [1; 2]. Attempts to merge the richness of video with the navigability/accessibility of hypermedia, however, have not provided adequate support for multidimensional data.The term multidimensional is used in this paper to include the full spectrum of media: multiple audio and video tracks (layering); multiple channels (such as left and right audio, video, and mouse inputs); 3-D graphics and 3-D audio; and time-varying information. The term hyperpresentation refers to the concept of linking time-based multimedia presentations together, based on the time-varying context of the presentation. For example, consider a scenario where the results of research projects are stored in a database as multidimensional hyperpresentations. A person browsing through such a database could initially be presented with an overview of a selected research projects. The overview could be a concise video containing a description of each of the key results of each project. If, while viewing the information, the person finds a topic of interest, he or she could press a “Tell Me More” button that would alter the video to show more detailed information on the specific topic being discussed at the time the button was pressed. Or further, the person could use a mouse to click on a specific item that he or she sees on the screen; the video would freeze, and another window would pop up, showing a video that relates to the item selected. This example of hyperpresentation is illustrated in Figure 1.access, current support services for this type of interaction are not sufficient. To adequately address this problem, a need exists for (1) a model to express the multidimensional relationships between related media, as well as modeling user interactions, and (2) mechanisms for executing the specified relationships, such as hyperlinks, in real time.We have defined a model for multidimensional hyperpresentations, in which multimedia presentations are logically linked so as to occur either sequentially or simultaneously. This model includes the time-based linking of multimedia presentations, based on the contexts of the presentations, whereby each presentation’s context is defined by markers and segmentsdelineated over time. The generalized model we have defined can be applied to solve many of the issues associated with multimedia information representation and access.2.MuX DISTRIBUTED MULTIMEDIA SYSTEMIn this section, we will present a very brief and high-level overview of the multimedia data processing model employed and the MuX distributed multimedia I/O system that uses the model. A more detailed description of the MuX model is provided by Baker et al. [3].2.1MuX Multimedia Data Processing ModelThe MuX model comprises a stream layer, a multimedia presentation layer, and ahyperpresentation layer, as shown in Figure 2. The stream layer provides services similar to those of a video router used in video production studios, where sources and destinations may be local or remote files or devices (such as microphones, musical instruments, video cameras,displays, and audio speakers). The multimedia presentation layer services are based on the concept of a programmable media multiplexer or media mixer, such as an audio mixer used in concert productions, or a switch or video editor used in a video production studio. In this layer,the media multiplexer takes input from a variety of sources, mixes them according to controllable parameters, and directs the result to an output port or destination. Thehyperpresentation layer is a generalization of the links described above, where the documentOverview Subject Bmay include time-based media such as audio and video, and the links are dynamic and vary over time. In the paragraphs that follow we describe the services provided by these three layers in greater detail.The stream layer provides services to access a multimedia data stream from a file, a device, or a network connection, or from the higher layers; performs processing on the stream (e.g.,compression or format conversion using filters); delivers multimedia data to a destinationconsisting of either a file, a device, or a network connection, or to the higher layers, in a timely manner; and allows the selection of an input from one or more streams and distribution of the input to one or more destinations using a switch. To provide these services, several abstractions and mechanisms have been defined, including a stream, source, destination, filter, and switch.These mechanisms are illustrated in Figure 3.The multimedia presentation layer builds on the stream layer. A multimediapresentation is a collection of streams that are coordinated with respect to time and space. Streams within a presentation are synchronized and have shared presentation control. There are logical groupings of media streams for integration and media-specific presentation control. The streams of dissimilar media (e.g., aural and visual) are synchronized and presented simultaneously.Streams of similar media can be cut, reordered, processed, and mixed to form a new stream:such streams are grouped together as a channel for presentation or further processing. Key components of a multimedia presentation include tracks, channels, and presentations.Tracks are time-ordered streams that may be reserialized via a set of clips or a clip list .Channels are a group of tracks that are mixed together to form a new stream. A presentation is the complete set of synchronized channels. To provide these media integration and synchronization services, the following mechanisms have been defined: logical time system (LTS), cue, presentation context,Figure 2.Multimedia Data Processing ModelStream Stream Figure 3.Stream Layer MechanismsDestinationSourcemixer, clip, track, channel, and presentation. The relationships among these mechanisms are illustrated in Figure 4. The presentation, channel, track, and clip mechanisms are used primarily to specify timing and synchronization relationships and are described in further detail in Subsection 2.3.A presentation context (PC) specifies parameters that define how media streams are mixed,integrated, and presented to the end user. PCs are specific to the medium associated with a stream. The presentation context value associated with a PC can vary over time, offeringconsiderable flexibility in defining how media streams are integrated and mixed. For example,a PC for an audio stream would specify the volume at which that stream is presented. Thevolume can be specified as a function of time, as shown in Figure 5. Presentation contexts also define parameters like origin and extent of video windows.The highest level of the three-layer model is the hyperpresentation layer. As noted above, the abstraction of the hyperpresentation layer is that of a dynamic network of multimedia presentations connected by links, forming multidimensional (time and space) multimediapresentations. Links, or hyperlinks, that connect multimedia presentations together are dynamic in that they change over time and have fixed lifetimes during which they can be activatedFigure 4.Multimedia Presentation MechanismsV ol u m e Volume = “interval(0,3)*40 +Figure 5.Time-Based Presentation Context[4;5;6;7;1]. Within the context of this model, hyperlinks can be associated with tracks, channels, and presentations, and can link them to any of the aforementioned components, as illustrated in Figure 6. A “publication” binds together a set of presentations linked together withFigure6.HyperLinks between Multimedia Presentation Components hyperlinks. The hyperpresentation layer, including hyperlinks, is the focus of this paper and will be described in detail in subsequent sections.2.2The MuX Multimedia I/O SystemWe have implemented a distributed multimedia I/O system, called MuX, based on the multimedia data processing model described in the previous section. MuX supports a set of distributed multimedia I/O services, including the integral services of media translation, compression, and synchronization. It provides network-transparent access to stored multimedia data, real-time multimedia I/O devices, and multimedia processing. The MuX system supports the routing and delivery of real-time multimedia data; real-time video, audio, and graphics acquisition; and media processing and output.To facilitate its support of a distributed environment, the MuX system is based on theclient-server paradigm. The system consists of a multimedia I/O server [8]; a presentation manager; the MuXLib client library, which embodies the MuX application programmer’s interface (API) [9]; and a scripting language [10]. In this environment, the client application specifies the timing and synchronization relationships among media streams, and the multimedia I/O servers exchange real-time data directly among themselves to meet these specifications.The system services are defined according to object-oriented concepts, both in design and in implementation (using C++). Though most of the services are implemented in the server, the client library utilizes a distributed object messaging service that allows objects to send messages to remote objects in a network-transparent fashion, giving the applications the illusion that objects are being executed locally when in fact they may be executed on another machine. Thus, application programmers can develop distributed applications that transparently utilizereal-time resources on multiple machines, without having to deal with the complexities of interprocess communication and coordination of resources.2.3Treatment of TimeThe notion of time is central both to the hyperpresentation implementation and to the MuX model in general. Time may be specified in several different ways, including absolute real time or wall-clock time; relative real time specified in seconds and/or microseconds from a start time (which may be either absolute or relative); or time specified via the SMPTE time code*(e.g., 00:07:05.10). Since the playout times of multimedia presentations are not known in advance and are determined by user interaction, it is extremely useful to specify timing relationships in terms of relative time. For this reason, all timing specifications are defined in either relative real time or SMPTE time code, which is inherently relative.The MuX system supports four methods for specifying timing and synchronization relationships between media: (1) hierarchical relationships specified by the relationship between presentations, channels, and tracks, where track times are relative to a channel’s time, and channel times are relative to a presentation’s time; (2) timing relative to a time-line using either a logical time system (LTS) [8] or a relative real-time system (RRTS) [11; 12]; (3) relationships between streams using cues, e.g., “before,” “after,” “starts,” and “finishes”[13;14;15]; and (4) direct interactions with the user via the application. These timing and synchronization specifications are illustrated in Figure 7. Note that during execution the relative time system is elastic, meaning that it can stretch and shrink by changing the speed of the presentation. Guimaraes et al. [15] have developed a set of tools for defining temporal relationships between media types, and a system that supports periodic, time-event, and state-change actions; their work is based partially on Allen’s set of thirteen temporal relationships between pairs of objects [16]. Similar relationships have also been explored [1;2;4;6;7;15;17;18;19;20;21]. In the MuX system, the graphical specification of relationships is not explicit; rather, it is implicit, expressing the start-time and end-time or duration by means of a cue that is associated with another component. The timing specifications employed by MuX are fundamental to hyperlink trigger detection and will be discussed in further detail in the following section.3.HYPERPRESENTATION IMPLEMENTATION“Classical” hypertext is discussed in terms of links and nodes [22]. However, the definitions of links and nodes have varied among authors, researchers, and applications [5; 12; 23]. For example, nodes could be the current and subsequent points of reference, and a link could be a definition of the way to traverse from one node to another. Nodes could contain bits of information, and links could define how these bits of information relate to one another. In the context of the MuX model and system, we define a hyperpresentation node to be a medium stream or a set of media streams associated with a span of time; and a hyperlink to be a set of parameters and actions controlling the transition between two hyperpresentation nodes. In terms of the model defined in Section 2.1, the hyperpresentation node may correspond to a track, a channel, or a presentation.*The SMPTE (Society of Motion Picture and Television Experts) time code is an 80-bit standard for representing time specified in <hours>:<minutes>:<seconds>.<frames>, and includes other synchronization and user-specified information.3.1Classes of HyperlinksThere are many ways to categorize classes of multimedia hyperlinks. For example, hyperlinks might be categorized by whether they connect nodes that operate simultaneously or nodes that operate sequentially; or by whether or not the hyperlinks return the user to the originating node.Brondmo and Davenport [4], in the “Elastic Charles” project, used links whose physicalappearance denoted the type of media they connect: a link to a video segment, for example, was indicated by a micon , or motion icon . Ogawa et al. [7] categorize links as user driven, time driven, or condition driven. Broadly speaking, we have classified the links in MuX according to•Their activation periods•Their context sensitivity•The plasticity of their activation areas.**Another term for “activation areas” is “hotspots,” defined by Michon as: “scalable, polygonal regions within images that trigger actions on pointer events such as mouse enter or mouse button presses. From the usersperspective they behave as transparent buttons that exactly conform to an object in a video image...Once hotspots are defined, they are automatically scaled to register correctly with a video image even if the image is resized,moved, or stretched.” [6, p. 365].Figure 7.Timing and Synchronization SpecificationHierarchical relationships Relative to time line Relationship between two streams using cue User controlLEGENDWith respect to activation periods, some links are active throughout the “life” of their associated track or channel. For example, a fixed “Tell Me More” button can be presented on the main window at all times (its location and appearance don’t change, although what it points to might). On the other hand, some links are active only during very specific portions of a presentation, such as a link associated with a particular person’s image within a video; that link is active only when the person is on screen. We denote the span of time that a link is active as a segment. Segments are generally associated with the underlying content of the presentation. The notion of a segment is similar to the concept of stratification described by Aguierre Smith and Davenport [24].The Tell Me More button mentioned above is an example of a context-sensitive link: the location pointed to by the link may vary according to the current position in a presentation. Other, less dynamic links may always point to the same place, regardless of when they are activated. The Tell Me More button is also an example of a link with a nonchanging activation area: its location, extent, and appearance do not change over time (although a Tell Me More button could conceivably change from dark to light to indicate that it points to more information). However, a highly changeable hyperlink could be associated with a moving object, such as an airplane changing direction and approaching the camera within a video. The activation area for this link is highly plastic; it can change its location, extent, and appearance (the outline of the plane).A special type of link is the automatic context-sensitive trigger, which enables an author to link pieces of information together automatically, and also enables users to determine which of these automatic links is traversed while they view a presentation, based on the context in which the viewing takes place. For example, a “More Detail on ‘Time’” button may be offered while a user is viewing a presentation on this paper. If the viewer selects this button prior to initiating the presentation, and a point in the presentation is reached where the concept of “time” is introduced, the presentation would automatically pause and a link to a related presentation would be activated.3.2Hyperlink ParametersOne may assume that, at a minimum, links need to identify where they point, and what action causes them to be activated. Links can also contain much other information. In this section, we list and briefly define the categories of hyperlink information used in the MuX system, and present some simple examples. In Section 4, we will present more detailed examples of hyperlink usage.Identifier: A unique character string that can be used in expressions andprocedures to refer to a particular link; optional; e.g., “FirstJump,” “143,” “videoIntro.”Control Stream:The type of stream containing the triggering event: e.g., mouse,microphone, keyboard.Trigger Event:The event that activates the hyperlink, e.g., <control-C>,mouseWithin, leftButtonDown, “jump” or “link” (spoken); morethan one trigger can activate a link [6;15].Action:What happens when the link is activated; multiple actions can betaken when the link is activated; e.g., stop presentation of currentvideo and audio tracks, present specified video in currentwindow, open new window and present specified video in it.Start Time:The initial point at which the hyperlink is active; the time isexpressed as relative to the track, channel, or presentation inwhich the hyperlink is defined.Duration:The length of time in which the hyperlink is active.Appearance:The physical form taken by the hyperlink’s activation area; e.g.,dashed or solid line; opaque or transparent; circle, square, ormicon [4;6]; outlined or invisible.Origin:The location of the upper-left corner of the physical activationarea, expressed as the pixel specification of the x and ycoordinates (and z coordinates, for 3-D graphics and animation).The specification of the origin of hyperlink’s activation area inwhich the physical link changes location over time can berelatively complex.Extent:The shape or bounding region of the physical activation area,defined by its maximum length and height (and depth, if 3-D). Aswith the origin, the extent can change over time.Visual links can contain any of these parameters; audio links do not contain the parameters for appearance, origin, or extent because they have no physical activation area. Note that duration can be expressed either as an explicit length of time, or as the difference between a supplied end time and the start time.3.3HyperLink MechanismsThe key objective of the hyperlink mechanisms is to detect when an event in the control stream meets the trigger conditions specified by the hyperlink. Since the underlying medium or media stream(s) associated with a hyperpresentation node may change over time, as with video, these mechanisms should execute in real time and be closely tied to the presentation of the medium or media stream(s).There are a couple of different approaches to achieving this objective. One method is to mark the individual bits associated with the hyperpresentation media stream to indicate that they are associated with a particular activation area. For example, each pixel of each frame of a video stream could be marked to indicate that it is associated with an activation area on which a hyperlink’s trigger is based. While this method may be effective in determining exactly when an area has been selected, it would require a significant amounts of storage space for specification and time for an author to specify which pixels are associated with an activation area. Thus, until high-speed image recognition mechanisms are available to do on-line processing of video streams, this method is not viable for video.An alternative approach is to specify the hyperlink regions of interest as a formal expression. Depending upon their complexity, expressions can be used to describe the location and shape of activation areas as they change over time. For example, consider an activation area associated with an object that enters a video clip from one side, gradually moves across the screen, and exits on the other side. This movement can easily be expressed as a linear function of time, whereby an author would only be required to specify the location and size of the activation area at two locations in the video sequence.Given the current computing capabilities of workstations and the authoring flexibility afforded by the latter method, we have based our hyperlink trigger mechanisms on formal expressions expressed as functions of time. The mechanisms that embody the specification of the parameters that define hyperlink nodes, execute the hyperlink trigger detection in real-time, and perform the necessary operations to instantiate the hyperlink include the following: Condition Context: A condition context describes an event that triggers a hyperlink.It specifies parameters such as trigger event, and activation areaorigin and extent. Each of these parameters may be specified asan absolute value or as a formal expression stated as a function oftime. A condition context is responsible for evaluating theseexpressions to determine their value at specific instants of time.Control Stream: A control stream, as noted above, is the mechanism that activatesa hyperlink when it meets a condition context.Comparator: A comparator is the primary mechanism for detecting a link ortrigger condition and executing the actions necessary to link thepresentations together. To perform this operation, a comparatorhas associated with it a control stream(s) and a condition. Duringoperation (between the start time and the end time), a comparatorcontinuously compares the input from the control stream to theassociated condition that activates a hyperlink or trigger. Whensuch a condition is met by the control stream, the comparatorexecutes an action to hyperlink the presentations.Action:An action is a set of programmatically defined operations thatperform a hyperlink traversal. When a comparator determines thata condition is met (e.g., that a button has been mouse selected atthe location of a hyperlink), the comparator executes the actionassociated with the hyperlink.In addition, a hyperlink mechanism bundles these mechanisms together and associates with them the appearance of the activation area, and the temporal aspects of start time and duration. The mechanisms are illustrated in Figure 8.Figure 8.HyperPresentation MechanismsNote that a key issue in the implementation of real-time hyperlink trigger detection is coordinating the temporal relationships between the presentation mechanisms (i.e., the mechanisms that present what the user sees and hears) and the trigger detection mechanisms. This issue is further complicated by the use of presentation contexts. As a basis for this synchronization coordination, the hyperlink mechanism uses the start time, which is relative to the presentation medium, and the duration. Specifically, the start time and duration are used to register a “cue” with the medium stream with which the hyperlink is associated, as illustrated previously in Figure 7. The cue mechanism keeps the hyperlink mechanisms in synch with the timing of the presentation by sending update messages as to the timing status of the presentation in relation to the hyperlink, such as “start,” “stop,” “pause,” and “change of speed.” The hyperlink passes these messages to the comparator, which uses them to correlate the timing of the control stream with the timing specified by the condition context.3.4HyperPresentation Scripting and the MuX APIAn integral support facility for hyperpresentations is a scripting language that allows users to directly or indirectly specify the relationships and hyperlinks between media[1;6;2]. The MuX system provides a scripting toolkit that allows users to script hyperpresentations. The MuX scripting language [10] is an object-oriented language that provides facilities to define publications that may consist of multiple presentations (and their associated channels, tracks, and so on) and hyperlinks between the presentations. The action definitions in the script specify execution-oriented operations such as “play,” “pause,” and “stop.”There is a tight relationship between the scripting toolkit and the client library, whose interface is defined by the MuX API [9]. The client library provides facilities to construct objects on the server and set the object parameters. The objects constructed are primarily control structures used for executing real-time presentations. After the objects have been constructed and the control structures defined, the API defines execution operations, such as “play,” “pause,” and “stop,” that allow applications to control the viewing of multimedia presentations. These same operations can also be used by the script language to define hyperlink actions. Thus, a hyperpresentation script can model user interactions and define the appropriate responses. 3.5HyperPresentation System OverviewTo gain an overview of how the MuX system implements a hyperpresentation, consider the interactions, illustrated in Figure 9, between the MuX system components (the application, scripting toolkit [10], client library [9], and server) to load a script and execute a presentation. This operation is initiated by the application, which instructs the script toolkit to load a publication. The publication defined by a script specifies the construction and parameters of stream, multimedia presentation and hyperpresentation objects. Messages to construct these objects are propagated through the system, as illustrated in the leftmost column in Figure 9. After construction of the objects that define a publication, the application can start the hyperpresentation by sending a “play” message to the “root” presentation specified by the publication. As shown in the second column from the left in Figure 9, this play message bypasses the script toolkit and is relayed to the appropriate presentation object in the server. This presentation object in turn sends messages to its associated channel, track and stream objects to execute the presentation.If the user triggers a hyperlink while the presentation is playing, this triggering event is detected by the comparator associated with the hyperlink. When this triggering event is detected from the control stream (which may be generated from a user’s mouse inputs), the action object associated with the hyperlink sends a message to the client library, as illustrated in the third。