Formal models for cognition - taxonomy of spatial location description and frames of refere

Formal Models for Cognition -Taxonomy of Spatial Location Description andFrames of ReferenceAndrew U. FrankDepartment of GeoinformationTechnical University ViennaA-1040 Vienna, Austriafrank@geoinfo.tuwien.ac.atAbstract. Language uses location description with respect to spatial referenceframes. For the transformation from a visual perception to the relativeexpression the reference frames must fix three parameters:•origin (e.g., the speaker, an object, another person),•orientation (e.g., the ax ial frame of the speakers, of the addressee, ofanother object),•handedness of the coordinate system (same as a person’s or inverse).These parameters characterize a reference frame. The paper describes themethods used in the English language and proposes exact definitions of ego-centric, intrinsic or retinal relative reference frames, and egocentric or allocentriccardinal relative reference frames. Invariants of descriptions with respect toclasses of reference frames are discussed and some hints for t he pragmaticpreference of one or the other reference frame suggested.The paper demonstrates two alternative computational methods for Levelt’sperspective taking, which deduces another person’s egocentric perspective fromthe speaker’s egocentric (perceptive) perspective. One method is assumingimagistic (analog) representations and the other method works with apropositional (qualitative) representation. Precise hypotheses can be formulatedin the formalized framework to construct human subject tests to differentiatebetween these alternatives.1 IntroductionTerminology and implied assumptions often confuse the discussion of cognitive abilities of human beings. An example for terminological problems is the extensive discussion of deictic versus intrinsic frames of reference to describe spatial locations, which is ongoing for many years without conclusive results. It has been pointed out that this was - at least partially - due to varying definitions of terminology: “The analysis of spatial terms in familiar European languages remains deeply confused, and those in other languages almost entirely unexplored.” (Levinson 1996, p.134) The standard solution for clarification of terminology is formalization: defining a few base concepts - preferably not from the discipline to be investigated, thus less likely to lead to confusions - and deriving other concepts from these by formaldefinition (Tarski 1946). This is often attempted with complex computer programs, simulating human p erformance (Gopal and Woodstock 1994). However, the resulting models are so complex, contain so much detail and are so difficult to read even for their authors that they contribute little to the clarification and formalization of the discussion.The current trend towards functional descriptions in cognitive science (Bierwisch 1996) naturally leads to models formulated in a functional language (Bird and Wadler 1988). Formalizations in a functional language can be read as definitions, but can be executed as programs to observe their effects. This allows to understand the interaction of definitions and to test our intuition. It becomes also possible to compare the defined behavior with observed behavior (Montello and Frank 1996).A detailed description is achieved through the construction of a formal model to produce the sentences to describe the spatial relations between objects as they are typically found in the literature (e.g., (Talmy 1988; Levinson 1996): “The bicycle is in front of the church”, “The ball is left of the chair”, etc.). The description presented covers the simple case, just sufficient to capture the ongoing discussion (e.g., (Levinson 1996)). This approach is based on the conviction that an explanation must list all the inputs necessary to produce the observed outputs; computational, formal models are the best way to check that all necessary details are considered. Applying Occam’s razor whenever possible, I strive for the simplest model and posit complications only when justified by observable behavior.A taxonomy of reference frames must be sufficient to define mechanisms which are sufficient to produce and differentiate the different verbal descriptions of spatial location using different reference frames. The formalization shows that the current taxonomy is not sufficient for this purpose. Levinson differentiates for English 3 cases: intrinsic, absolute and relative (1996). Levelt combines this with a differentiation between egocentric and allocentric, to give 6 different cases; he uses the term deictic to mean relative egocentric, intrinsic to mean intrinsic allocentric and absolute for absolute allocentric(1996). (A detailed discussion how these terms are used by the authors is delayed till section 11, when the geometric foundation has been laid). The current taxonomy is not sufficient to differentiate the use of left/right for another person or a stage (both are intrinsic in the standard terminology) (figure 1 left) or with respect to a person or an object (both are deictic) (figure 1 right):Der Ball ist links von Peter (The ball is to the left of Peter).-> from the observers perspective right of Peter,Der Ball ist auf der linken Seite der Buehne (The ball is on the left side of the stage).-> from the observers perspective on the left side.Der Ball ist links von mir (The ball is to my left).Der Ball ist links vom Baum (The ball is to the left of the tree).Front Left Front RightFig. 1. left) Two different uses of ‘left’ (right-handed and left -handed coordinate system)right) Two different forms of deictic reference frame (egocentric and retinal)The formal model presented here leads to the classification of reference frames by 3 values:• The reference object (ground ): the speaker, an observer or another object;• The orientation of the reference frame (which implies the selection of the frame type): speaker, ground, and direction from speaker to ground, or one of an externally fixed system (cardinal directions, up/downhill etc.);• The handedness of the reference frame (right- or left-handed).For English, 4 different situations using the ‘front, left, back, right’ terminology (in subsection 10.1.1) and 2 using the cardinal direction terms ‘north, west, south, east’ (in subsection 10.1.2) are characterized. The classification scheme is deduced from a formal (mathematical) model. The formalization can be used as code, which is sufficient to produce acceptable English descriptions for all cases. The classification can be used for other languages as well. The model is not intended to explain the pragmatics of the selection of one or the other perspective, the use of default values and ellipsis in general, etc.Three points seem novel in this formal approach presented here:• The model includes the world and the observing cognizant subject and gives a formal description of the observation operation. This can lead to quantitative modeling of errors in human performance (Frank 1998).• The model combines distance and directions in a single framework following the discussion in (Frank 1991; Frank 1991; Frank 1992; Frank 1996).• The m odel includes an imagistic and a propositional alternative of spatial reasoning. Specific hypotheses to differentiate between the two can now be formulated and tested.2 Overview of the Overall Cognitive Model UsedThere seems to be sufficient agreement on the overall content of a cognitive model to situate the discussion. For example, Jackendoff gives a ‘coarse sketch of the relation between language and vision’ (1996, p.3 Figure 1.3), which can be summarized as Figure 2.Fig. 2. Relation between language and vision (after (Jackendoff 1996), p.3)In an attempt to achieve a formal discussion, a simpler model (figure 3) is proposed, for which each step can be formally defined. It merges transformations Jackendoff separates into single units if a distinction seems not to be definable, but adds a transformation from the propositional representation of the environment to a perspective representation, following Levelt (1996 p.96, figure 3.11). It merges syntax and phonology, which are of less interest here. The perspective transformations - which are at the core of this paper - and the respective representations are then given a formally defined meaning in this paper. This fixes the intermediate representations and the code for the transformation. ‘Frame of Reference’ gets in this context a well defined, operational meaning: it describes the geometric principles that are used to transform from a primary geometric representation of the environment to a perspective representation, which contains all the elements to translate to a verbal expression by a linguistic (non-geometric) transformation.Unlike Jackendoff's model, the model in figure 3 includes a representation of the world. The representation of the world is not part of the ‘cognitive model’ sensu strictu, and is created and manipulated by special commands. The human observers or cognizant beings, in the sequel called EGO, exist in this world. The EGOs have the facility to observe the world - for example, by a visual channel - and to build an imagistic representation. From this imagistic representation a propositional (qualitative) spatial representation can be deduced by ‘internal inspection’. It would be possible that the subjects in the model (humans, animals) interact with the world and change it, but this is not modeled here (it would lead to models as used in Artificial Life research).Such a simple model is rich enough to posit a number of interesting and testable hypotheses. It can be formalized and provides the framework for other questions, a number of which are sketched at the end of this paper. The novel step is to include the world into the model; this permits to formalize the observation process (perception and cognition in one - here undifferentiated - transformation, following Talmy’s ‘ception’ (1996, with previous literature) in the same model.Fig. 3. The relation between language and vision used here In such a model the question of intermodal transfers - of which the first part of Bloom’s introduction is mostly concerned - can be discussed (Bloom, et al. 1994): what are the transformation functions, which translate from one to the other modality. One can posit a haptic observation channel from the world and then state precisely which kind of knowledge is acquired and how it is integrated with the knowledge gained through the visual channel. Is it integrated with the imagistic representation or with the propositional one (or partially one, partially the other)? The question can be posed much more specifically, because the formalization forces to define the observations that belong to a specific modality and the representation it builds.3 Formalization of the ModelI use here a functional programming language, specifically Gofer (Jones 1991), a derivative of the Haskell language (Hudak, et al. 1992). This is a language in the ML tradition, strictly typed with type inference (Milner 1978), which assures that all operations are called with arguments of the correct type. The language is referentially transparent, meaning that substitution is generally permitted as all variables retain the value they are initially assigned; side effects are not allowed. Code can therefore be read as high-school algebra, the left-hand side of an equation is defined as the right-hand side. The code can be obtained from the author.4 Representation of the WorldThe representation of the world is not part of the ‘cognitive model’ sensu strictu, but it must be included in order to permit to formalize a model of the observation function. The representation of the world - and in this case a spatial world - does not matter from a cognitive point of view; it is not part of the cognitive model, but is the model of theobject, the cognizant beings are observing. The model of the world is not the model of the concept of the world the EGO has. Only the observation function must be cognitively sound. The representation of the world must be sufficient to contain the facts that are necessary to produce the human knowledge about the environment that is compatible with human behavior.The world consists of a collection of objects that can be identified by a name. Some of the objects are cognizant subjects (persons), which can observe the world. The world is constructed at the beginning, but it could also be manipulated with special operations (for example, an object is moved). As a running example, the configuration in figure 4 will be used.yFig. 4. The example worldThe objects in the world are seen as points in a two dimensional plane without extension. We follow Levinson (1996, p. 135) in confining the discussion to the horizontal plane, as this is sufficient to reconstruct and clarify the current discussion in the literature. The objects are ‘axial’ (Landau 1996) and for each object, the location is given with 2 coordinates (location of the centroid) and an orientation. The orientation for things that have a natural orientation (for example, persons) is the azimuth for the ‘front direction’ (i.e. the angle with the positive x coordinate axis, measured clockwise) (see figure 5). Objects that do not have a natural coordinate frame are marked with OmniDir .5 Visual Perception and Imagistic RepresentationThe complete visual observation process is captured here - for simplicity - in a single process. It observes the world from the perspective of the EGO and produces a single -The extension and shape of the objects are not considered; this is sufficient to reconstruct the discussion by Levinson (1996), but would need extension to capture the wo rk of Eschenbach (this volume). The model of the world could be more complex, for example, space could be. This is left out here, however, to present the core of the modeling concept without additional complications.ego-centered - representation of the object positions. The visual obser vation is captured in a distance and direction to each object in the world on a ratio scale and in the relative orientation of this object with respect to the observer (see figure 5). At this stage, the arbitrary orientation of the world coordinate system, in which the position of the object is represented in the world model, is removed and replaced by the egocentric coordinate system.Fig. 5. The construction of the ego-centered representation The (minimalist) EGO consists of i ts own name, its orientation in space (which will be used to deduce an absolute representation - see section 8) and the observation of the world (as seen from its perspective). This ‘view’ consists of a list of objects with their name and the vector (distance and direction) from the observer (same as (O'Keefe 1996)) to these and their orientation (with respect to the observer’s orientation). One could introduce limitation to what is visible from the world, but this is not done here.It is assumed here that the observer builds first a representation in which he uses himself and the organization of space as it emanates from himself as ground and represents the other objects on this ground (egocentric view). Since Piaget, there is an extensive discussion how allocentric representation of space is built from such an egocentric representation, but the observations reported in the literature are not sufficient that a specific, more elaborate formal description can be justified possible. For present purposes, an egocentric representation is sufficient.6 Propositional Representation for Direct (‘Egocentric’) View Following the model of Jackendoff (1996) an abstract propositional representation is deduced from the imagistic representation. An observation function accesses theimagistic representation and deduces relative positions for each object in a propositional representation. Little is known about the encoding of this propositional representation. Levelt and Levinson report experiments, which seem to come to contradictory conclusions (Levelt 1996; Levinson 1996).We assume here qualitative representation in an egocentric system (figure 6), which differentiates 4 distance relations, where each successive range reaches twice as far as the previous one and 8 equidistant directions, following the schema proposed in (Frank 1992; Hong, et al. 1995). This system seems ecologically plausible; reasoning with directions, human performance gives approximately the same level of errors as a model with 8 direction cones (Montello and Frank 1996). Distance is encoded in zones: the zone up to 1 unit is here , between 1 and 2 units is near , 2 to 4 units is far and further is very far(see figure 9).Back Left herenearfarvery farFig. 6. The qualitative distances and directionsThe transformation discretizes for each object the distance and the direction value in 4 levels for distances and 8 values for the direction and replaces the quantitative representation in the imagistic representation by a qualitative representation (“Ball” 3.2 45) becomes (“Ball” Far FrontRight). The propositional representation of the world by the EGO is a propositional, qualitative encoding of the imagistic one. This means that the vectors are discretized (i.e. distances encoded as far, near, etc. and directions as front, left, etc., directions by 8 cardinal direction values). Jackendoff’s model assumes that in the propositional representation sufficient information is available for the production of the linguistic code. Sentences likeSimon says:Paul steht links vor mir. (Paul is to my front left) Der Sessel steht gerade vor mir. (The chair is in front of me)etc. can be produced. This produces the most direct representation of a spatial situation. It is often called intrinsic, but Levinson shows the difficulty of this label. We propose the term egocentric for this. In Levinson’s characteristic, it is described as anintrinsic coordinate system, with origin at the speaker and using as relatum (gro und) the speaker.7 Perception from Other PerspectivesIt is generally observed that people are capable of transforming their perception of space into the perception from a different point - Levelt’s ‘perspective taking’ (Levelt 1996). This ability is fundamental to understand other people’s description of space from their perspective. This is often called the intrinsic or deictic description of the relative position of objects: the position of an object (figure) is described not with respect to the speaker but with respect to another object (ground) as if this were the observer. Very often the addressee is used as ground, as in:Peter speaking to Simon: Der Ball liegt vor Dir. (The ball is in front of you)but any person can be used as a ground (relatum):Peter speaking to Paul: Der Ball liegt vor Simon. (The ball is in front of Simon).This section concentrates on the deduction of the relative position of an object with respect to another one. Perspective taking consists of three steps:PaulPaulSimonvector subtraction rotation + mirroringrotationFig. 7. vector subtraction: V ground->object =W observer->object - U observer->ground, rotation and mirroring1. ‘Path completion’ (Frank 1992; Hong, et al. 1995) or vector subtraction, computesthe vector from the ground to the figure given the vector from the observer to the ground, and the vector from the observer to the figure, (figure 7 left). Vector subtraction gives (in the coordinate system of the observer):(vector Simon - chair(w)) - (vector Simon Paul(u))= (vector Paul - chair(v))2. To achieve Paul’s view one has to rotate the result into Paul’s orientation (figure 7middle).3. The handedness of the resulting coordinate system has to be selected, whichdecides how the reference axis of the ‘new’ coordinate system is or iented (figure 7 right). This differentiates the two cases:Der Ball ist links von dir (the ball is to your left).Die Muenze ist in der linken Schublade des Tisches (the coin is in the left drawer of thetable).Two computational paths could be used to deduce this information: The relative position between two objects observed is either deduced from the imagistic representation or from the discrete representation - using the same formulae. One must assume that in the imagistic representation, some analogue computation similar to vector addition and subtraction can be performed. In the discrete representation, vector operations are carried out as table look up, and the combination for all inputs is stored in tables.7.1 Construction of an Imagistic Representation from Another PerspectiveThe person constructs the imagistic (2D top view or distance-direction vectors) representation as it would be deduced from the other perspective. This is, mathematically, simply a transformation of origin and orientation of the imagistic representation gained directly from one’s own perspective. For the distance-direction representation, it amounts to the subtraction of vectors, rotation and mirroring. The results are exactly the same as if the other person had observed the world.From this imagistic view from another person’s perspective the discrete representation can be deduced by the same discretization as described in section 6. 7.2 Deduction of the Propositional Representation for Another Perspective from thePropositional Ego-Centered RepresentationThis requires a qualitative reasoning about the combination of distance, directions and orientations. Simon reasons (figure 7): if the chair is in front of me and far, and Paul is front-left of me and very far, then the chair is (in Simon’s orientation system) left and near from Paul. The transformation to Paul’s orientation gives: if the chair is in Simon’s orientation system left and near from Paul, and Paul is facing left (in Simon’s system), then the chair is in front (and near) from Paul.The deduction of qualitative relations is known as relation combination and traditionally written with the ‘;’ operator (Schroeder in (Bird and Moor 1997)). In the literature several proposals for qualitative relation combination calculations have been made. They are mostly influenced by Allen’s calculus for time relations (Allen 1983) and follow a mathematical tradition, to give as a result the conjunction of all possible results. If a combination of two relations can more than one relation, the disjunction of these is given as a result (written as a list of terms connected by or).This was applied to direction reasoning by Freksa (Freksa 1991; Hernandez 1993) and to reasoning with topological relations (Egenhofer, et al. 1994). I have proposed an approximate mode for the composition operation, which gives always a single relation for the combination of two relations - the most plausible or most likely one (Frank 1992;, Egenhofer et al. 1995; Frank 1996). This is cognitively more plausible, because there are no indications that humans would handle disjunctions of relations. It agrees with the observed ‘preferred model’ tendency of human subjects (Knauff, this volume).The method used here is based on 4 distances and 8 directions, which results in 32 different positions. It is possible to devise methods using symmetry, such that not the full 32 square matrix for all combination must be present in memory, but only about 120 entries; thus a propositional table look-up seems mentally possible (comparable to the memorized multiplication tables with 100 entries).7.3 Comparison of an Imagistic or a Qualitative Transformation to AnotherPerspectiveIn principle, perspective taking should produce the same result whether applied to an imagistic or a propositional, qualitative representation:discretize (a-b) = discretize (a) - discretize (b)Due to the errors introduced by the discretization process, the deduction i n the discretized (qualiative) representation is only an approximation. This difference can be used to perform experiments, which help to decide which process humans use.Given the data reported in the literature, it cannot be decided if the perspective transformation is performed before or after translation to a qualitative model. We have shown here that a qualitative model is possible and can be formalized. Experiments with human subjects will be necessary to answer this hypothesis; the model here presented is the necessary formal framework to formulate a precise hypothesis and design the corresponding experiments.8 Construction of an Absolute Frame of ReferenceHumans have a potential to refer to the location of objects in an absolute frame, in English mostly the cardinal directions, but also up/down in a valley, towards the sea are used. These are particular cases of perspective taking, different from the cases discussed in section 7. There only the relative position of figure, ground and observer were relevant. If the objects in a relative frame are rotated together, the same expression results. In an absolute frame of reference the directions are invariant under individual rotation of the speaker, the observer or the ground, but if the whole configura tion rotates, then the expression changes. If the speaker, ground and figure all together are rotated by a quarter turn, then what was north becomes west (figure 8).Fig. 8. After rotation ‘the ball is north of the chair’ becomes‘the ball is west of the chair’The difference in what is and is not affected by changes is likely to affect the pragmatics of selecting an absolute or a relative reference frame: Absolute directions,are invariant under individual rotation of the speaker, the observer or the ground object. What is to my north is also to your north, independent of our relative position and orientation (provided we are relatively close together with respect to the figure). Absolute directions are therefore mostly used in geographic setting, where both distance relations are given and rotation of the configuration is not possible. The invariance under rotation of the individual objects and the (relative) invariance to the small changes in the positions of ground may explai n why rural people pragmatically prefer absolute frames. Direction expressions related to the body frame are invariant if the full configuration (speaker, figure, and ground) are rotated together; this compensates for the uncertainty of the position in an absolute frame typically experienced indoors. This would explain the preference of urban people for expressions related to the body frame (Pederson 1993).To construct and understand this kind of spatial relation, the EGO needs to know its orientation in this absolute frame. Then all the orientations can be translated from the egocentric reference frame to the absolute reference frame with a single subtraction of angles. This operation can be performed - similar to subsection 7.1 and 7.2 - in the imagistic or in a qualitative, propositional representation.It is worth noting that absolute directions depend also on the ground object; only the orientation of the reference frame is given by the cardinal direction, or the up/down valley direction etc. This is often not noted, because absolute directions are in English mostly used to express positions in geographic space where the speaker and the listener are relatively close. This advantage is lost when used in proximity: Simon says: The ball is to the west.Peter says: The ball is to the east.9 Definition of a Frame of ReferenceThe computation for ‘perspective taking’ has as input a view and produces another view - for example, the view, as the observer would have it. Section 7 offered an imagistic or a propositional method for the computation in detail. In both cases and for a relative or an absolute reference frame (section 7 and section 8), the computational model shows that a new frame of reference must be specified with three characteristics:•the origin of the coordinate system (independent if one assumes the conventional orthogonal coordinate system or not);•the orientation of the coordinate system, given by the direction of its primary axis (even if the secondary axis is not orthogonal to it);•the handedness of the coordinate system (i.e. the relations of the axes).Any description of a spatial relation is thus a relation between: the figure F and the ground G as set up by the reference frame defined by an Origin (which is the ground), an Orientation and a Handedness. All of these can be left away in a verbal description and replaced by default values (for example, ground is speaker, the origin and orientation of the reference frame is to be taken from the speaker and the handedness is conventional right-handed). The abundant use of default values in verbal description and the fact that certain combinations do not or even cannot occur, has certainly led to the confusion in the characterization of spatial frames of reference.。