– Scene Geometry – Scene Appearance

Projective Image WarpingAs suggested by its name,image based rendering is concerned with creating new images from old ones.Recently there have been lots of descriptions of novel representations,such as light-fields, and the warping of depth images to produce3d like parallax effects.But the basic idea of using old images to make new ones is quite old.In particular the use of texture mapping takes input image data(the texture),places it in some virtual3d world,and views it(texture mapping)on the image plane.The original texture image is warped onto the new image plane using what is called a projective warp.In this section we will discuss a simple family of warps called projective warps.These warps do not use any sort of depth or opticalflow information.Warps that use depth andflow will be discussed in detail later in the course.We will demonstrate two powerful ideas.First that if one views a planar object using two(or more)pin hole cameras,then those two images are related by a projective warp;ie.if we apply the warp to thefirst image,we obtain the second image. This result is at the core of texture mapping.In texture mapping,we use one image,the texture, to represent the photometric appearance of a(nearly)planar object,and this is viewed from an arbitrary viewpoint to create a second image.The second idea about projective warps we will demonstrate is that if one creates two(or more)images from a single center of projection(the camera only rotates,and changes its intrinsic properties,such as its zoom),then those two images are related by a projective warp.This result is at the core of environment mapping and panoramic images.If one takes a number of pictures from a single point of view,one can stitch this panorama of images together using a projective warp. From this data one can simulate a new view from the same center of projection by applying the projective warp.In this section we will discuss two types of applications and benefits of simple warps.These two types applications run through the core of most IBR work.First image warping lets us use photographs of real world scenes in an interactive graphics setting.Second,image warping is fast, and can allow us to render imagery faster than other techniques.First,simple image warping allows us to easily model complicated real world appearances that may be difficult to model manually.By using a photograph as a texture,one effectively captures real world photometric information,and places it in a virtual environment.Projective warps know nothing about the depths of objects in the world,and so they can be applied to real photographs without any sort of active scanning techniques or vision-type analysis.We discuss texture mapping and how it is a simple example of a projective image transform.We discuss environment maps and image panoramas which give a simple description of the world as seen looking out from a single point.These panoramas offer a simple way to photographically capture interesting real world environments in which to place a virtual user.The second benefit of simple warps is fast rendering.Image warping involves simple and regular computation.It can therefore be computed very efficiently,especially with the use of hardware.Texture mapping is useful,even if the texture is generated by a computer,and not from a photograph.This efficiency of image warping is used to render complicated appearance, (even synthetically modeled appearance)more quickly than if we were to use simpler rendering techniques,such as smoothly shaded polygon rendering.These simple types of warps can also be used to speed up the rendering of purely synthetic en-vironments.In typical virtual settings,as the user or objects move around,many parts of the imagestay the same,or perhaps warp in simple ways.V arious rendering systems have been designed to make use of this temporal coherence by essentially“reusing”various image regions.These system rely on similar types of image warps seen in texture mapping and environment mapping.In this section we will discuss these systems as well.Finally,to draw the connection between projective warping,and image based methods that use depth information,we will discuss the relationship between these warping methods and the warping methods that have motion or depth information associated with the pixels.Basics:Projective MapsA projective map is a mapping that maps from one image coordinate system x i1y i1to another image coordinate system x i2y i2.It is computed multiplying thefirst coordinates by a3by3 matrix,and dividing out by r.x i2r y i2rr m1m2m3m4m5m6m7m8m9x i1y i11Explicitly this can be written out asx i2m1x i1m2y i1m3m7x i1m8y i1m9Note that if we multiply the whole matrix by a scale factor,we will get the same transformation, and so there are really eight degrees of freedom specifying the map.In general during texture mapping,the mapping from texture coordinates to the new image is a projective transformation.There is a special simple subfamily of projective maps that come from matrices with a bottom row of001called affine maps.x i2 y i2 1m1m2m3m4m5m6001x i1y i11Affine maps can translate,scale,skew,and rotate2d images.A full projective map can do fancier things,like make parallel lines converge to a point.This is necessary to get effects like foreshortening.Affine maps can be computed a bit quicker than projective maps,because no division is required.Basics:Pin hole camerasIn graphics we take the three dimensional world,and project it down to a two dimensional image using a pin-hole camera model.This is described using matrix operations.Computer graphics practitioners like to have an image space z value after projection,in order to compute the visible surface,so they tend to use four by four matrices.During rendering,somepoint in space x g y g z g,perhaps some polygon vertex is mapped to“image coordinates”x i y i z i using the four by four matrix C.x i w iy i w i z i w i w i Cx gy gz g1The four by four matrix C V F E can be considered a composition of three matrices.E is a Euclidian transform expressing a rotation and a translationE r11r12r13t1 r21r22r23t1 r31r32r33t1 0001F expresses the projection using a camera frustum projection.And V,a viewport transformation maps us to pixel coordinates.In order to extract the actual image coordinates,one needs to divide out the w i coordinate.Computer vision practitioners tend to describe real cameras,that don’t keep around any z val-ues,and as such they tend to describe the imaging process asx i w iy i w i w i Px gy gz g1P A N E is a3by4matrix,composed of the following parts.The position and orientation of the camera is determined by E.The canonical camera projection is described byN 1000 0100 0001And the intrinsic camera parameters are defined by the three by three upper diagonal matrix matrixA f s xθt x 0f s y t y 001Where f is the focal length,s x s y are horizontal and vertical pixel scales,t x t y positions the focal center,andθallows for skewed,non-rectangular pixels.Texture MappingIn this section we briefly visit the topic of texture mapping and describe how it is an application of a projective mapping operation.Texture mapping is a process where we place some interesting looking pattern onto a polygon in space.This pattern may be computer generated,hand drawn,or be a photograph.The pattern is represented as an image made up of pixels called texels.During the rendering process,these texels are conceptually mapped onto the polygon and then viewed through the pin hole camera and viewed on the screen.This can be expressed in matrices as[1]x i w iy i w i w i P3x4G4x3T3x3x ty t1The3by3T matrix scales and translates and rotates the texture to put it on the two dimensional polygon;this is an affine map.The4by3matrix G takes the two dimensional polygon and puts it into3d space.This is also an affine map.The3by4matrix P is the pin hole projection that maps the3d geometry onto the image plane.The concatenation of these three matrices is a three by three matrix M P G T.This matrix represents the transformation from texture space to image space. Thus texture mapping is a projective map from texture space to image space using a3by3matrix M.In a real graphics language,such as openGL,one does not specify M directly.P is specified by the camera commands.M is specified by the placement of the polygon in the world.T is specified by specifying texture coordinates x t y t at the three vertices of a triangle.The above mapping describes the map from texture to image.Typically,in a texture mapping implementation,we in fact use the inverse map,that maps from image domain to texture domain. For each image pixel location x i y i,we compute the mapped texture location,and then the proper color value is interpolated.Texture mapping can be implemented quite efficiently.During the rendering,as the renderer moves across a scan line,the matrix multiply can be evaluated incrementally with a few add oper-ations,and the division by the third coordinate.Two views of a planar objectIn this section we show how two views of a planar object,such as the facade of a building,are related by a projective map.If I take a picture of a facade,and wish to view it from a different position in space,I can create the new view by applying a projective map to thefirst image.Suppose one takes a picture of a planar object,and without loss of generality lets assume that the plane is defined by z g0,then the image of the planar points in camera1arex i1w i1y i1w i1 w i1P1x gy g1By dropping the third column of P1,we can obtain a three by three matrix M1that maps points on the planar object to the image plane.x i1w i1y i1w i1 w i1M1x gy g1Suppose the same plane is viewed from a second camera in a different location,apply the same reasoning to obtainx i2w i2y i2w i2 w i2M2x gy g1Assuming the M matrices are invertible,we obtainx i1w i1y i1w i1 w i1M1M12x i2w i2y i2w i2w i2Dividing both sides by w i2gives usx i1ry i1r r M1M12x i2y i21Hence we see that one can map the view of a planar object from one camera to another camera,simply by multiplying thefirst image coordinates by a three by three matrix,and dividing throughby r.Returning to our simple example,If I take a picture of a facade,and wish to view it from adifferent position in space,I can create the new view by applying a projective map to thefirst image.This can be thought of as essentially a texture mapping operation.One models the3dgeometry of the planar facade,one specifies that the texture should be mapped onto that plane,andthen one views it from a some virtual camera.But the situation is slightly trickier.In simple texturemapping M P G T,the T matrix representing the mapping of the texture onto the geometry,was assumed to be an affine map.This mapping was specified by specifying the x t y t texture coordinates at3vertices of a triangle.This affine does not allow for perspective effects in theoriginal texture.In our case here,the texture itself is a photograph with pin hole perspective.Wereally need the ability to specify a general projective transform for T.OpenGL allows us to do thisin two different ways.Thefirst way is to directly manipulate the so called texture matrix stack.The other way is to pass three texture coordinates x t q y t q q per triangle vertex instead of two. This q essentially accounts for the divide of the mapping M2,and so it should be set to w i2Environment Mapping and PanoramasAnother quite similar technique to texture mapping is environment mapping.The basic idea of en-vironment mapping is that as I look out into my environment from some single center of projection point,I can measure what is seen in each direction,and represent that in some appropriate data structure.This data structure organizes the data sampled at the various directions.Examples for the organizing data structures include cubes,spheres,and cylinders.Graphics systems have used environments maps as ways of simply representing the distant part of the world,either for direct viewing,or for approximate reflection computation.More recently, people have been interested in capturing environment maps photographically,and then using this data to place a user in a virtual version of a real world environment.The types of projections that let us do texture mapping,also allow us to do environment map-ping.If I take a camera and move it back and forth,parallax occurs and certain objects can occlude one another.But when I take a camera and rotate it around its center of projection,this cannot occur.All I can see in these cameras is what can be seen by the pencil of rays that meet at the center of projection.This suggests that there should be a way to map the pixels in one view to pixels in the other.This is in fact true,and can be achieved by a projective transformation.We can see this with an argument similar to the one used for planar objectsSuppose one has a single camera,and without loss of generality,lets us place it at the origin, then the image of the scene points in camera1arex i1w i1y i1w i1w i1A1N R1001x gy gz g1A1R1x gy gz gwhere R1is the3by3matrix describing the rotation of thefirst camera.Hence the mapping from world coordinates to camera1coordinates is described by a simple3by3matrix A1R1 If we have a second camera with the same center of projection,namely the origin,then the mapping from world to camera2coordinates is also described by a3by3matrix A2R2.As a result,one can map from image2coordinates to image1coordinates by multiplying by the3by3 matrix A1R1R12A12,and dividing by the homogeneous coordinate.Hence we conclude two views of a scene from a single center of projection are related by a projective transformation.Image MosaicsImage mosaics and panoramic images are based on the idea that all images that share a center of projection can be easily warped to each other’s coordinate systems.A user takes multiple pictures from a common point.These images are then warped into a common coorditate system.This represents a widefield of view,or perhaps a full surround view from this point.New planar images can be construced from this representation.Here we sketch the basic steps in creating a planar image mosaic.Step1:take multiple pictures from a single center of projection.If one wants to be very careful about this,one should use a tripod.As one spins the camera,one should check to see that there is no parallax(occlusion changes)occurring.If there is parallax,one must adjust the camera on the tripod so that it rotates about a different point.Step2:Pick a single image to the be the reference view.All of the other images will be mapped to this view.Step3:For each other image,I,determine the projective transform that maps I to the reference view coordinate system.This transform will make the overlapping parts of the two images align perfectly.There are a few ways this transform could be computed.If the user marks four corre-sponding points in the reference image,and in I,then one can directly solve for the appropriate matrix.Another method is to use numerical optimization tofind the correct matrix[9].Essen-tially one solves for the matrix such that after warping,the region of the I,that overlaps with the reference image is as similar as possible.If an image does not contain significant overlapping data with the reference view,then we cannot directly compute the required projective transform.But this is not a problem.As long as Ihas overlap with an image J,and we know the correct transformation from J to the reference view, then we can compute the transform from I’s coordinate system to J’s,and compose that with the transform from J to the reference view.step4:Warp all images to the reference view’s coordinate system.Resampling will be neces-sary since the warped sample locations will not generally lie on integer values.PanoramasThere are a few limitations to the image mosaicing approach outlined above.First it artificially picks a single view to be the reference coordinate system.Secondly a single image coordinate system can at best represent a180degreefield of view.Thirdly,in a typical example,the images are gathered by rotating a camera around a single axis,and so there is really only one unknow degree of freedom specifying the relationship between any two images,not eight.To solve for the limitedfield of view,global data structures,such as a cylinder,sphere,or cube can be used to represent the view of the world from a single point.If one knows the intrinsic and extrinsic camera parameters for each image(ie.the P matrix),then one knows the mapping between pixels in the image,and rays in the world,and can easily map them onto the global data structure.If one does not know this information,then one must solve this information from the given views.Details of how to do this are described for example in[5]and[8].In these algorithms, assumptions are made on the real degrees of freedom,such as that the camera does not change its intrinsic parameters as the pictures are taken,and that it may only be rotating about the y axis. Image reuse for fast renderingImage warping techniques are used not only to capture“real”world environments,but also to speed up the rendering of synthetic scenes.During many rendering sequences,subsequent frames can be quite similar.As a user translates,distant objects don’t appear to move at all.As a user rotates,distant objects move simply across the screen with almost no parallax effects.A number of rendering algorithms have been developed that take advantage of this coherence by reusing image regions over a series of frames.When a frame is generated,some parts of the scene are rendered simply by warping the associated pixels from the previous frame.Parts of the image where the visual changes are dramatic are rerendered directly from the geometry.There are a variety of issues in determining which scene elements to image warp,and which ones to redraw from geometry.There are also issues determining what is the best warp to use for the reused regions.Shade et al.[6]describe a system for accelerating walkthroughs of complex environments.In their system,the objects in the environment are organized into a spatial BSP tree.Each BSP tree leaf node contains an object,and the internal nodes hierarchically contain collections of objects. When thefirst frame of a sequence is rendered all of the objects are rendered from their geometry. Each of these images is stored as an“image cache”.They also store image caches for the hierarchy of object collections corresponding to the internal BSP tree nodes.When the user moves,the BSP tree of the scene is traversed to rerender the scene.For each node,an error criteria is checked to see if the corresponding image cache can be reused.To reuse the image cache,it is used as a texture that is mapped on to a planar billboard in space.Their error metric approximates the discrepancy between the correct rerendered object and the appearance ofthe texture mapped billboard.If the error of using the image cache is too high,then the object is rendered by recursively calling the algorithm on the BSP children of this node.When a BSP leafs fail the error measure,then the appropriate geometry is rerendered.In effect what happens is that nearby objects get rerenderd from the geometry almost at every frame.As we consider objects further away from the viewer,they get rerendered less often,and their billboards are just rendered as texture maps.At very far distances,large clusters of objects are rerendered as a single textured billboard.The Talisman rendering architecture[10]is a hardware rendering architecture based on the concept of image reuse.The hardware has a standard polygon renderer that renders each object into its own sprite.Another component of the hardware takes sprites,applies an arbitrary affine warp to it,and composites it onto the image screen.As a geometric objects move about a scene, its sprite can be affinely warped(ex.scaled and translated)to approximate the appearance of the proper motion.If the affine transform is a poor approximation to the proper appearance of the object,its sprite can be updated,by rerendering it in the rendering hardware component.In Talisman,these two hardware components,the renderer and the warper,are decoupled.The warper runs at frame rate,and must warp and composite each sprite per frame.The renderer up-dates sprites when requested,and does so as fast as it can.A controlling program decides which sprites are invalid in each frame must be rendered,typically,only a fraction of the sprites are up-dated in a frame.The controlling program also decides what is the best affine warp to use,and and passes that information onto the warping engine.Lengyel and Snyder[3]describe the systematic way in which such a program can decide on the best affine warp.They also perform a compari-son between using affine warps,and the full power of a projective transform.Their experiments find that the full projective transform does not add a significant advantage over affine transforms. Affine transforms are much cheaper to implement,since they require no divide operations. Warping with depthIn subsequent sections of this course,we will see uses of warping that use depth andflow type in-formation.These warps are more general than the projective transforms discussed so far.Projective warps can in effect only simulate the motion of planar geometry.This is a very poor approximation for nearby non planar objects.In this section we bridge the gap by briefly discussing these types of warps.In a real scene as a user’s view translates,different objects appear to move at different speeds. Closeby objects move quickly,while distant objects appear not to move at all.To achieve this type of effect by image warping,one must use a more complicated warp that has can achieve the proper opticalflow.If one knows the intrinsic parameters of some virtual or real camera,and one knows the actual depth(z)values at each pixel,then warping to some new known view is straightforward. If the four by four matrix C1describes thefirst camera,and C2,the desired view,we havex i1w i1y i1w i1 z i1w i1 w i1C1x gy gz g1x i2w i2y i2w i2 z i2w i2 w i2C2x gy gz g1and sox i2ry i2r z i2r r C2C11x i1y i1z i11Like texture mapping,much of the computation of this warp can be done incrementally.As one moves across across a scanline of image1and warps the pixels,x i1is simply incremented and y i1remains the constant,and so most of the multiplies can be avoided.In contrast,z i1can change arbitrarily and so these multiplies must be performed per pixel.And like texture mapping,one must perform the divide per pixel.Computationally,there a big difference between these warps and projective warps.Projective warps are easily invertible,simply by using the inverse of the3by3matrix.Inverse warping is very useful.It allows us to warp each output pixel location to some position in the input image. One can then interpolate the color at that position.In the depth warp describe here,z i1can change arbitrarily.As a result,it is difficult to compute an inverse map that goes from i2coordinates to i1coordinates.This means that we can’t simply interpolate image1pixel colors for each output pixel as done in texture mapping.There have been a variety of approaches taken to this problem include forward splatting type methods[7],micropolygon scan conversion methods[4],and more complicated inverse warping methods[2].Wrap UpImage based rendering is concerned with the use of using input images to create output images. The images are typically reused using some type of warp.Many image based method use simple warps that do not need any depth forflow per pixel information.We have focused on one such family of warps,the projective warp.Projective warps are simple warps computed by multiplying with a3by3matrix and dividing by a third coordinate.Projective warps are fundamental to computer graphics,and are at the root of texture mapping and environment mapping.More recently,These techniques have been used to reuse photographs of real world scenery.In particular,simple warps are used to create and render from image mosaics and panoramas.Projective warps have been used recently in new rendering systems that try to reuse much of the imagery from frame to frame.These systems attempt to approximate the view of some of the objects by warping those pixels to the new view.Where it is determined that the warping approximation is too inaccurate,traditional rendering is used.These kinds of systems are important methods for enabling the interactive rendering of complicated environments.Given a z value stored at each pixel,one can compute where that point would be viewed in a second image.This can be used to drive a more complicated kind of image warping.Peopleare currently understanding how these kinds of warps can be used to allow us to virtually interact with real world captured images.These kinds of warps are also being investigated for image reuse rendering systems.References[1]P.Heckbert.Fundamentals of texture mapping and image warping.Master’s thesis,TheUniversity of California at Berkeley,June1989.[2]veau and O.Faugeras.3-D scene representation as a collection of images and funda-mental matrices.Technical Report2205,INRIA-Sophia Antipolis,February1994.[3]Jed Lengyel and John Snyder.Rendering with coherent layers.SIGGRAPH1997,pages233–242.[4]William R Mark,Leonard McMillan,and Gary Bishop.Post-rendering3d warping.Proceed-ings1997Symposium on Interactive3D Graphics,pages7–16.[5]Leonard Mcmillan and Gary Bishop.Plenoptic modeling:an image based rendering system.SIGGRAPH1996,pages39–46.[6]Johnathan Shade,Dani Lischinski,David Salesin,Toney DeRose,and John Snyder.hiear-archical image caching for accelerated waslkthroughs of complex evironments.SIGGRAPH 1996,pages75–82.[7]Jonathan Shade,Steven Gortler,Li wi He,and Richard yered depth images.SIGGRAPH1998.[8]Richard Szeliski and H.Shum.Creating full view panoramic image mosaics and environmentmaps.SIGGRAPH1997,pages251–258.[9]Rick Szeliski.Video mosaics for virtual environments.IEEE CG&A,pages22–30,march1996.[10]Jay torborg and James Kajiya.Talisman:Commodity realtime3d graphics for the pc.SIG-GRAPH1996,pages353–364.。