1 Introduction

Shaping with PatternsJiayuan Meng Department of Computer Science University of Virginiajm6dg@December14,20061IntroductionPeople like magic.It is fun to imagine a cloud is shaped like a Mickey mouse,or the pattern of leaves andflow-ers are associated with human face.Sometimes infiction movie or cartoon production,we like to see similar visual effects.In this paper,we define the problem as following: Given two images,one for pattern and one for shape.We output another image,which draws the shape provided in the shape image,but using the patterns defined in the pat-tern image.This is a typical texture synthesize problem with user interaction.In a typical texture synthesize prob-lem,a source pattern is used to produce another texture, which visually resembles the pattern.Some algorithms have also been invented to guide the texture synthesize process by providing another user defined image as a hint to grow the texture.Based on these algorithms,we suc-cessfully solved the problem stated above.2Related WorkTexture synthesize has been widely discussed.Efros and Leung[1]used markov randomfields to grow a new tex-ture from a seed cut from the original texture.Hertzmann et al.has generalized the idea of MRF and the multi-resolution method,the resulting technique is image anal-ogis[2].The program learns the features from a pair of image,A and A’.A’is thefiltered image of A.Given a new image B,the program can produce image B’with the sim-ilarfilter effect.Image analogies is a powerful tool and it can achieve multiple effects including texture synthesize, texture transfer,and texture-by-numbers.Our algorithm is mainly based on image analogies.3Methodology3.1Brief Review of Image AnalogiesImage analogies works briefly as follows:1.Build the gaussian pyramid for the source image pairA and A’,as well as the target unfiltered image B.2.From the coarser level to thefinest level,build the gaussian pyramid of thefiltered image B’.3.At each level,we construct image B’row by row. At each pixel p,we search for a corresponding position s(p)in A’in the same level.Then we compute B’(p) from the feature at s(p)in A’.The computed pixel is an optimization of an energy function which attempts to maintain both the approximation and the coherence.We describe each step in a bit more detail:First,a user has to define a feature for each pixel.The feature is selected by the ually it is simply the RGB color or the luminance.Note that the feature for image A can be different from that of image A’.However, the feature of A and B are usually the same,and so as the features of A’and B’.Next,we need to construct the feature vector.For the source image pair A and A’,we build a gaussian pyramid.A feature vector F(i,j)is associated with a position(i,j), and it is a concatenation of features in the5*5neighbor-hood of the current level and the3*3neighborhood of the coarser level on both the images.So in this case,the feature vector for the source image pair has a length of68.We compute the feature vector for the image pair B and B’in the same way.The only difference is that when B’is constructed row by row,we don’t know all the feature around the current pixels neighborhood.Therefore,we neglect those invalid neighborhood pixels and build a partial feature vector.We use F a to denote the feature 1vector for image pair(A,A’),and F b to denote the feature vector for image pair(B,B’).Now we have defined the feature vectors.When looking for the best position s(p)for the current pixel in B’,wefirst do an ANN search tofind the approximate nearest neighbor in the feature vector space of the image pair(A,A’)at the current level.Care must be taken since F a has a length different with F b.We resolve this by selecting in F a only thefields that also present in F b.We denote this feature vector F appIn addition tofinding the feature in F a which best ap-proximates F b,we also have tofind the F a that exhibits the best coherence.We search in the neighborhood of p in B’.Each resolved neighbor q is associated with a source position s(q)in A’.Suppose p is an offset of r from q(p=q+r),we thenfind the F a(s(q)+r)that best approximates F b(p).we denote this feature vector F coh.The user has to define a coefficient K to select between F app and F coh.Basically,if F app<K∗F coh,then we select F coh,otherwise,F app is selected.We assign s(p)to the the position in A’that corresponds to the selected feature vector.3.2Analysis of Image AnalogiesImage analogies has two functions that are similar to our goal of rendering a shape with apattern.Thefirst is texture transfer,illustrated in Figure1.It uses the same fabric pattern image for A and A’,and set the target image to be B.Thefiltered image is a girl’s face rendered with the fabric texture.However,the pattern is applied globally.Although e can confine thefilter to be applied to certain boundaries,that will result in hard boundary edges that doesn’t”bleed”and merge naturally with the background.Anotherfilter is texture-by-number.In which A will be a segmentation of A’denoted with numbers(or different colors).By providing another segmentation image with the same set of numbers,we can construct a B’which has heterogeneous textures arranged according to the seg-mentation.This is illustrated in Figure2.This turns out to be very similar to our goal.The only difference is that we need to render a foreground with the pattern,and the texture-by-number is rendering a numbered image with the pattern.3.3Extended Texture-by-numberGiven the above observation,we made an attempt to build a numbering of the pattern image from the target image.First,we have to define the numbering of the pattern image.Illuminance is a good candidate.However,in most conditions,we want to distinguish the foreground from the background,so that the target shape can be rendered using only the foreground pattern.Therefore, segmentation information is required(Figure4).The resulting numbering is two fold:a segmentation mask, Figure1:Texture transfer using image analogiesFigure2:Texture-by-number using image analogies2and the illuminance of the foreground.Thus,A is a gray scale image with a matt.Second,we have to convert the target shape in to the numbering defined above.We follow the same procedure.First,we encode the segmentation information of the target image in a mask,then we add the illuminance of the foreground.B is also a gray scale image with a mask.To provide better results,the illuminance of B is nor-malized to the illuminance of A.Histogram match is an-other way which provides similar results.We can then define the feature of A and B to be a vector of 2values [α,l],where αis the matt value,and l is the illuminance.Now we have the input images ready,we startimage analogies to produce B’.This is illustrated in Figure 3.Figure 3:Raw result by numbering texture with segmentation and illuminationFigure 4:matting is important,without matting,the foreground pattern will be confused with the background pattern 3.4Key Point EnhancementAlthough we have preserved shape,there are still someimportant features missing.Some positions in B is more detailed and is the key feature for B,such as the eyes of a face,etc.At these key points,we want B’to resemble B more than anywhere else.We solve this by first finding the key points in B using the SIFT method by Lowe [3].In general,SIFT works by finding the positions where the image has a local extrema in the Laplacian of Gaussian field.If the image has significant contrast at this point,then we regard it as a key point.Key points are found at every level of the Gaussian pyramid.At each level,we decrease the value of K at each key point so that B’can preserve more similarity with B at that point.On the other hand,coherence is enhanced at places where there is no key point and the image exhibits low contrast.The resulting image is shown in Figure 54ConclusionWe have also tried synthesize in B’s gradient field,and then use poisson editing [4]to recover the target shapeFigure 5:Enhanced result by preserving more details at key points.The similarity with B can be further tuned globally byuser.3with a gradient style of the given patterns.However,this resulted in a blurry image.This phenomenon has been identified by Wei and Levoy[5]Image analogies is a powerful tool in texture synthesis. Our application is just an extension of image analogies. Using scale invariant and orientation invariant key points, we may able to accelerate the algorithm by dividing the image to patches according to the key points,and then merge the patches,rather than synthesize the pixels one by one.Another possibility is to introduce the orientation invariant texture synthesis.A key point with a patch can be rotated in any direction.Therefore,it is capable to synthesize a new texture with different orientations. Finally,since SIFT can be used to match objects that appear with different positions,scales,and orientations in different images,it provides another possibility to extend the image analogies to handlefilters that warp the image. References[1]Alexei A.Efros and Thomas K.Leung.Texture syn-thesis by non-parametric sampling.In IEEE Interna-tional Conference on Computer Vision,Sep.1999. [2]Aaron Hertzmann,Charles E.Jacobs,Nuria Oliver,Brian Curless,and David H.Salesin.Image analo-gies.In Proceedings of SIGGRAPH,2001.[3]David G.Lowe.Distinctive image features fromscale-invariant keypoints.In Invernational Journal ofComputer Vision,2004.[4]Patrick Perez,Michael Gangnet,and Andrew Blake.Poisson image editting.In Proceedings of ACM SIG-GRAPH,2000.[5]Li-Yi Wei and Marc Levoy.Fast texture synthesis us-ing tree-structured vector quantization.In Proceed-ings of ACM SIGGRAPH,2003.4。