Using resolution pyramids for watershed image segmentation

Using resolution pyramids for watershed image segmentationMaria Frucci *,Giuliana Ramella,Gabriella Sanniti di BajaInstitute of Cybernetics ‘‘E.Caianiello’’,Italian National Research Council (CNR),Via Campi Flegrei 34,80078Pozzuoli (Naples),ItalyReceived 25November 2005;received in revised form 26June 2006;accepted 12July 2006AbstractIn this paper we build a shape preserving resolution pyramid and use it in the framework of image segmentation via watershed trans-formation.Our method is based on the assumption that the most significant image components perceived at high resolution will also be perceived at lower resolution.Thus,we detect the seeds for the watershed transformation at a low resolution,and use them to distinguish significant and non-significant seeds at any selected higher resolution.In this way,the watershed partition obtained at the selected pyr-amid level will include only the most significant components,and over-segmentation will be considerably reduced.Segmentations of the image at different scales will be available.Moreover,since the seeds can be detected at different pyramid levels,alternative segmentations of the image at a given resolution can be obtained,each characterized by a different level of detail.Ó2006Elsevier B.V.All rights reserved.Keywords:Segmentation;Watershed transformation;Resolution pyramid1.IntroductionA frequently adopted and powerful grey-level image segmentation technique is based on the watershed trans-formation [1–3].This method works well for images with sufficiently high contrast,so that edges can be detected in correspondence with the perceived contours of the objects of interest.As it is well known,the watershed transformation includes two main steps.The first step is devoted to the identification of suitable seeds.These are,generally,detected as the pixels having minimal val-ue in the gradient image computed for the grey-level image.The gradient image is used to enhance the edges and,so,to identify the local minima in a more correct way.The second step is aimed at region growing,start-ing from the found seeds.In this way,a partition of the input grey-level image is obtained,where pixels assigned to the same region are characterized by a com-mon property, e.g.,similar grey-levels.Once thewatershed partition has been obtained,the regions of the partition have to be assigned to either the foreground or the background,in order to identify the objects of interest.This process can be accomplished by taking into account the difference in grey-level between adjacent regions,or other morphological/geometrical properties expected to characterize the foreground components.A 2D grey-level image can be conveniently interpreted as a 3D landscape,where for every pixel in position (x ,y ),its grey-level plays the role of the z -coordinate in the landscape.By using this interpretation,the paradigm on which watershed based segmentation is founded can be easily illustrated.High grey-levels correspond in the landscape to mountains and hills,while low grey-levels cor-respond to valleys.If the bottom of each valley is pierced and the landscape is immersed in water,then valleys are flooded.The water will first fill the deepest valleys,and then all remaining less deep valleys.Any valley will start to be flooded when the water level reaches its pierced bot-tom.A dam (watershed)is built wherever water originating from a basin is going to spread into the close basins.When the landscape has been completely flooded,the top border of the dams (watershed lines)is interpreted as the set of0262-8856/$-see front matter Ó2006Elsevier B.V.All rights reserved.doi:10.1016/j.imavis.2006.07.014*Corresponding author.Tel.:+390818675167;fax:+390818675326.E-mail addresses:m.frucci@r.it (M.Frucci),g.ramella@r.it (G.Ramella),g.sannitidibaja@r.it (G.Sanniti di Baja)./locate/imavisImage and Vision Computing 25(2007)1021–1031lines separating the parts(basins)into which the image is partitioned.A drawback limiting the effectiveness of watershed based image segmentation is the excessive fragmentation into components,which are not all equally meaningful. In fact,the number of detected seeds generally exceeds the number of regions perceived as meaningful.Thus,seed reduction and region merging have to be applied,so as to obtain a simpler and more significant partition.Purpose of this work is to introduce an alternative way to reduce over-segmentation,based on the use of a multi-resolution image representation.When a grey-level image is observed at different scales,the most significant regions will be perceived at all resolutions,while regions with lower significance,e.g.,fine details,will be perceived only at suf-ficiently high resolution.Thus,if the seeds for watershed segmentation are detected at low resolution and are,then, used as markers to divide into significant and non-signifi-cant the seeds found at higher resolution,the resulting par-tition is expected to consist,at any resolution,mainly of the regions that are perceived as the most significant ones.In this way,over-segmentation is considerably reduced.Thefirst step of our process is devoted to the construc-tion of a pyramid to represent the input grey-level image at different resolutions.We follow the approach introduced in [4]for binary images and extended in[5]to grey-level imag-es,where a3·3mask of weights was used to compute the grey-level of any parent pixel,based on the grey-levels of its children.Here,we use a modified3·3mask of weights, which provides more faithful representations of the origi-nal image at all resolutions.The second step deals with segmentation.First,the gra-dient image is computed at all pyramid levels.Then,the regional minima,i.e.,the seeds for watershed transforma-tion,are detected at all resolutions and watershed transfor-mation is applied along the pyramid to obtain basic partitions.To reduce the number of seeds at higher resolu-tion and,hence,limit over-segmentation,we project the seeds detected in a given level of the pyramid,characterized by low resolution,onto a selected higher resolution levels of the pyramid.At the selected level,the projected seeds are interpreted as markers and are used to distinguish, among the seeds actually detected at that level,the signifi-cant and non-significant seeds.During the third step,only significant seeds are used to compute once again the watershed transform and a simple region merging is performed.Different segmentations are possible at the selected res-olution level,depending on the pyramid level used to iden-tify the seeds to be projected and,hence,the desired detail of information to be preserved.Thus,an advantage of our method is the possibility to obtain from coarse tofine seg-mentations of the selected image,by using different lower resolution pyramid levels to detect the seeds to be project-ed.Both the level of the pyramid at which segmentation is computed,and the pyramid level at which to detect the seeds to be projected depend on the desired accuracy,as well as on the size of the objects that the user wants to extract.When resolution decreases,unavoidably some blurring is created and the contrast between adjacent regions becomes too small to extract significant edges. For images including only a few large objects of interest, the seeds to be projected can be selected even at a quite lower resolution with respect to the resolution of the level selected to achieve segmentation.When the size of the objects of interest is small,the resolution at which to detect the seeds to be projected cannot be too much smaller than the resolution at which segmentation is accomplished.The computational cost of the whole process in O(N2) for an N·N image.The algorithm runs in a few seconds on standard personal computers.A few sample images are given in this paper to describe the process and show its performance.Our method works satisfactorily for a wide repertory of images,provided that these are well contrasted.A preliminary version of this work has been presented in[6].The paper is organized as follows.Section 2describes the pyramid construction method;Section3 introduces alternative criteria to identify the significant seeds for watershed segmentation at the selected pyramid level;finally,some concluding remarks are given in Section4.2.The grey-level pyramidIn this work,we consider grey-level images where grey-levels are in the range[0,255],and follow the standard con-vention according to which black corresponds to the lowest grey-level0and white to the highest grey-level255.Let G1be a grey-level image with size2n·2n.For a dif-ferently sized image,a suitable number of rows/columns are added to obtain the desired size2n·2n.Pixels belong-ing to additional rows/columns are assigned a grey-level value definitely characterizing background components. This value is the maximum or the minimum grey-level pres-ent in the original image,depending on whether the fore-ground is perceived as including regions that are locally darker or locally lighter.The28·28image G1,shown in Fig.1,is used as running example.This image(provided by courtesy of Drs.Daniel L.Roe and Douglas L.Rosene) is taken from area TE in the Rhesus monkey and shows mostly layer II and layer III.G1is interpreted as a3D land-scape,where the grey-level g of the pixel in position(x,y),is used as the third coordinate in the landscape.Pyramids are well known multi-resolution representa-tion systems;they provide from coarse tofine representa-tions of a discrete image[7],and have been employed for a number of applications,such as line-drawing analysis or object contour extraction[8–13]and segmentation [14–18].The general regular pyramid construction strategy is based on the use of a uniform subdivision rule that associ-ates tofixed size regions in the image at a given resolution, single pixels in the image at immediately lower resolution. The so obtained representations tend to smooth out small1022M.Frucci et al./Image and Vision Computing25(2007)1021–1031variations within regions when resolution decreases,result-ing in the unavoidable loss of some information.Variations of this scheme are obtained by changing the criteria to link to each other pixels belonging to images at different resolu-tion,or the sampling grid used to compute lower resolution images.Resolution reduction can be obtained in different ways.The Gaussian pyramid is a sequence of images,where each image is a low-pass filtered copy of its predeces-sor.The Laplacian pyramid is obtained by using two basic operations.The first operation applies a low-pass filter and sub-samples by a factor of 2in each dimension;the second operation up-samples by a factor of 2and then applies the same low-pass filter.Also the wavelet transform can be used for multi-resolution image representation.Regular pyramids have been used in the framework of image seg-mentation (e.g.,see [19,20]).Both the horizontal relation-ships,i.e.,those among elements in the same pyramid level,and the vertical relationships,i.e.,those among ele-ments in different pyramid levels,are taken into account.Segmentation is obtained by using an iterative approach involving a bottom-up linking process,a bottom-up updat-ing of elements values and a top-down reassignment of ele-ments values.Pyramids with irregular tessellation can also be computed,where the decimation processes dynamically adapt to the image layout,and the obtained data structure is more flexible (see,e.g.,[21]).The general scheme is to eliminate some pixels to pass from one level to the next one.The pixels that are eliminated link to the surviving pixels,whose grey-levels are computed as the average of the eliminated pixels linking to them.Irregular pyramids have some advantages with respect to regular pyramids.For example,they better preserve elongated regions.However,irregular pyramids involve a random process and may result not satisfactory for image segmentation.In fact,the same segmentation is not necessarily obtained,if the input image is analysed again.A deep analysis of various schemes to compute irregular pyramids and a comparison of segmentation methods based on the use of irregular pyramids can be found in [22].In this paper,we build a regular pyramid by using 3·3operators with integer weights.In general,regular pyra-mids suffer for lack of invariance under translation,do not preserve shape (especially in the presence of elongated objects,which disappear or are fragmented at low resolu-tion),and do not preserve topology.Our regular pyramid,however,is almost shift invariant and shape preserving.In fact,to compute the grey-level of the parent pixels we take into account all the values that parent pixels would have when shifting the partition grid in all the possible ways.As regards topology preservation,a general definition suit-ed to the grey-level case is hard to be found since this depends on the specific application and on what the expert in the field regards as constituting the foreground in the input image.In [4],a method for binary images was proposed to gen-erate a pyramid,which is shift invariant and topology pre-serving,within the limits of decreasing resolution.The method was extended to the case of grey-level images [5]and used in the framework of image segmentation [6].In [4,5],to reduce translation dependency,we combined the different results originated by the decimation process,when the sampling grid is differently placed on the input image.Actually,decimation was performed only once.The combi-nation of the results was simulated,by using a suitable neighbourhood and proper weights to compute lower reso-lution levels.Here,we present an improved version of the discrete method [5],aiming at a more faithful computation of grey-levels for the parent pixels.The sampling grid is superimposed onto the input image G 1with size 2n ·2n to perform decimation.Each cell of the grid includes a block of 2·2children on G 1and identifies a single parent in the immediately lower res-olution image G 2with size 2n À1·2n À1.The grid is,then,superimposed onto G 2to build G 3and so on.At each step,resolution decreases by four.Thus,after n steps the decimation process terminates and the obtained image G n +1includes one single pixel.Actually,resolution levels including less than 32·32pixels are not considered in this paper,as they would generally give too coarse repre-sentations of G 1.Thus,for our running example,four levels constitute the pyramid:the base of the pyramid,level 1,is the image G 1at full resolution (256·256),the next level of the pyramid,level 2,represents the image at a uniformly lower resolution (128·128),level 3is the successive lower resolution image (64·64)and the apex of the pyramid is the 32·32image,which constitutes level 4.At each step of the decimation process,we inspect in forward raster fashion only pixels belonging to even rows and columns of the current image G k ,meaning that we use the bottom right child pixel in a block to find the coor-dinates of the parent pixel in G k +1.For each inspected pixel in position (i ,j )of G k ,the parent pixel in G k +1will be in position (i /2,j /2).The grey-level of the parent pixel in posi-tion (i /2,j /2)of G k +1is computed by taking into account the grey-levels of the pixel in position (i ,j ),as well asofFig.1.The 256·256grey-level image used as running example.M.Frucci et al./Image and Vision Computing 25(2007)1021–10311023its eight neighbours,in G k .The parent–child relations are,then,very easy to keep.The sampling grid could be shifted on G k in four differ-ent ways and,hence,any pixel in the 3·3window centred on (i ,j )could be the bottom right pixel of the block.Thus,we introduce a multiplicative mask of weights to evaluate the contribution given to the grey-level of the parent pixel in position (i /2,j /2)of G k +1by the nine pixels in the 3·3window centred on (i ,j )in G k .The grey-levels of the pixels in the window are multiplied by the relative weights;the average value of these values is then linearly rescaled to be in the range [0,255]and constitutes the grey-level of the parent pixel.In order the grey-level of the parent pixel be computed almost independently of the position of the sampling grid,we have two slightly different alternatives.We note that,depending on the position of the sampling grid over G k ,a pixel p in the image G k belongs to one out of four possi-ble cells of the grid,including three of the eight neighbours of p.In turn,each neighbour of p belongs to a different number of cells,depending on its position with respect to p.Each horizontal/vertical neighbour of p (edge-neigh-bour)belongs to two different cells including p,while each diagonal neighbour of p (vertex-neighbour)belongs to exactly one such a cell.See Fig.2.The above observation suggests the use of the 3·3mask of weights,shown in Fig.3left,with central value 4,edge-neighbours valued 2and vertex-neighbours valued 1.Alternatively,we can also consider the nine 3·3win-dows centred on (i ,j )and on each of its eight neighbours in the image G k .Pixel in position (i ,j )is included in all nine windows,its edge-neighbours are included in six windows and its vertex-neighbours in four windows.In this case,the number of windows including a pixel can be used as the corresponding weight for the multiplicative mask.See Fig.3right.We have used both masks and have experimentally found that the mask in Fig.3right provides more accurate results.This mask is the one used in the examples shown in this paper.We point out that,similarly to a Gaussian filter,our mask produces a weighted average of each pixel’s neigh-borhood,providing gentler smoothing and preserving edg-es better than it would be possible by using a mean filter with the same size.Differently from the Gaussian filter,based on a 5·5kernel,our mask has size 3·3.Once the computation of G k +1is done,the successive lower resolution level is built by the same procedure.The pyramid built for the running example is shown in Fig.4.3.Watershed segmentationThe gradient image $k of G k is computed at all pyramid levels.$k is used in place of G k to enhance the differences in grey-level and,accordingly,favour a more reliable identifi-cation of the seeds starting from which watershed transfor-mation is accomplished.The mountains in the landscape representation of $k ,i.e.,the locally higher grey-levels,cor-respond to the edges,where a transition between adjacent regions characterized by different grey-levels occurs.In turn,the valleys,i.e.,the regional minima,correspond to the kernels of regions with almost uniform grey-level distri-bution,and constitute the seeds for the region growing pro-cess.The growing process ascribes to each region the elements located along the slopes in the landscape that are closer to the relative seed than to any other seed,both in terms of spatial distance and of the difference in grey-level.A dam is built in correspondence of the elements of the landscape that could be ascribed to more than one region.The top border of the dams in the 3D landscape is interpreted in the 2D grey-level image as the set of lines separating the regions into which the image is partitioned.The basic partitions of the four pyramid levels for the running example can be seen in Fig.5,where the watershed lines are superimposed on each image in the pyramid and the computed relative number of regions is indicated.It is straightforward to note the excessive fragmentation of G k ,k 2{1,2,3,4},which is due to the large number of regional minima found in $k .This drawback is typical of the watershed transformation.Oversegmentation can be reduced by resorting to seed selection or region merging or,better,to both seed selection and region merging.To perform region merging,each region of the partition can be associated a representative grey-level.In our case,the representative grey-level of a region is computed as the average of the grey-levels with the maximal occurrence in the region itself.If only region merging is used,the obtained result,though certainly less fragmented,are still remarkably affected by oversegmentation.See Fig.6,show-ing the watershed lines found at the four pyramid levels after merging regions whose representative grey-levels slightly differ from each other.It is clear that region merg-ing is not enough and,necessarily,seed selection has to be applied before resorting to region merging.A suitable process,aimed at selecting among the seeds found in $k only those that are significant to correctlypar-Fig.2.The eight neighbours of p,top,and the four 2·2blocks that can be obtained by shifting the sampling grid.1024M.Frucci et al./Image and Vision Computing 25(2007)1021–1031tition G k ,should be accomplished to obtain a significant partition [23].A filtering technique can be used to remove non-significant minima,but a priori knowledge on the class of images is necessary to identify the features to be used to design the filter.In some applications,the seeds are defined manually.A fully automatic seed selection process is possi-ble,see e.g.,[24]but such a process,though certainly effec-tive,is computationally expensive.It involves criteriato256×256128×12864×6432×32Fig.4.The four levels of the pyramid computed for the runningexample.(4127) (1083) (383)(115)Fig.5.The watershed lines found at the four pyramid levels starting from the relative regional minima are superimposed on the images at all pyramidlevels.(1236) (499) (260) (93)Fig.6.The watershed lines found at the four pyramid levels after region merging.M.Frucci et al./Image and Vision Computing 25(2007)1021–10311025evaluate the significance of each partition region with respect to the adjacent regions,as well as suitable opera-tions to remove the seeds corresponding to regions found to be non-significant so that,by repeating the growing pro-cess starting from the new,reduced,set of seeds,a better watershed partition is obtained.Seed reduction and watershed transformation are iterated until a stable result is obtained,i.e.,until all partition regions are found to be significant.We propose an alternative criterion to identify,at any selected segmentation level of the pyramid G k,16k<4, only seeds corresponding to the significant regions.The underlying idea is that since G k is well represented even at low resolution G s,k<s64,the seeds identified at G s do correspond to significant regions of the image at G k. Thus,for the running example,the seeds detected in$s, 1<s64could be used as markers to select the seeds at any level G k,k>s,and watershed transformation would not need to be iterated until a stable partition is achieved.We project the seeds found at G s,1<s64,onto the selected segmentation level G k.This is easy to do since the coordinates of parent pixels are immediately derivable from those of their children.Thus,for each pixel at level G s we identify its descendants at G k.Since any pixel has four children,and each of these children has in turn four children and so on,for each seed pixel found in$s we iden-tify a2·2block of descendants in$s+1,a4·4block of descendants in$s+2and so on.In the following,we call segmentation level the level G k at which segmentation is performed,seed level the level G s at which the seeds to be projected onto G k are selected,and projected seeds the sets of pixels in the image at level G k, which are covered by the projection of the seeds detected at level G s.See Fig.7.We refer to the watershed transform computed by using all detected regional minima as to basic watershed transform.A simple method,convenient from a computational point of view but non-effective,is to use directly the pro-jected seeds to compute the watershed transform at the seg-mentation level G k.Unfortunately,the projected seeds do not include all and only the regional minima that could be regarded as significant at G k.Thus,the obtained watershed partition does not provide satisfactory results. In particular,the watershed lines do not closely follow the perceived borders separating the foreground from the background.In fact,though our pyramid construction method is almost translation invariant and the selected mask of weights allows us to compute the grey-levels of parent pixels in a reliable manner,a perfect correspondence between the projected seeds and the significant regional minima that could be actually found at level G k is seldom obtained.A projected seed may not include any original regional minimum,or that it includes distinct regional min-ima.See Fig.8left,where for the running example the seeds projected at level256·256from level64·64are black,the original seeds at level256·256non overlapping the projected seeds are grey,and the original seeds at level 256·256overlapping the projected seeds are white.In Fig.8right,the top-left part of the image is magnified for easier visualization purpose.When a projected seed does not include any original minimum,the partition would be originated starting from a not properly selected seed,i.e.,by pixels that are not regional minima in the gra-dient image at hand.As a result,the watershed lines could be misplaced with respect to the expected positions.When a projected seed includes distinct original seeds,excessive merging could be caused.To solve the above problem,we propose a method, which requires the computation of the watershed trans-form two times.First we compute the basic watershed transform at the segmentation level G k by using all the seeds detected at that level as regional minima.The basic watershed transform is used during the process done to remove non-significant seeds,after seed projection.Then, we compute again the watershed transform,starting from the reduced set of significant seeds,i.e.,those surviving the removal process.Once the watershed partition has been computed,adjacent regions whose representative grey-levels slightly differ from each other are merged. For all the examples shown in this paper,region merging is done for adjacent regions such that the difference in grey-level is at most10.Fig.7.From left to right:seeds projected from levels32·32,64·64,and128·128,onto level256·256. 1026M.Frucci et al./Image and Vision Computing25(2007)1021–1031We suggest two alternative criteria to detect the signifi-cant seeds.Criterion IWe regard as significant a seed,originally detected in $k ,only provided that the partition region associated to that seed in the basic watershed transform of G k includes at least one pixel belonging to a projected seed.In this way,the pixels that will be used as seeds for the watershed transformation at level G k are actually minima at that level.Moreover,exces-sive merging is avoided,since seeds covered by the same pro-jected seed will be used as individual seeds.In Fig.9,the seeds found at level G 1and those surviving at level G 1,when the seed level is G 3and Criterion I has been used for the removal process,are shown.In Fig.10,the watershed partition obtained at level G 1when the seed levels are,respectively,G 2,G 3and G 4,Cri-terion I has been used for the removal process,and region merging has been accomplished are shown.For each parti-tion of the image G 1,the relative number of regions is also indicated.The obtained partitions include a number of regions remarkably smaller than those in the basic watershed transforms.We note that,due to their large size,the projected seeds overlap at level 256·256a large number of regions of the basic watershed partition and some of them are not significant.Thus,we suggest a second seed removal criterion better performing.Criterion IITo reduce the number of seeds detected as significant at the segmentation level G k ,we reduce the overlapping among the seeds projected from the seed level G s and the regions of the basic watershed transform of G k .To this aim,we identify within each projected seed in the gradient image $k ,the pixels whose grey-levels are minima within the projected seed itself.All non-minima pixels are removed from the projected seed.In this way,the size of the projected seed is shrunk,limiting the risk of excessive overlapping among projected seeds and regions of the basic watershed transform at G k .The shrunk projected seeds obtained at the segmentation level 256·256for the running example,when the seed level is 64·64and Criterion II is used,are shown in Fig.11.The shrunk projected seeds at level 256·256are shown in Fig.11left,while the seeds identified as significant among those detected at level 256·256,are shown in Fig.11right.Fig.8.Left,projected seeds are black,original seeds non overlapping the projected seeds are grey,and original seeds overlapping the projected seeds are white.Right,magnified top-left portion of theimage.Fig.9.Seeds found at level 256·256,left,and significant seeds remaining after seed removal based on Criterion I.M.Frucci et al./Image and Vision Computing 25(2007)1021–10311027。