b Semantic labeling of digital photos by classification a

Semantic labeling of digital photos by classification

G. Ciocca a*, C. Cusano a ?, R. Schettini b ?, C. Brambilla c ?a

TIM-ITC, CNR, Via Ampere 56, 20131 Milano, Italy

b

DISCO, University of Milano Bicocca, Via Bicocca degli Arcimboldi 8, 20126 Milano, Italy

c

IMATI, CNR, Via Ampere 56, 20131 Milano, Italy

ABSTRACT The paper addresses the problem of annotating photographs with broad semantic labels. To cope with the great variety

of photos available on the WEB we have designed a hierarchical classification strategy which first classifies images as pornographic or not-pornographic. Not-pornographic images are then classified as indoor, outdoor, or close-up. On a database of over 9000 images, mostly downloaded from the web, our method achieves an average accuracy of close to 90%.

Keywords: image classification, pictorial features, Classification And Regression Trees, bagging, reject option.

1. INTRODUCTION

We address here the problem of annotating photographs with broad semantic labels in order to facilitate their processing, organization, storage, and retrieval. To cope with the great variety of photos available on the WEB we have designed a hierarchical classification strategy which first classifies images as pornographic or non-pornographic. Non-pornographic images are then classified as indoor, outdoor, or close-up. The classification hierarchy is illustrated in figure 1. The classification is performed by tree classifiers constructed according to the CART methodology. The key idea of our method is to generate multiple versions of a basic tree classifier, and use these to derive an aggregate classifier which has proved very successful in increasing accuracy. Section 1.1 of the paper briefly outlines related studies. Section 2 describes the classification strategy we have adopted, while Section 3 lists and describes the low-level features used to index the images. In Section 4 we report the results obtained.

Outdoor Pornographic Non-Pornographic

Fig. 1. The classification hierarchy.

*ciocca@https://www.360docs.net/doc/2f10707318.html,r.it, ?cusano@https://www.360docs.net/doc/2f10707318.html,r.it, ?schettini@disco.unimib.it, ?carla@https://www.360docs.net/doc/2f10707318.html,r.it

1.1Related works

Athitsos and Swain1and Gevers et al.2have investigated the problem of distinguishing photographs and graphics, Schettini et al.3,4that of distinguishing photographs, graphics and texts. As far as photographs classification is concerned, Szummer and Picard5 have performed indoor/outdoor classification by using k-nearest neighbors classifiers. To classify an image, they have first classified sub-blocks of the image independently, and then combined the results in different ways. Vailaya et al.6 have considered the hierarchical classification of vacation images: at the highest level the images are sorted into indoor/outdoor classes; outdoor images are then assigned to the categories of city or landscape; and finally landscape images are classified as sunset, forest or mountain categories. These authors have used vector quantization to estimate the class-conditional densities of the features, and then derived a MAP (maximum a posteriori probability) criterion. Lienhart and Hartmann7 presented algorithms for distinguish photo-like images from graphical images, actual from only photo-like photos, and presentation slides/scientific posters from comics.

We think that the key difference between our approach and those described in5 and6 is that, besides using different classification methods, the employ of trees has allowed us to consider many features at the same time, and handle the possible non-homogeneity of the feature space.

2.IMAGE CLASSIFICATION USING CLASSIFICATION AND REGRESSION TREES (CART)

To perform the classification we used tree classifiers built according to the CART methodology8. Briefly, these are classifiers produced by recursively partitioning the predictors’ space, each split being formed by conditions related to the predictors’ values. In tree terminology subsets are called nodes: the predictors’ space is the root node, terminal subsets are terminal nodes, and so on. Once a tree has been constructed, a class is assigned to each of the terminal nodes, and when a new case is processed by the tree, its predicted class is the class associated with the terminal node into which the case finally moves on the basis of its predictors’ values. The construction process is based on training sets of cases of known class.

Since in high dimensional and very complex problems, as is the case here, it is practically impossible to obtain accurate results in one step, no matter how powerful the chosen class of classifiers, we decided to use what is called a “perturbing and combining” method9,10 to perform the classification. Methods of this kind, which generate in various ways multiple versions of a base classifier and use these to derive an aggregate classifier, have proved very successful in improving accuracy. We used bagging (bootstrap aggregating), since it is particularly effective when the classifiers are unstable, as trees are, that is, when small perturbations in the training sets, or in the construction process of the classifiers, may result in significant changes in the resulting prediction. With bagging the multiple versions of the basic classifier are formed by making bootstrap replicates of the training set and using these as new training sets. The aggregation is made by majority vote. In any particular bootstrap replicate each element of the training set may appear repeated times, or not at all, since the replicates are obtained by resampling with replacement.

To provide a measure of confidence in the classification results and at the same time greater accuracy, we applied an ambiguity rejection rule11 to the bagged classifier: the classification obtained by means of majority vote is rejected if the percentage of trees that contribute to it is lower than a given threshold.In this way only those results to which the classifier assigns a given confidence, as set by the threshold, are accepted. The rule is “global” in the sense that it is constant over the feature space.

3.IMAGE DESCRIPTION USING PICTORIAL FEATURES

We have used color features and texture/edge features to index the images. The color features include the color distribution (moments of inertia of the color channels, and the main color region composition) and the skin color distribution. The texture/edge features include statistics on the wavelets decompositions and on edge and texture distributions.

3.1Color distribution

The color distribution of an image can be considered a probability distribution. Because any probability distribution is uniquely characterized by its central moments, color distribution can be characterized in this manner. In12 the first three

moments, the mean (E), standard deviation (σ), and skewness (s), of each color channel of the HSV color space, derived by transforming the R, G, B color coordinates, were computed.

The HSV color space was partitioned into eleven color zones corresponding to basic color names (red, orange, yellow, green, blue, purple, pink, brown, black, gray and white). This division was defined and validated empirically by different groups of examiners. Since after quantization the image presents noisy points due to the non uniform color regions, a max filter is applied to remove these points. From the uniform color regions that are formed by the filtering process, four spatial composition features are identified: fragmentation (the number of color regions), distribution of the color regions with respect to the center of the image, and distribution of the color regions with respect to the x axis, and with respect to the y axis13.

3.2Skin color distribution

We have used a statistical skin color detector based on the r, g chromaticities of the pixels in a training set of 30000 color skin data taken from different sources14. To model the probability distribution of skin color more precisely, we analyzed the skin color of three different human races: African skin, Caucasian skin and Indian skin. For each race we determined the corresponding chromaticity statistics and used these to detect the skin pixels in the images. The percentage of skin pixels was used as a feature for discriminating between indoor, outdoor and close-up images.

We used a second more sophisticated skin detector, to identify pornographic images, labeling as skin all those pixels with a small texture amplitude, selected by the first detector.

The texture amplitude is computed as in15:

))( (1 2I

med

I

med

A?

=.

where I is the luminance image, and med1, med2are two median filters of different size (the size of med1 is 5×5, the size of med2 is 7×7).

Regions of connected skin pixels are labeled and the following descriptors are computed:

?number of skin regions and total area;

?percentage of skin pixels belonging to small, medium size and large regions (skin regions covering less than 1% of the image area are considered small; regions covering more than 5% of the image are considered large);

?the average distance between the baricenter of the regions and the center of the image itself.

Figure 2 shows an example of skin detection and labeling process.

(a)(b)(c)(d)

Fig. 2. Skin detection and labeling: input image (a), after texture filter (b), after color filter (c), labeled regions (d). Small regions are depicted in blue, medium-size regions in green, large regions in red.

3.3 Wavelets

Multiresolution wavelet analysis provides representations of the image data in which both spatial and frequency information are present. In multiresolution wavelet analysis we have four bands for each level of resolution, resulting from the application of two filters: a low-pass filter (L) and an high-pass filter (H). The filters are applied in pairs in the four combinations LL, LH, HL and HH, followed by a decimation phase that halves the resulting image size. Each band (the output of the filtering and decimation processes) corresponds to a coefficient matrix one forth the size of the processed image. The final image, of the same size as the original, contains a smoothed version of the original (LL band) and three bands of details (LH, HL and HH). In our procedure the features are extracted from the luminance image using a three-step Daubechies multiresolution wavelet expansion producing ten sub-bands 16. Two energy features, the mean and variance, are then computed for each sub-band.

3.4 Edge and texture distributions

The statistical information on image edges is extracted by Canny's algorithm 17. The Canny operator, works in a multi-stage process, computing first the luminance image. After smoothing the image by a Gaussian convolution, a simple 2D first derivative operator is applied to the smoothed image on both the x and y directions to reveal the edges with high first derivative values. With these values, the algorithm computes the gradient magnitude for each pixel. Edge pixels have higher gradient values than non-edge ones. The algorithm tries to track the pixel with higher gradient value, setting at zero all pixels that are not on the edge line, and producing in output an image with lines only 1-pixel wide. This process is called non-maximal suppression . After which, all the non-zero pixels are tagged as being possible edge pixels. To find the real edge, an hysteresis phase is applied. Starting from the possible edge pixels, the algorithm searches for the next edge pixels This phase ensures that the edges have a smooth behavior. At the end of the execution of the Canny algorithm twelve statistics are computed.

We have based our estimate of texture features on the Neighborhood GrayTone Difference Matrix (NGTDM) proposed by Amadasun and King 18 and Tamura et al.19. These authors developed a set of five measures that correspond to the textual properties: of coarseness, contrast, busyness, complexity, and strength. These measures are based on the spatial changes in intensity of the pixel’s image. The difference between the intensity level of the pixel and that of the neighboring pixels is used to build the one-dimensional difference matrix on which feature computation is based.

4. EXPERIMENTAL RESULTS AND DISCUSSION

The first task in the hierarchy concerned the classification of pornography. The image database used in this experiment contained 750 pornographic images and 750 non-pornographic images. The training set was randomly selected and contained 1000 images (500 of each class); the test set consisted of the remaining 500 photos. We used as predictors the features listed in Section 3. Table 1 shows the classification accuracy of a single tree classifier on the test set.

Predicted class

Not Pornographic

Pornographic

Not Pornographic 0.7720.228Pornographic 0.208

0.792

T

r u e c l a s s Table 1. Classification accuracy obtained on the test set using a single tree.

Tables 2 shows, instead, the classification accuracy achieved on the test set using a bagged classifier obtained by aggregating 25 trees built on the basis of bootstrap replicates of the training set (the aggregation of a larger number of trees brought no significant improvement). As expected, the use of the bagged classifier produced a marked improvement in classification accuracy of 3% and 7% in the test set, for non-pornographic and pornographic classes respectively.

Predicted class

Not Pornographic

Pornographic

Not Pornographic 0.8010.199Pornographic 0.138

0.862

T

r u e c l a s s Table 2. Classification accuracy obtained on the test set using the bagged classifier.

Figure 3, shows the accuracy of the classification as function of the percentage of rejected images.

Fig. 3. Accuracy-rejection trade-off.

As said, the second task was to classify a digital photo as indoor, outdoor, or close-up. The indoor class included photographs of rooms, groups of persons, and details in which the context also indicated that they were taken inside.The outdoor class included natural landscapes, buildings, city shots, and details in which the context concurred to indicate that the photographs were taken outside. The close-up class included portraits and photos of objects in which the context supplied no information of where the photo was taken. The image database used in this part of our experiments contained over 9000 images collected from various sources, such as images downloaded from the web, or acquired by scanner. It included some 2100 indoor images, 4700 outdoor images, and 2300 close-ups. All this material varied in size (ranging from 150x150 pixels to 900x900 pixels), resolution, and tonal depth.

Table 3 shows the classification accuracy achieved using a single tree classifier on the test set. The training set was equally distributed among the typologies present in the three classes, and contained about 4200 images (1400 indoor,1400 outdoor and 1400 close-up). The test set contained some 4900 photos (700 indoor, 3300 outdoor and 900 close-up) which had not been utilized in the training set.

Predicted class

Indoor Outdoor Closeup Indoor 0.7870.1170.096Outdoor 0.0850.8170.098Closeup

0.079

0.077

0.844

T r u e c l a s s Table 3. Classification accuracy obtained on the test set using a single tree.

Table 4 shows the classification accuracy achieved on the test set, using a bagged classifier obtained by aggregating 25trees built on the basis of bootstrap replicates of the training set. The use of the bagged classifier produced an improvement in classification accuracy of 9%, 8% and 4%, for the indoor, outdoor and close-up classes respectively.

Predicted class

Indoor Outdoor Closeup Indoor 0.8810.0500.069Outdoor 0.0270.9020.071Closeup

0.054

0.061

0.885

T r u e c l a s s Table 4. Classification accuracy obtained on the test set using the bagged classifier.

The misclassified images are generally photographs that are either overexposed or underexposed, or with a background that provides little information about the class to which the images belong. Indoor images misclassified as outdoor often show a window, while outdoor images misclassified as indoor are images of building details, with little outdoor background. We consider acceptable the misclassification of close-up images as indoor or outdoor and viceversa: it simply reveals the overlapping between the close-up and the other categories.

Figures 4, shows the effects of the application of the rejection rule in the accuracy-rejection plane. Some examples of misclassified images are showed in fig. 5 and 6.

Fig. 4. Accuracy-rejection trade-off.

(a)(b)(c)(d)

Fig. 5.Some misclassified images: (a,b) non-pornographic images classified as pornographic; (c,d) pornographic

images classified as non-pornographic.

(a)(b)(c)(d)

(e)(f)(g)(h)

Fig. 6.Some misclassified images: (a,b) indoor images classified as outdoor; (c,d) outdoor images classified as indoor; (e,f) outdoor images classified as close-up; (g,h) close-up images classified as indoor.

5.CONCLUSIONS

We believe that content-based image classification can play an important role in many branches of imaging and multimedia. We have presented here a digital photo classifier that has provided a reasonably good performance on a generic database of over 9000 images collected from various sources. This classifier could be further refined. In particular, we should like to create a more robust rejection rule by incorporating, together with the global measure proposed here, feature space-dependent information, such as the accuracy inside the terminal nodes.

An online version of our system is available at the address: https://www.360docs.net/doc/2f10707318.html,r.it/qclass/quickclass.html.

REFERENCES

1.V. Athitsos, M. Swain, “Distinguishing photographs and graphics on the World Wide, Web”. Proc. Workshop in

Content-based Access to Image and Video Libraries, 10-17 (1997).

2.T. Gevers, AWM Smeulders, “PicTo Seek: combining color and shape invariant features for image retrieval”,

IEEE Trans. On Image Processing, 19(1), 102-120 (2000).

3.R. Schettini, C. Brambilla, A. Valsasna, M. De Ponti, “Automatic image classification using pictorial features”,

Proc. VIII Color Imaging Conference, Scottsdale (Arizona), 184-188 (2000).

4.R. Schettini, C. Brambilla, G. Ciocca, A.Valsasna, M. De Ponti, “A Hierachical Classification Strategy for Digital

Documents”,Pattern Recognition, 35, 1759-1769 (2002).

5.M Szummer, R. Picard,“Indoor-outdoor image classification”, Proc. Int. Workshop on Content-Based Access of

Image and Video databases, 42-51 (1998).

6. A. Vailaya, M. Figueiredo, A. K. Jain, and H.-J. Zhang, “Image classification for content-based indexing”, IEEE

Transactions on Image Processing, 10(1) ,117-130 (2001).

7.R. Lienhart, A. Hartmann, “Classifying images on the web automatically”, Journal of Electronic Imaging, 11(4),

445-454 (2002).

8.L. Breiman, J.H. Friedman, R.A. Olshen, C.J. Stone, Classification and Regression Trees, Wadsworth and

Brooks/Cole, 1984.

9.L. Breiman, “Bagging predictors”, Machine learning, 26, 123-140 (1996).

10.L. Breiman, “Arcing classifiers”, Annals of Statistics, 26, 801-849 (1998).

11.Vailaya and A. Jain, “Reject option for VQ-based bayesian classification”, Proc. 15th International Conference on

Pattern Recognition, Barcelona, Spain, September, 2000.

12.M.A Stricker, M. Orengo, “Similarity of color images”, SPIE Storage and Retrieval for Image and Video

Databases III Conference, (1995).

13.P. Ciocca, R. Schettini, “A relevance feedback mechanism for content-based retrieval”, Information Processing

and Management, 35, 605-632 (1999).

14.R. Schettini, A. Valsasna, C. Brambilla and M. De Ponti, “A new classification strategy for color documents”,

Internet Imaging II, Proc of SPIE, 4311, 70-78, (2001).

15.M. Fleck, D. Forsyth, and C. Bregler, “Finding Naked People”, European Conf. on Computer Vision ,2, 592-602

(1996).

16.P. Scheunders , S. Livens, G. Van de Wouwer, P. Vautrot, D. Van Dyck, “Wavelet-based texture analysis”,

International Journal Computer Science and Information management. wcc.ruca.ua.ac.be/~livens/WTA/, (1997).

17.J. Canny, “A computational approach to edge detection”, IEEE Trans. On Pattern Analysis and Machine

Intelligence, IEEE-8, 679-698 (1986).

18.M. Amadasun, R. King, “Textural features corresponding to textural properties”, IEEE Transaction on System,

Man and Cybernetics, 19(5), 1264-1274 (1989).

19.H. Tamura, S. Mori, T. Yamawaki, “Textural features corresponding to visual perception”, IEEE Transaction on

System, Man and Cybernetics, 8, 460-473 (1978).

20.Y. Miyake, H. Saitoh, H. Yaguchi, and N. Tsukada, “Facial Pattern detection and color correction from television

picture for newspaper printing”, Journal of Imaging Technology, 16, 165-169 (1990).