Image Interpolation Using Classification-based Neural Networks

Image Interpolation Using Classification-based Neural Networks Hao Hu, Student Member, IEEE, Paul M. Hofman and Gerard de Haan, Senior Member, IEEEAbstract — Standard image interpolation methods generally use a uniform interpolation filter on the entire image. To achieve a better performance on specific structures, some content adaptive interpolation methods, such as Kondo’s method [1], have been introduced. However, these content adaptive methods are limited to fit image data into a linear model in each class. We investigate replacing the linear model by a flexible non-linear model, such as a feed-forward neural network. This results in a new interpolation algorithm based on known classification, but achieving better results. In this paper, such a classification-based neural network approach and its evaluation are presented. Both objective and subjective image quality results indicate that the proposed method gives an additional improvement in the interpolated image quality1.Index Terms — image interpolation, neural network, non-linear models, classificationI.I NTRODUCTIONMAGE interpolation is commonly performed by linear filters. The well-known bilinear and bicubic interpolation algorithms use linear interpolation filters, which have a drawback that they tend to produce undesirable blurring in the interpolated images. To enhance the sharpness and edge performance of image interpolation, some content adaptive interpolation methods have been introduced, such as Kondo’s method [1] and Atkins’ method [2]. Kondo’s method uses a simple method to classify the local image structure in the filter’s aperture and computes the optimal linear filter for each class. Kondo’s method has a better performance on edges than common linear methods [3]. Atkins’ method uses a more complicated classification method based on a stochastic model and obtains a high resolution image by mixing the outputs of optimized linear filters with coefficients that depend on the classification. Atkins’ method can also reduce edge blurring but is computationally more intensive than Kondo’s method.As an approach of non-linear filtering methods, neural networks can be used for image interpolation. Some applications use neural networks as function approximators. Plaziac [4] proposed a forward neural network to restore the1Hao Hu is with Eindhoven University of Technology, Den Dolech 2, 5600 MB Eindhoven, the Netherlands (e-mail: hao.hu@).Paul M. Hofman is with Philips Research Laboratories Eindhoven, Prof. Holstlaan 4, 5656 AA Eindhoven, the Netherlands (e-mail: paul.hofman@ ).Gerard de Haan is with Philips Research Laboratories Eindhoven, Prof. Holstlaan 4, 5656 AA Eindhoven, the Netherlands (e-mail: g.de.haan@ ). high resolution image by filling the gaps from a decimated high resolution image. This neural network model had 24 input pixels from the low resolution content and 16 hidden neurons. The paper reported that the method outperformed linear and median filters. A similar method was also proposed by Go [5] to realize color interpolation with a Bayer pattern for digital still cameras.Other interpolation schemes use the neural network as a pattern classifier. Marsi [6] used a neural network to segment the image according to the presence of oriented edges, then applied a set of different directional adaptive filters to interpolate the image. The filter outputs were weighted according to the output of the neural network. Ahmed [7] proposed to use a neural network to segment and label document images into text, halftone images, and background. Once the segmentation was performed, a specific enhancement or interpolation kernel was applied to each document component.In this paper, we propose an interpolation method using feed-forward neural networks with coefficients that are specific for a class of local structure. The proposed method classifies the local structure of a 3X3 pixel neighborhood in the same way as Kondo’s method, but uses non-linear filters (neural networks) for the interpolation, rather than linear filters. The optimal coefficients are obtained on the basis of the same error criteria as Kondo’s method. Experimental results show that the proposed neural network approach has a better interpolation performance than Kondo’s method.In Section II, we give a brief introduction about Kondo’s method. Then we describe how to apply the proposed interpolation method and obtain the optimal coefficients in Section III. Section IV includes experiment results. Finally, the conclusion and discussion are provided in Section V.II.K ONDO’S METHODKondo’s method is a content adaptive filtering approach, that is, the coefficients of interpolation filters depend on the pattern of the local structure. A block diagram of the method is presented in Fig. 1.In a 3X3 aperture, all pixel values are encoded into binary 0 or 1 by using Adaptive Dynamic Range Coding (ADRC) [8]. The 1-bit ADRC code of every pixel is defined by:1,()()0,()avavi iADRC ii i≥⎧=⎨<⎩(1) where i is the value of pixels in the aperture and av i is the average value of all pixels in the aperture.IFig. 1 The interpolation process of Kondo’s method: the local image structure is classified using ADRC and the filter coefficients can be obtained from LUT.The concatenation of ADRC(i) of all pixels in the aperture gives class code, c . The class, c , is used as the index to a Look-Up-Table (LUT) that contains a set of filter coefficients for every class. The high resolution pixels are computed by weighing the pixel values with the coefficients and subsequently summing the results.11,12,29,9c c c y w i w i w i =+++" (2)where 1,c w ,2,c w , … 9,c w are the coefficients for the classwhich index in the LUT is c . To interpolate the value of 2y ,the aperture is flipped horizontally as shown in Fig. 2. Then 2y can be calculated in the same way as 1y . Similarly, 3y and4y can be calculated using vertical flipping.Fig. 2 Aperture flipping: the high resolution pixel y 2 is flipped to the position of y 1 after the aperture flipping.To obtain the optimal interpolation coefficients, Kondo’smethod employs original high resolution images and down-scaled images as training material and uses the Least MeanSquares (LMS) method to get the optimal coefficient for every class. Suppose in one class, c , the number of total training samples is N . Let orig y be the original high resolution pixel value and inpl ybe the interpolated pixel value. The sum square error then is: 22,,1()Nc orig n inpl n n e y y ==−∑(3) Insert (2) into (3), then the sum square error becomes22,1,1,2,2,9,9,1[()]Nc orig n c n c n c n n e y w i w i w i ==−+++∑" (4) To get the minimal value of 2c e , let the first derivatives of 2c eto 1,c w , 2,c w , …, 9,c w equal zero.21,,1,1,2,2,9,9,11,22,,1,1,2,2,9,9,12,29,,1,1,2,2,9,9,19,2[()]02[()]02[()]0Nc n orig n c n c n c n n c N cn orig n c n c n c n n c Nc n orig n c n c n c n n c e i y w i w i w i w e i y w i w i w i w e i y w i w i w i w ===∂=−+++=∂∂=−+++=∂∂=−+++=∂∑∑∑""""(5)Equations (5) comprise of a set of linear equations which are solved in a straightforward manner by matrix inversion. Then, the optimal coefficients for class c are obtained.III. T HE PROPOSED METHODThe block diagram of the proposed interpolation method is shown in Fig. 3. The system interpolates high resolution images using neural networks with coefficients that areselected from the LUT according to the classification.Fig. 3 The interpolation process of the proposed method. Note that the overall scheme is identical to Fig. 1 except that the linear interpolation model is replaced by a neural network.A. Interpolation ProcessSimilar to Kondo’s method, we use all pixels in a 3X 3 aperture in the low resolution image to estimate the corresponding interpolated pixels. The nine pixels in theaperture are presented to the neural network as an input vector.The ADRC code of the input is used to choose thecorresponding set of coefficients for the neural network from the LUT.The neural network model (Fig. 4) that we use is a simplethree-layer feed forward architecture which has only a fewhidden neurons and one output neuron. The non-linear function used in the hidden layer is the hyperbolic tangent function whereas a linear function is used at the output layer. Suppose there are h N hidden neurons in the network. Thelow-resolution imageinterpolated high-resolution imagelow-resolution imageinterpolated high-resolution imageoutput n h of one hidden neuron n is:tanh()n n n h w i b =⋅+KK (6)where i K is the input vector (1i ,2i , … , 9i ), n w Kis the weight vector between the input layer and the hidden neuron n and n b is the bias to the hidden neuron n . The output of thenetwork is:10y u h b =⋅+KK (7)where u Kis the weight vector between the hidden layer andthe output neuron, h Kis the output vector at the hidden layer (1h ,2h , …, h N h ) and 0b is the bias to the output neuron. Theother outputs 2y , 3y and 4y can be calculated by flipping the aperture as in Kondo’s method according to Section II.Fig. 4 The model uses a three-layer feed-forward architecture and it has a nonlinear transfer function in the hidden layer and a linear transfer function in the output layer.The total amount of parameters of the neural network in Fig.4 is about h N times the amount of the linear filter coefficients in Kondo’s method. Furthermore, note that the term n w i ⋅KK in(6) represents a single linear filter and that the neural network can be considered as direct extension of the linear model.B. Selective Training Our training procedure is shown in Fig. 5. We use the down-scaled image and the original image as the input and output target. Before training, input and output target pairs (i K , y ) are classified using ADRC on the input vector i K .The pairs that belong to one specific class are used for neural network training, resulting in optimal coefficients for thisclass. The optimal coefficients are obtained by using the Levenberg-Marquardt optimization algorithm [9] with the sumsquare error criterion. During training, the errors between outputs and targets are computed and the derivatives of the errors are back propagated to adjust the coefficients of the network and minimize the sum square errors. The optimization is done for each separate class. Once the trainingin one class is complete, the coefficients are stored in the corresponding index of the LUT and the training procedure is repeated for the next class. Because a similar classification is used, the number of coefficient sets is equal to that of Kondo’s method.Fig. 5 The selective training procedureIV. E XPERIMENTS AND R ESULTSTo compare the performance of Kondo’s method and the proposed method, we obtain the coefficients of these two methods by training on the same training material. Both of the methods use algorithms that minimize the sum of square errors to get optimal coefficients, therefore we also choose the Mean Square Error (MSE) as the criterion for objective quality comparison. We use the same evaluation process as Zhao et al. [3]. A. Training MaterialsThe training materials consist of 30 high resolution pictures with dimensions 3000X 4500 pixels. These pictures include a variety of natural images, including people, building, animals and landscape. We only use the luminance component of these images. At least 200,000 samples in each class are used for training. Note that the images which we evaluated the methods are not included in this training set.B. Neural Network SettingThe neural network used in our experiments has one hidden layer with five hidden neurons. Our experiments show thatabout two to five hidden neurons in the hidden layer are required to obtain significantly better results than Kondo’s method. Before training, the input vector and target value are rescaled from the range [0, 255] to [-1, 1], which corresponds to output range of hyperbolic tangent function. During the interpolation process, the output of the neural network will berescaled back from [-1, 1] to [0 255]. C. The down-scaling and MSE calculationTo generate training material and calculate the MSE for performance comparison, one needs to down-scale the high resolution material to a low resolution. Here, we use four-pixel averaging and a 3-tap peaking filter to give the best compensation to the averaging filter in [3].The down-scaled test image or sequence is up-scaled using Kondo’s method as well as the proposed method. The MSE between the interpolated and the original high resolutionmaterial can be calculated as:i 1 i 2i 9y 1neural network imagesdownscaled images2111((,)(,))J Korig inpl j k MSE y j k y j k JK ===−∑∑ (8) where J , K are the numbers of columns and rows in the image, (,)orig y j k , (,)inpl y j k is the pixel value of theoriginal image and the interpolated image respectively.D. Testing images and sequences The experiments are tested on the images and sequences used by Zhao et al [3]. This test material (shown in Fig. 6) includes: a sequence with high contrast edges in all directions (Bicycle), a sequence with low contrast details (Football), a still portrait image (Lena), a sequence with detailed lettering and fine structure (Siena), and a sequence with abundant detail (Tokyo).E. Results In Table I, the MSE score on every test image or video sequence and the average MSE score are provided forKondo’s method and the proposed method.(A) Bicycle(B) Football(C) Lena(D) Siena(E) TokyoFig. 6 Testing images and video sequencesTable I shows the proposed method has about 10%reduction in the MSE score on every test image and sequence. Especially on Bicycle sequence the difference is large probably because of many high contrast edges in this sequence. From the results one can see that by introducing aflexible neural model one can estimate the interpolated pixel more accurately and achieve a lower MSE score.T ABLE IMSE SCORE ON THE TEST IMAGE AND SEQUENCEMSE Kondo’s method Proposed methodBicycle 47.4 39.4 Football 59.8 55.2Lena 20.8 19.4Siena 80.1 76.9 Tokyo 53.4 51.8 Average 52.3 48.5To enable a subjective comparison, some image fragments from the Bicycle sequence interpolated by both methods are showed in Fig. 7. This figure demonstrates that the proposed method produces images closer to the original images. The image fragments of the first row show that Kondo’s method causes blockiness near the letters, while the propose methodrenders the borders more correctly. In image fragments of thesecond row, the proposed method generates thinner and highercontrast grid lines. The image fragments of the last row illustrate the differences of high contrast edge performance of the two methods. Kondo’s method causes some overshoots near the edges, while such overshoots are absent in the proposed method. V. C ONCLUSION A ND D ISCUSSION We have presented a classification-based neural network approach for image interpolation. The results show thatreplacing the linear filter in Kondo’s method with highly flexible neural models produces better results and avoids some artifacts of Kondo’s method. By using a large data set it waspossible to obtain a stable fit of the neural network despite a significant increase in the number of parameters. The use of pre-classification in the proposed method limits the complexity of the neural network while still achieving good results. In other words, the use of classification simplifies the function approximation of the neural network. The analysis of the trade off between the complexity of the interpolation model and the amount of classes merits furtherattention.R EFERENCES[1] T. Kondo, T. Fujiwara, Y. Okumura, and Y. Node, “Picture conversionapparatus, picture conversion method, learning apparatus and learning method ”, US-patent 6,323,905, Nov. 2001 [2] C. B. Atkins, C. A. Bouman, and J. P. Allebach, “Optimal image scaling using pixel classification”, 2001 International Conference on Image Processing , vol.3, pp. 864-867, 2001. [3] M. Zhao, J. A. Leitao and G. de Haan, “Towards an Overview of Spatial Up-conversion Techniques”, Proceedings of ISCE'02, pp. E13-E16, Sep. 2002[4] N. Plaziac, ‘Image interpolation using neural networks,” IEEE Trans. Image Processing , vol. 8, no. 11, pp. 381-390, 1999.[5] J. Go, K. Sohn and C. Lee, “Interpolation using neural networks fordigital still cameras”, IEEE Trans. Consumer Electronics , vol. 46, No. 3,Aug. 2000.[6] S. Marsi and S. Carrato, “Neural network-based image segmentation forimage interpolation”, Neural Networks for Signal Processing V. Proceedings of the 1995 IEEE Workshop , pp. 388 - 397, Sept. 1995. [7] M. N. Ahmed, B. E. Cooper, Shaun T. Love, “Adaptive image interpolation using a multilayer neural network,” Applications ofArtificial Neural Networks in Image Processing VI, Proc. SPIE , Vol.4305, pp. 30-38, Apr. 2001.[8] T. Kondo, and K. Kawaguchi, “Adaptive dynamic range encoding method and apparatus ”, US-patent 5,444,487, Aug. 1995. [9] R. Fletcher, Practical Methods of Optimization , Second Edition, JohnWiley & Sons Ltd, ISBN 0-471-49463-1, pp. 100-107, Aug. 2000.Kondo’s method Proposed methodOriginalFig.7 Image fragments processed by Kondo’s method, the proposed method and the original. Note that the proposed method has better performance on edges, especially in the indicated area in the image fragments.。