BP神经网络原理及MATLAB仿真

BP Neural Network principle and MATLAB Simulation

Xiong Xin Nie Mingxin

School of Information Engineering School of Information Engineering

Wuhan University of Technology Wuhan University of Technology

Wuhan, P. R. China 430070 Wuhan, P. R. China 430070

sense1559@https://www.360docs.net/doc/8e3396635.html, niemx@https://www.360docs.net/doc/8e3396635.html,

Abstract

This paper introduces the prevalent BP algorithm in neural network, and discusses the goodness、problem and training process of BP neural network, as well as using MATLAB software to simulate the numbers on the basis of it. At last, several improved BP training algorithms have been compared in the paper.

Keywords: BP neural network; number recognition; MATLAB

1 Introduction

The development of neural network is rapid since the first neural network model——MP model came up in 1943[1]. Hopfield neural network proposed in 1982 and opposite phase broadcast algorithm proposed by Rumelhart in 1985 make the neural network of Hopfield model and multilayer feedforward model to be the prevalent neural network model. They are effective in many applications of fields such as speech recognition, mode recognition, image processing and industry controlling.

Neural network is an theory which is imitative of the biological processing model to get the function of information intelligent processing. It treats with the pattern information which is hard to be expressed in certain language by the method from bottom to top and parallel distribution way formed by self-study, self-organization and non-linear dynamics. Neural network is a parallel and distributed information processing network architecture. It is generally composed by massive neurons, each of which has only one output that can connect many other neurons. The reciprocity between neurons is embodied by their connected weighs. The output of neurons is its input function. The types of functions in common use have linear function, Sigmoid function and value function.

There are two phases of positive transmitting processing and error reverse transmitting processing in the study processing of BP neural network[2]. The signal inputted from outside spreads to the output layer and gives the result through processing layer for layer of neurons in input layer and hidden layer. If the expected output can’t be obtained in output layer, it shifts to the conversed spreading processing and the true value and the error outputted by network will return along the coupled access formerly. The error is reduced by modifying contacted weight value of neurons in every layer and then it shifts to the positive spreading processing and revolves iteration until the error is smaller the given value. Take a three layer network for example, the network is composed of N input neurons, K hidden neurons and M output neurons(as showed in fig.1). O2pm and O1pk are the output value of output layer and hidden layer respectively. w2km and w1nk are the connected weight value from the hidden layer to the output layer and from the input layer to the hidden layer respectively. Suppose the input studying sample is Xpn, so its corresponding expected output value is tpm.

Fig.1 BP neural network configuration

2 BP Neural Network

2.1 the Discussion about the Advantages

and Disadvantages of BP Neural Network

BP neural network is a kind of neural network forms which has most applications currently[3], but it isn’t very perfect. In order to understand how to apply the neural network to resolve problems, we carry on the discussion about its advantages and disadvantages here.

The advantages of BP neural network:

① Network realizes the mapped function from input to output and mathematical theory has proved that it has the function to achieve any complex non-linear mapping;

②Network can extract the “logical” solution rules automatically through studying the examples with correct results. It has the ability of self-study;

③Network has the definite abilities of promotion and generalization.

The disadvantages of BP neural network:

①The study speed of BP algorithm is very slow. The main causations of it are:

a. Because BP algorithm is grads declining method essentially and the aim function optimized by it is very complex, the “sawtooth-shaped phenomenon” is bound to appear which makes the BP algorithm inefficiency;

b. The torpid phenomenon exists. Because the optimized aim function is very complex, it can appear some flat areas in the case of the output of neurons approach 0 or 1. In these areas, the error of weight value changes very little, which can hardly make the training processing break down;

c. In order to execute the BP algorithm in the network, we can’t use the traditional one-dimensional search method to solve the interative step length every time. We should put the network the updated rules of step length in advance. The former algorithm will make the algorithm inefficiency.

②the network training is much more likely to fail, the reasons as below: a. From the perspective of mathematics, BP algorithm as a kind of local searching optimized method, it is used to solve the overall extremum of complex non-linear function, so the algorithm is likely to be gotten into the local extremum and make the training fail;

b. The approaching and promoting abilities are closely linked with the representative of studying sample. It is a hard problem to choose the training collection composed of the typical sampling examples.

③The contradiction between the scale of examples and network is hard to solve, which refers to the relationship of possibility and feasibility of network capacity, viz. the problem of studying complexity;

④The choice of network configuration has still no a uniform and integrate academic guidance and it can be selected by experience. Therefore some people call the structure choice of neural network is a kind of art. The network structure infects the approaching ability and promoting character directly. So how to choose an appropriate network structure is an important problem;

⑤New samples can infect the network which studies successfully and the number that describes the character of every input sample should be equal;

⑥There is contradiction between the predictive ability of network(also called generalization ability or promoting ability) and training ability(also called approaching ability or study ability). Usually when the training ability is poor, the predictive ability will be poor and in a certain extent, with the improvement of training ability the predictive ability is also improved. However, this trend has a limit. When achieving this limit, with the improvement of the training ability the predictive ability will be decline on the contrary, which is also called the over fitting phenomenon. And now the network studies

https://www.360docs.net/doc/8e3396635.html,

too much detail of samples and can’t reflect the embedded laws of samples.

2.2 BP Network algorithm

The training process of BP network is as below [4].

(1) Initialization. Endow every connected weight value and threshold value with a lesser random value.

(2) Input the corresponding neurons in input layer with an component of a eigenvector Xpk = (Xpk1 ,Xpk2 Xpk3 ,…,Xpkn) .

(3) Use the eigenvector of input samples to calculate the corresponding output value

O1pk=f(Xpkn) of neurons in hidden layer. (4) Use each unit output O1pk in hidden layer to calculate the input value in each output layer and then further calculate the corresponding output O2pm ,O2pm

=f(O1pk) of each unit in output layer. (5) Calculate the generalized error of each unit in output layer via the teach signals.

(6) Use the connected weight value W2km between middle layer and output layer, the generalized error δi in each unit of output layer and the output O1pk of each unit in middle layer to calculate the generalized error of each unit in middle layer.

(7) Use the generalized error δj of each unit in output layer and the output O1pk of each unit in middle layer to modify the weight value

w2km between output layer and middle layer and the threshold value Yj of each unit in output layer.

(8) Use the generalized error ei of each unit in middle layer and the input Xpkn of each unit in input layer to modify the weight value

w1nk between input layer and middle layer and the threshold value θi of each unit in hidden layer.

(9) Select the next sample in order and return 2 until the samples in training collection are all studied over.

(10) Return 2 over again until the error function is lower than the predetermined value, viz. the times of network constringency or study is greater than the predetermined value.

3 the Applications of BP Network in digital recognition

Firstly, the numbers 0～9 should be carried on the digital processing to constitute the input samples. Considering that using the 0-1 image of 5×5 matrix will represent every number clearly and then it designs and trains an neural network which can recognize the ten numbers 0~9 once again. When giving the trained network with an input that can represent some number, network can represent this number correctly through the 8421BCD code in output terminal. This network can remember all the ten numbers by study and training. The neural network training should be supervised to train ten groups of arrays which can represent numbers 0~9 and show the corresponding four two-scale numbers of 1~10 in output terminal.

3.1 the Design of network structure

From the analysis above, we can get that the neural network need to have N=5×5 input neurons and M=4 output neurons. The two layer logsig/logsig network adopting the logarithm Sigmoid type active function in the scope of (0,1) is quite effective to the 0,1 Boolean values. So the network may adopt the N-K-M structure, where the hidden layer chooses one layer, N is the number of neurons in input layer, K is the number of neurons in hidden layer and M is the number of neurons in output layer. The number ‘0’ and ‘9’ can represent respectively by 0-1 chart as:

le0=[1 1 1 1 1 le1=[0 0 1 0 0 1 0 0 0 1 0 0 1 0 0

1 0 0 0 1 0 0 1 0 0

1 0 0 0 1 0 0 1 0 0

1 1 1 1 1] 0 0 1 0 0]

le2=[1 1 1 1 1 le3=[1 1 1 1 1

0 0 0 0 1 0 0 0 0 1

1 1 1 1 1 1 1 1 1 1

1 0 0 0 0 0 0 0 0 1

1 1 1 1 1] 1 1 1 1 1]

le4=[1 0 0 0 1 le5=[1 1 1 1 1

1 0 0 0 1 1 0 0 0 0

https://www.360docs.net/doc/8e3396635.html,

1 1 1 1 1 1 1 1 1 1

0 0 0 0 1 0 0 0 0 1

0 0 0 0 1] 1 1 1 1 1]

le6=[1 1 1 1 1 le7=[1 1 1 1 1

1 0 0 0 0 0 0 0 1 0

1 1 1 1 1 0 0 1 0 0

1 0 0 0 1 0 1 0 0 0

1 1 1 1 1] 1 0 0 0 0]

le8=[1 1 1 1 1 le9=[1 1 1 1 1

1 0 0 0 1 1 0 0 0 1

1 1 1 1 1 1 1 1 1 1

1 0 0 0 1 0 0 0 0 1

1 1 1 1 1] 1 1 1 1 1] 3.2Network MATLAB Simulation

To the applied BP network above, we can use the functions in MATLAB neural network toolbox to simulate[5]. The neural network toolbox is one of toolboxes under the MATLAB environment. It constructs the excitation functions of typical neural network such as the S type, linear type, competitive type and saturated linear type functions based on the artificial neural networks theory and using the MATLAB language. These functions can make the designer change the selected network output calculation to the transfer of excitation functions.

Hidden layer gets 10 neurons relied on experience. To the network structure which is 25-10-4 structure, input layer has 25 neurons and each number can be represented with 0-1 chart of 5×5 matrix whose elements can constitute a numeric column matrix. The 10 numbers are represented separately by a 25 × 10 input matrix which is composed of 10 numbers column matrix. And then send the 10 numbers to the variable named P:

P= [le0, le1, le2, le3, le4, le5, le6, le7, le8, le9]

Output layer has 4 neurons, because the objective vector expects that when each number is inputted, the four binary elements of 1~10 can be corresponded in the output terminal correctly.

The design parameters of network are: the largest training times 5000-2000(according to the training function); no noise recognized

training precision 0.1; the rate of study 0.01; momentum constant 0.95; the biggest error ratio 1.05.

The program flow as below:

le0=[1 1 1 1 1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 1]';

le1=[0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0]';

le2=[1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1]';

le3=[1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1]';

le4=[1 0 0 0 1 1 0 0 0 1 1 1 1 1 1 0 0 0 0 1 0 0 0 0 1]';

le5=[1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1]';

le6=[1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1]';

le7=[1 1 1 1 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0]';

le8=[1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1]';

le9=[1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1]';

P=[le0,le1,le2,le3,le4,le5,le6,le7,le8,le9];

T=[0 0 0 0;0 0 0 1;0 0 1 0;0 0 1 1;0 1 0 0;0 1 0 1;0 1 1 0;0 1 1 1;1 0 0 0;1 0 0 1]';

%adopting feedforward BP network algorithm %and the BP algorithm training function

%Levenberg_Marquardt

net=newff(minmax(P),[10,4],{'logsig','logsig'} ,'trainlm');

%the biggest training times is 5000 here

net.trainParam.epochs=5000;

net.trainParam.goal=0.1;

net.trainParam.lr=0.01;

net.trainParam.mm=0.95;

net.trainParam.er=1.05

net=train(net,P,T);

The training curve that come out finally is shown in fig.2, in which we can that the network converges to the required precision at a great speed. In the condition that objective convergent precision and other training

parameter don’t change, compare the simulations separately of the other several BP algorithm training functions, the result shown in table 1. In this table we can conclude that in the condition of not affecting the precision, adopting the L-M optimized algorithm is the most fast.

Fig.2 Training curve of L-M algorithm Table 1 the comparison of training effect in

different BP algorithms

Training function

BP algorithm

biggest training times / actual training times

convergence precision

traingd Gradual descendent BPalgorithm 10000/ 6842 0.0999982

traingda

Gradual descendent self-adaptw/l BPalgorithm

5000/ 130

0.0971766

traingdm

Gradual descendent w/momentum BPalgorithm

20000/ 18373

0.0999995

traingdx

Gradual descendent w/momentum and self-adapt BPalgorithm

5000/ 156

0.0957755

trainlm

Levenberg_

Marquardt BP algorithm 5000/ 7

0.0994116The adjust of Levenberg – Marquardt

algorithm weight value is [6]

:

（1）

Where J is a Jacobian matrix which is composed of the differential coefficient by error to weight value, E is the error vector, m is a scalar quantity which depends on the width of m. This method is ranging between the two extreme cases smoothly which are the Newton method(when m →0) and the steepest decline method(when m →∞).

4 Conclusions

Train the network by five kinds of BP algorithm and the result shows that for the part of the rapidity of convergence, the L-M algorithm is the fast and its precision is the highest, but other algorithms are not so good as the L-M algorithm relatively. L – M algorithm is fit to some trainings which have a

great quantity of samples. In addition, the reliability of digital recognition using neural network can be gotten by using hundreds of vectors that have random noise. If a higher recognized precision is required, it can lengthen the time of network training and make the training error precision higher or increase the numbers of the neurons in hidden layer. Otherwise, it can enhance the resolving power of input vectors, for example, adopting 16×16 lattices and so on.

References:

[1] Ying Liandong.The Design and Application of

BP Neural Network [J]. Information Technology,2003,27(6):18-20.

[2] NG S C, CHEUNG C C,LEUNG S H. Fast

Convergence for Back - Propagation Network with Magnified Gradient Function[ J ].IEEE, 2003, 9 (3) : 1903 - 1908.

[3] He Qingbi, Zhou Jianli. The convergence and

improvements of BP neural network[J]. Journal Of ChongQing Jiao TTong University, 2005,24(1):143-145. [4] Fan lei, Zhang Yuntao, Chen Zhenjun.

Application of Improved BP Neural Network

Based onMatlab[J]. Journal of China West

Normal University(Natural Sciences), 2005,

26(1):70-73.

[5] Zhang Dexi, Bi Yuhua.The Application of

MATLAB in Pattern Recognition[J].Journal Of

XuChang Teachers COLLEGE, 2002,21(5):43-46.

[6] Yang Zhongjin, Shi Zhongke. Architecture

Optimization for Neural Network[J]. Project and

Application Of Computer, 2004,25:52-53.