Training a perception is a process of choosing values for the weights in the space H of all possible weight vectors: ox1,x2,…,xn =1if  w0+w1x1+w2x2+w3x4⋯+wnxn>0−1otherwise, (1) where wi is the weight that determines the contribution of input xi. From (1), the original perceptron is single-layer and can only express linear decision surface and the inputs have to be flt linearly separable.

To overcome these shortcomings, the perceptron was extended to multiple layers, or the multilayer perceptron (MLP). The major difference between the original perceptron and MLP is that each neuron’s output in MLP is a nonlinear and differentiable function (namely, activation function) of its inputs [8]. The MLP’s nonlinear feature allows for representing more complex systems. Later, Werbos [9] and Rumelhart et al. [10] developed efficient backpropagation training algorithms for the MLP which significantly extend the MLP’s applicability in various fields. It is apparent that the feedforward neural network treats all the data as new and cannot discover the possible temporal dependence between samples. This shortcoming

sometimes needs a feedforward neural network to be extended to a rather large scale to approximate complex systems. In other words, the feedforward neural network has a memoryless structure. By contract the RNN allows for the internal feedback and is more appropriate to solve certain types of dynamic problems. Jordan introduced the first RNN which feeds the outputs back to the input vector with time delay [11]. In other words, the RNN output at time t will be used as part of input information at t + 1. Mathematically, the outputs of a three-layer Jordan network with m, q, and n neurons on the input layer, hidden layer, and output layer, respectively, are as follows: ot+1,j=Fβj,0+∑h=1qβj,hGγh,0+xt′γh+ot′δh,j=1,2,…,n, (2) where xt′, ot′ are vectors of input and output at time t; δh is the vector of the connection weights between hth hidden neurons and input neurons which receive lagged outputs; βj = (βj,1, βj,2,…,βj,q)′ is the vector of the connection weights between the

jth output neuron and all q hidden neurons; γh = (γh,1, γh,2,…,γh,m)′ is the vector of the connection weights between the hth hidden neuron and all m input neurons; F and G are the activation functions on the output layer and hidden layer, respectively; and βj,0, GSK-3 γh,0 are biased terms to add the flexibility of activation functions. Similarly, Elman designed a RNN that the hidden neurons are connected to input neurons with time delay as in (3) [12]. Consider ot+1,j=Fβj,0+∑h=1qβj,hat,h, j=1,2,…,n,at+1,h=Gγh,0+x′γh+at′δh, h=1,2,…,q, (3) where at = (at,1, at,2,…,at,q)′ is the vector of lagged hidden-neuron activations; δh is the connection weights between the hth hidden neuron and all the inputs which receive lagged hidden neuron activations.