Mạng thần kinh thường xuyên cho dự đoán P6

Neural Networks as Nonlinear Adaptive FiltersPerspectiveNeural networks, in particular recurrent neural networks, are cast into the framework of nonlinear adaptive filters. In this context, the relation between recurrent neural networks and polynomial filters is first established. Learning strategies and algorithms are then developed for neural adaptive system identifiers and predictors. Finally, issues concerning the choice of a neural architecture with respect to the bias and variance of the prediction performance are discussed....
Nội dung trích xuất từ tài liệu:
Mạng thần kinh thường xuyên cho dự đoán P6 Recurrent Neural Networks for Prediction Authored by Danilo P. Mandic, Jonathon A. Chambers Copyright c 2001 John Wiley & Sons Ltd ISBNs: 0-471-49517-4 (Hardback); 0-470-84535-X (Electronic)6Neural Networks as NonlinearAdaptive Filters6.1 PerspectiveNeural networks, in particular recurrent neural networks, are cast into the frameworkof nonlinear adaptive filters. In this context, the relation between recurrent neuralnetworks and polynomial filters is first established. Learning strategies and algorithmsare then developed for neural adaptive system identifiers and predictors. Finally, issuesconcerning the choice of a neural architecture with respect to the bias and varianceof the prediction performance are discussed.6.2 IntroductionRepresentation of nonlinear systems in terms of NARMA/NARMAX models has beendiscussed at length in the work of Billings and others (Billings 1980; Chen and Billings1989; Connor 1994; Nerrand et al. 1994). Some cognitive aspects of neural nonlinearfilters are provided in Maass and Sontag (2000). Pearson (1995), in his article onnonlinear input–output modelling, shows that block oriented nonlinear models area subset of the class of Volterra models. So, for instance, the Hammerstein model,which consists of a static nonlinearity f ( · ) applied at the output of a linear dynamicalsystem described by its z-domain transfer function H(z), can be represented 1 by theVolterra series. In the previous chapter, we have shown that neural networks, be they feedforwardor recurrent, cannot generate time delays of an order higher than the dimension ofthe input to the network. Another important feature is the capability to generatesubharmonics in the spectrum of the output of a nonlinear neural filter (Pearson1995). The key property for generating subharmonics in nonlinear systems is recursion,hence, recurrent neural networks are necessary for their generation. Notice that, as 1 Under the condition that the function f is analytic, and that the Volterra series can be thoughtof as a generalised Taylor series expansion, then the coefficients of the model (6.2) that do not vanishare hi,j,...,z = 0 ⇔ i = j = · · · = z.92 OVERVIEWpointed out in Pearson (1995), block-stochastic models are, generally speaking, notsuitable for this application. In Hakim et al. (1991), by using the Weierstrass polynomial expansion theorem,the relation between neural networks and Volterra series is established, which is thenextended to a more general case and to continuous functions that cannot be expandedvia a Taylor series expansion. 2 Both feedforward and recurrent networks are charac-terised by means of a Volterra series and vice versa. Neural networks are often referred to as ‘adaptive neural networks’. As alreadyshown, adaptive filters and neural networks are formally equivalent, and neural net-works, employed as nonlinear adaptive filters, are generalisations of linear adaptivefilters. However, in neural network applications, they have been used mostly in sucha way that the network is first trained on a particular training set and subsequentlyused. This approach is not an online adaptive approach, which is in contrast withlinear adaptive filters, which undergo continual adaptation. Two groups of learning techniques are used for training recurrent neural net-works: a direct gradient computation technique (used in nonlinear adaptive filtering)and a recurrent backpropagation technique (commonly used in neural networks foroffline applications). The real-time recurrent learning (RTRL) algorithm (Williamsand Zipser 1989a) is a technique which uses direct gradient computation, and is usedif the network coefficients change slowly with time. This technique is essentially anLMS learning algorithm for a nonlinear IIR filter. It should be noticed that, with thesame computation time, it might be possible to unfold the recurrent neural networkinto the corresponding feedforward counterparts and hence to train it by backprop-agation. The backpropagation through time (BPTT) algorithm is such a technique(Werbos 1990). Some of the benefits involved with neural networks as nonlinear adaptive filters arethat no assumptions concerning Markov property, Gaussian distribution or additivemeasurement noise are necessary (Lo 1994). A neural filter would be a suitable choiceeven if mathematical models of the input process and measurement noise are notknown (black box modelling).6.3 OverviewWe start with the relationship between Volterra and bilinear filters and neural net-works. Recurrent neural networks are then considered as nonlinear adaptive filters andneural architectures for this case are analysed. Learning algorithms for online trainingof recurrent neural networks are developed inductively, starting from correspondingalgorithms for linear adaptive IIR filters. Some issues concerning the problem of van-ishing gradient and bias/variance dilemma are finally addressed.6.4 Neural Networks and Polynomial FiltersIt has been shown in Chapter 5 that a small-scale neural network can represent high-order nonlinear systems, whereas a large number of terms are required for an equiv- 2 For instance nonsmooth functions, such as |x|.NEURAL NETWORKS AS NONLINEAR ADAPTIVE FILTERS 93alent Volterra series representation. For instance, as already shown, after performinga Taylor series expansion for the output of a neural network depicted in Figure 5.3,with input signals u(k − 1) and u(k − 2), we obtainy(k) = c0 + c1 u(k − 1) + c2 u(k − 2) + c3 u2 (k − 1) + c ...