Combinatorial evolution of feedforward neural network models for chemical processes

[摘要] ENGLISH ABSTRACT: Neural networks, in particular feedforward neural networks architectures such as multilayerperceptrons and radial basis function networks, have been used successfully in many chemicalengineering applications. A number of techniques exist with which such neural networks can betrained. These include backpropagation, k-means clustering and evolutionary algorithms. Thelatter method is particularly useful, as it is able to avoid local optima in the search space and canoptimise parameters for which there exists no gradient information. Unfortunately onlymoderately-sized networks can be trained by this method, owing to the fact that evolutionaryoptimisation is extremely computationally intensive. In this paper, a novel algorithm calledcombinatorial evolution of regression nodes (CERN) is proposed for training non-linearregression models, such as neural networks. This evolutionary algorithm uses a branch-and-boundcombinatorial search in the selection scheme to optimise groups of neural nodes. The useof a combinatorial search, for a set of basis nodes, in the optimisation of neural networks is aconcept introduced for the first time in this thesis. Thereby it automatically solves the problem ofpermutational redundancy associated with the training of the hidden layer of a neural network.CERN was· further enhanced by using clustering, which actively supports niches in thepopulation. This also enabled the optimisation of the node types to be used in the hidden layers,which need not necessarily be the same for each of the nodes. (i.e. a mixed layer of differentnode types can be found.) A restriction that does apply is that in order to make the combinatorialsearch efficient enough, the output layer of the neural network needs to be linear.CERN was found to be significantly more efficient than a conventional evolutionary algorithmnot using a combinatorial search. It also trained faster than backpropagation with momentum andan adaptive learning rate. Although the Levenberg-Marquardt algorithm is neverthelesssignificantly faster than CERN, it struggled to train in the presence of many non-local minima.Furthermore, the Levenberg-Marquardt learning rule tends to overtrain, (see below) and requiresa gradient information.CERN was analysed on seven real world and six synthetic data sets. Oriented ellipsoidal basisnodes optimised with CERN achieved significantly better accuracy with fewer nodes thanspherical basis nodes optimised by means of k-means clustering. On the test data multilayerperceptrons optimised by CERN were found to be more accurate than those trained by thegradient descent techniques, backpropagation with momentum and the Levenberg-Marquardtupdate rule. The networks of CERN were also compared to the splines of MARS and were found to generalise significantly better or as well as MARS. However, for some data sets, MARS wasused to select the input variables to use for the neural network models. Networks of ellipsoidalbasis functions built by CERN were more compact and more accurate than radial basis functionnetworks trained using k-means clustering. Moreover, the ellipsoidal nodes can be translated intofuzzy systems. The generalisation and complexity of the resulting fuzzy rules were comparableto fuzzy systems optimised by ANFIS, but did not result in an exponential increase of thenumber of rules. This was caused by the grid-partitioning employed by ANFIS and for data setswith a relatively high dimensionality, in comparison with the data points, the resultinggeneralisation was consequently much poorer than that of the CERN models.In summary, the proposed combinatorial selection scheme was able to make an existing evolutionaryalgorithm significantly faster for neural network optimisation. This made it computationallycompetitive with traditional gradient descent based techniques. Being an evolutionary algorithm, theproposed technique does not require a gradient and can therefore optimise a larger set of parameters in comparison to traditional techniques.

[发布机构] Stellenbosch University