p. 1

[close]

p. 2

yaochu jin ed multi-objective machine learning

[close]

p. 3

studies in computational intelligence volume 16 editor-in-chief prof janusz kacprzyk systems research institute polish academy of sciences ul newelska 6 01-447 warsaw poland e-mail kacprzyk@ibspan.waw.pl further volumes of this series can be found on our homepage springer.com vol 1 tetsuya hoya artificial mind system ­ kernel memory approach 2005 isbn 3-540-26072-2 vol 2 saman k halgamuge lipo wang eds computational intelligence for modelling and prediction 2005 isbn 3-540-26071-4 vol 3 bozena kostek perception-based data processing in acoustics 2005 isbn 3-540-25729-2 vol 4 saman k halgamuge lipo wang eds classification and clustering for knowledge discovery 2005 isbn 3-540-26073-0 vol 5 da ruan guoqing chen etienne e kerre geert wets eds intelligent data mining 2005 isbn 3-540-26256-3 vol 6 tsau young lin setsuo ohsuga churn-jung liau xiaohua hu shusaku tsumoto eds foundations of data mining and knowledge discovery 2005 isbn 3-540-26257-1 vol 7 bruno apolloni ashish ghosh ferda alpaslan lakhmi c jain srikanta patnaik eds machine learning and robot perception 2005 isbn 3-540-26549-x vol 8 srikanta patnaik lakhmi c jain spyros g tzafestas germano resconi amit konar eds innovations in robot mobility and control 2005 isbn 3-540-26892-8 vol 9 tsau young lin setsuo ohsuga churn-jung liau xiaohua hu eds foundations and novel approaches in data mining 2005 isbn 3-540-28315-3 vol 10 andrzej p wierzbicki yoshiteru nakamori creative space 2005 isbn 3-540-28458-3 vol 11 antoni ligêza logical foundations for rule-based systems 2006 isbn 3-540-29117-2 vol 13 nadia nedjah ajith abraham luiza de macedo mourelle eds genetic systems programming 2006 isbn 3-540-29849-5 vol 14 spiros sirmakessis ed adaptive and personalized semantic web 2006 isbn 3-540-30605-6 vol 15 lei zhi chen sing kiong nguang xiao dong chen modelling and optimization of biotechnological processes 2006 isbn 3-540-30634-x vol 16 yaochu jin ed multi-objective machine learning 2006 isbn 3-540-30676-5

[close]

p. 4

yaochu jin ed multi-objective machine learning abc

[close]

p. 5

dr yaochu jin honda research institute europe gmbh carl-legien.str 30 63073 offenbach germany e-mail yaochu.jin@honda-ri.de library of congress control number 2005937505 issn print edition 1860-949x issn electronic edition 1860-9503 isbn-10 3-540-30676-5 springer berlin heidelberg new york isbn-13 978-3-540-30676-4 springer berlin heidelberg new york this work is subject to copyright all rights are reserved whether the whole or part of the material is concerned specifically the rights of translation reprinting reuse of illustrations recitation broadcasting reproduction on microfilm or in any other way and storage in data banks duplication of this publication or parts thereof is permitted only under the provisions of the german copyright law of september 9 1965 in its current version and permission for use must always be obtained from springer violations are liable for prosecution under the german copyright law springer is a part of springer science+business media springer.com c springer-verlag berlin heidelberg 2006 printed in the netherlands the use of general descriptive names registered names trademarks etc in this publication does not imply even in the absence of a specific statement that such names are exempt from the relevant protective laws and regulations and therefore free for general use a typesetting by the authors and techbooks using a springer l tex macro package printed on acid-free paper spin 11399346 89/techbooks 543210

[close]

p. 6

to fanhong robert and zewei

[close]

p. 7

preface feature selection and model selection are two major elements in machine learning both feature selection and model selection are inherently multi-objective optimization problems where more than one objective has to be optimized for example in feature selection minimization of the number of features and maximization of feature quality are two common objectives that are likely conflicting with each other it is also widely realized that one has to deal with the trade-off between approximation or classification accuracy and model complexity in model selection traditional machine learning algorithms try to satisfy multiple objectives by combining the objectives into a scalar cost function a good example is the training of neural networks where the main target is to minimize a cost function that accounts for the approximation or classification error on given training data however reducing the training error often leads to overfitting which means that the error on unseen data will become very large though the neural network performs perfectly on the training data to improve the generalization capability of neural networks i.e to improve their ability to perform well on unseen data a regularization term e.g the complexity of neural networks weighted by a hyper-parameter regularization coefficient has to be included in the cost function one major challenge to implement the regularization technique is how to choose the regularization coefficient appropriately which is non-trivial for most machine learning problems several other examples exist in machine and human learning where a tradeoff between conflicting objectives has to be taken into account for example in object recognition a learning system should learn as many details of a given object as possible on the other hand it should also be able to abstract the general features of different objects another example is the stability and plasticity dilemma where the learning systems have to trade off between learning new information without forgetting old information from the viewpoint of multi-objective optimization there is no single learning model that can satisfy different objectives at the same time in this sense pareto-based multiobjective optimization is the only way to deal with the conflicting objectives

[close]

p. 8

viii preface in machine learning however this seemingly straightforward idea has not been implemented in machine learning until late 1990 s liu and kadirkamanathan [1 considered three criteria in designing neural networks for system identification they used a genetic algorithm to minimize the maximum of the three normalized objectives similar work has been presented in [2 kottathra and attikiouzel [3 employed a branch-and-bound search algorithm to determine the structure of neural networks by trading off the mean square error and the number of hidden nodes the trade-off between sum of squared error and the norm of weights of neural networks was reported in [4 whereas the trade-off between training error and test error has been considered in [5 an interesting work on pareto-based neural network learning is reported by kupinski and anastasio [6 where a multi-objective genetic algorithm is implemented to generate the receiver operating characteristics curve of neural network classifiers in generating fuzzy systems ishibuchi et al [7 used a multi-objective genetic algorithm to minimize the classification error and the number of rules gomez-skarmeta et al suggested the idea of using paretobased multi-objective genetic algorithms to optimize multiple objectives in fuzzy modeling [8 though no simulation results were provided a genetic algorithm is used to minimize approximation error complexity sensitivity to noise and continuity of rules [9 with the boom of the research on evolutionary multi-objective optimization pareto-optimality based multi-objective machine learning has gained new impetus compared to the early works not only more sophisticated multiobjective algorithms are used but new research areas are also being opened up for example it is found that one is able to determine the number of clusters by analyzing the shape of the pareto front obtained by a multi-objective clustering method [10 see chapter 2 for more detailed results in [11 it is found that interpretable rules can be extracted from the simple paretooptimal neural networks further research reveals that neural networks with good generalization capability can also be identified by analyzing the pareto front as reported in chapter 13 in this book our most recent work suggests that multi-objective machine learning provides an efficient approach to addressing catastrophic forgetting [12 besides pareto-based multi-objective learning has shown particularly powerful in generating ensembles support vector machines svms and interpretable fuzzy systems this edited book presents a collection of most representative research work on multi-objective machine learning the book is structured into five parts part i discusses multi-objective feature extraction and selection such as rough set based feature selection clustering and cluster validation supervised and unsupervised feature selection for ensemble based handwritten digit recognition and edge detection in image processing in the second part multiobjective model selection is presented for improving the performance of single objective learning algorithms in generating various machine learning models including linear and nonlinear regression models multi-layer perceptrons mlps radial-basis-function networks rbfns support vector machines

[close]

p. 9

preface ix svms decision trees and learning classifier systems multi-objective model selection for creating interpretable models is described in part iii generating interpretable learning models plays an important rule in data mining and knowledge extraction where the preference is put on gaining insights into unknown systems from the work presented the reader can see that how understandable symbolic or fuzzy rules can be extracted from trained neural networks or from data directly within the framework of multi-objective optimization the merit of multi-objective optimization is fully demonstrated in part iv where techniques for generating ensembles of machine learning models are concerned diverse member of neural networks or fuzzy systems can be generated by trading off between training and test errors between accuracy and complexity or between accuracy and diversity compared to single objective based ensemble generation methods diversity is imposed more explicitly in multi-objective learning so that structural or functional diversity of ensemble members can be guaranteed to conclude the book part v presents a number of successful applications of multi-objective machine learning such as multi-class receiver operating curve analysis mobile robot navigation docking maneuver of automated ground vehicles information retrieval and object detection i am confident that by reading this book the reader is able to bring home a complete view of the emerging research area and to gain hands-on experience of a variety of multi-objective machine learning approaches furthermore i do hope that this book which is the first book dedicated to multi-objective machine learning to the best of my knowledge will inspire more creative ideas to further promote the research in this research area i would like to thank all contributors who prepared excellent chapters for this book many thanks go to prof janusz kacprzyk for his interest in this book i would also like to thank dr thomas ditzinger and ms heather king at springer for their kind assistance i am most grateful to mr tomohiko kawanabe prof dr edgar k¨rner dr bernhard sendhoff and mr andreas o richter at the honda research institute europe for their full understanding and kind support offenbach am main october 2005 yaochu jin references [1 g.p liu and v kadirkamanathan learning with multi-objective criteria in iee conference on artificial neural networks pages 53­58 1995 [2 s park d nam and c h park design of a neural controller using multiobjective optimization for nonminimum phase systems in proceedings of 1999 ieee int conf on fuzzy systems volume i pages 533­537 1999.

[close]

p. 10

x preface [3 k kottathra and y attikiouzel a novel multicriteria optimization algorithm for the structure determination of multilayer feedforward neural networks journal of network and computer applications 19:135­147 1996 [4 r de a teixeira a.p braga r h.c takahashi and r.r saldanha improving generalization of mlp with multi-objective optimization neurocomputing 35:189­194 2000 [5 h.a abbass a memetic pareto approach to artificial neural networks in proceedings of the 14th australian joint conference on artificial intelligence pages 1­12 2001 [6 m.a kupinski and m anastasio multiobjective genetic optimization of diagnostic classifiers with implementations for generating receiver operating characteristic curves ieee transactions on medical imaging 188 675­685 1999 [7 h ishibuchi t murata and b turksen single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems fuzzy sets and systems 89:135­150 1997 [8 a gomez-skameta f jimenez and j ibanez pareto-optimality in fuzzy modeling in proc of the 6th european congress on intelligent techniques and soft computing pages 694­700 1998 [9 t suzuhi t furuhashi s matsushita and h tsutsui efficient fuzzy modeling under multiple criteria by using genetic algorithm in ieee conf on systems man and cybernetics volume 5 pages 314­319 1999 [10 j handl and j knowles exploiting the trade-off ­ the benefits of multiple objectives in data clustering in evolutionary multi-criteria optimization lncs 3410 pages 547­560 springer 2005 [11 y jin b sendhoff and e k¨rner evolutionary multi-objective optimizao tion for simultaneous generation of signal-type and symbol-type representations in evolutionary multi-criteria optimization lncs 3410 pages 752­766 springer 2005 [12 y jin b sendhoff and e k¨rner avoiding catastrophic forgetting using o pareto-based multi-objective optimization connection science 2005 in preparation.

[close]

p. 11

contents part i multi-objective clustering feature extraction and feature selection 1 feature selection using rough sets mohua banerjee sushmita mitra ashish anand 3 2 multi-objective clustering and cluster validation julia handl joshua knowles 21 3 feature selection for ensembles using the multi-objective optimization approach luiz s oliveira marisa morita robert sabourin 49 4 feature extraction using multi-objective genetic programming yang zhang peter i rockett 75 part ii multi-objective learning for accuracy improvement 5 regression error characteristic optimisation of non-linear models jonathan e fieldsend 103 6 regularization for parameter identification using multi-objective optimization tomonari furukawa chen jian ken lee john g michopoulos 125 7 multi-objective algorithms for neural networks learning ant^nio p´dua braga ricardo h c takahashi marcelo azevedo o a costa roselito de albuquerque teixeira 151

[close]

p. 12

xii contents 8 generating support vector machines using multi-objective optimization and goal programming hirotaka nakayama yeboon yun 173 9 multi-objective optimization of support vector machines thorsten suttorp christian igel 199 10 multi-objective evolutionary algorithm for radial basis function neural network design gary g yen 221 11 minimizing structural risk on decision tree classification daeeun kim 241 12 multi-objective learning classifier systems ester bernad´-mansilla xavier llor ivan traus 261 o a part iii multi-objective learning for interpretability improvement 13 simultaneous generation of accurate and interpretable neural network classifiers yaochu jin bernhard sendhoff edgar k¨rner 291 o 14 ga-based pareto optimization for rule extraction from neural networks urszula markowska-kaczmar krystyna mularczyk 313 15 agent based multi-objective approach to generating interpretable fuzzy systems hanli wang sam kwong yaochu jin and chi-ho tsang 339 16 multi-objective evolutionary algorithm for temporal linguistic rule extraction gary g yen 365 17 multiple objective learning for constructing interpretable takagi-sugeno fuzzy model shang-ming zhou john q gan 385 part iv multi-objective ensemble generation 18 pareto-optimal approaches to neuro-ensemble learning hussein abbass 407

[close]

p. 13

contents xiii 19 trade-off between diversity and accuracy in ensemble generation arjun chandra huanhuan chen xin yao 429 20 cooperative coevolution of neural networks and ensembles of neural networks nicol´s garc´ a ia-pedrajas 465 21 multi-objective structure selection for rbf networks and its application to nonlinear system identification toshiharu hatanaka nobuhiko kondo katsuji uosaki 491 22 fuzzy ensemble design through multi-objective fuzzy rule selection hisao ishibuchi yusuke nojima 507 part v applications of multi-objective machine learning 23 multi-objective optimisation for receiver operating characteristic analysis richard m everson jonathan e fieldsend 533 24 multi-objective design of neuro-fuzzy controllers for robot behavior coordination naoyuki kubota 557 25 fuzzy tuning for the docking maneuver controller of an automated guided vehicle j.m lucas h martinez f jimenez 585 26 a multi-objective genetic algorithm for learning linguistic persistent queries in text retrieval environments mar´ luque oscar cord´n enrique herrera-viedma 601 ia o 27 multi-objective neural network optimization for visual object detection stefan roth alexander gepperth christian igel 629 index 657

[close]

p. 14

part i multi-objective clustering feature extraction and feature selection

[close]

p. 15

1 feature selection using rough sets mohua banerjee1 sushmita mitra2 and ashish anand1 1 2 indian institute of technology kanpur india {mohua,aanand iitk.ac.in indian statistical institute kolkata india sushmita@isical.ac.in summary feature selection refers to the selection of input attributes that are most predictive of a given outcome this is a problem encountered in many areas such as machine learning signal processing and recently bioinformatics/computational biology feature selection is one of the most important and challenging tasks when it comes to dealing with large datasets with tens or hundreds of thousands of variables areas of web-mining and gene expression array analysis provide examples where selection of interesting and useful features determines the performance of subsequent analysis the intrinsic nature of noise uncertainty incompleteness of data makes extraction of hidden and useful information very difficult capability of handling imprecision inexactness and noise has attracted researchers to use rough sets for feature selection this article provides an overview on recent literature in this direction 1.1 introduction feature selection techniques aim at reducing the number of irrelevant and redundant variables in the dataset unlike other dimensionality reduction methods feature selection preserves the original features after reduction and selection benefit of feature selection is many fold it improves subsequent analysis by removing the noisy data and outliers makes faster and more costeffective post-analysis makes data visualization easier and provides a better understanding of the underlying process that generated the data here we will consider an example which will serve us as an illustration throughout the chapter consider gene selection from microarray data in this problem the features are expression levels of genes corresponding to the abundance of mrna in a sample e.g particular time point of development or treatment for a number of patients and replicates a typical analysis task is to find genes which are differentially expressed in different cases or can classify different classes with high accuracy usually very few data samples are available altogether for testing and training but the number of features genes ranges from 10,000 to 15,000 m banerjee et al feature selection using rough setsx studies in computational intelligence sci 16 3­20 2006 c springer-verlag berlin heidelberg 2006 www.springerlink.com

[close]