Now that you have trained a large number of submodels and identified the best ones, you are going to improve further the predictions of your model by using a committee of submodels, rather than just one submodel.
The idea is to calculate the combined test error for a committee if models, and then find which combination of models gives the minimum TE.
To do this, we consider a given number of submodels, calculate their predictions on the second half of the database (the unseen data), and calculate the combined test error.So, how do we choose which combination we want to try ?
The most straighforward method is to use the submodels as sorted by LPE. We
then build different committees by adding each time the next model in the
decreasing LPE list. This is what the Model Manager will do if you do not touch
the committee list.
However, the Model Manager gives you more freedom, and you can actually build any committee you want. This can be usefull in some cases, when two submodels have very similar LPE but different TE.
When you first click on "Select Committee", you are
presented with the following screen:
You need here to select how many (that is, the maximum) models you want to try to include in your committee. Make sure you do not select to few, as you can not modify this figure later.
The Model Manager is then going to calculate the predicitions on the whole database, for all the n first models, where n is the number you just selected.Foreground or Background
Background: If you select a large number (50 ?) of models here, the calculation time can be quite long. You might want to log out and go to have lunch.. choose background in this case.
Foreground: If you think that it is going to be quick, or you don't mind waiting a few minutes, select this option. The window will display the progress made in the calculation of the predictions.
For the moment, there is no message from the Model Manager when the calculations are finished. However, if you click again on "Select Committee" before the background calculation is finished, you will be asked to wait more.
Now that the pre-calculation are done, we can search the best combination of submodels. If everything goes right, the next time you click on "Select Committee", you should obtain the following screen: you can use this image to find out what the different functions are, or directly read below.
Clicking on any submodel present in this window will add it to the list of submodels to build the committee.
This is the list of all the submodels that will be included in
the calculation. The committee which are build are as many as there are
the first contains only the first submodel, the second committee contains
the two first submodels etc..
For example, in the case of this calculation, the image on the left shows that the best committee is that constituted by the 6 first submodels of the list shown in the "committee" window.
To remove a submodel from this list, simply click on the submodel line
Use this to switch the sorting method from decreasing LPE to increasing TE.
Use this button to clear the list of submodels present in the committee window.
Use this to reset the committee window to its initial state. If you have switched to TE, the new default list is sorted by increasing TE.
This will start the calculation of the combined test error for all the committees built with the first, first and second, etc submodels.
If you are satisfied with the proposed committee
(here, the 6 first model of the list), click on this button.
You can then leave, but you can also modify the committee again. Always remember to validate at the end.
You can save the graph for your committee selection as a postscript or GIF file (on some version, GIF is not available). When you do so, a text file containing the list of models corresponding to your graph is also saved. If you saved many graphs, they are followed by a number which differenciate them.
Model Manager Tutorial
Copyright Neuromat Ltd, All rights reserved
Author: Thomas Sourmail, Neuromat Ltd
|Back to main||Top|