Selecting your committee

1. What it is about.
2. The first time.
3. Building your committee
4. I am in a hurry..

What it is about:  Top

Now that you have trained a large number of submodels and identified the best ones, you are going to improve further the predictions of your model by using a committee of submodels, rather than just one submodel.

The idea is to calculate the combined test error for a committee if models, and then find which combination of models gives the minimum TE.

To do this, we consider a given number of submodels, calculate their predictions on the second half of the database (the unseen data), and calculate the combined test error.

The most straighforward method is to use the submodels as sorted by LPE. We then build different committees by adding each time the next model in the decreasing LPE list. This is what the Model Manager will do if you do not touch the committee list.
However, the Model Manager gives you more freedom, and you can actually build any committee you want. This can be usefull in some cases, when two submodels have very similar LPE but different TE.

When you start  Top

When you first click on "Select Committee", you are presented with the following screen:

You need here to select how many (that is, the maximum) models you want to try to include in your committee. Make sure you do not select to few, as you can not modify this figure later.

The Model Manager is then going to calculate the predicitions on the whole database, for all the n first models, where n is the number you just selected.

• Background: If you select a large number (50 ?) of models here, the calculation time can be quite long. You might want to log out and go to have lunch.. choose background in this case.

• Foreground: If you think that it is going to be quick, or you don't mind waiting a few minutes, select this option. The window will display the progress made in the calculation of the predictions.

For the moment, there is no message from the Model Manager when the calculations are finished. However, if you click again on "Select Committee" before the background calculation is finished, you will be asked to wait more.

Selecting your committee  Top

Now that the pre-calculation are done, we can search the best combination of submodels. If everything goes right, the next time you click on "Select Committee", you should obtain the following screen: you can use this image to find out what the different functions are, or directly read below.

• Models available (by default, sorted by decreasing LPE):

  Clicking on any submodel present in this window will add it to the list of submodels to build the committee.

• Committee:

  This is the list of all the submodels that will be included in the calculation. The committee which are build are as many as there are submodels: the first contains only the first submodel, the second committee contains the two first submodels etc..
  For example, in the case of this calculation, the image on the left shows that the best committee is that constituted by the 6 first submodels of the list shown in the "committee" window.
  To remove a submodel from this list, simply click on the submodel line

• Switch LPE/TE:

  Use this to switch the sorting method from decreasing LPE to increasing TE.

• Delete all Committee:

  Use this button to clear the list of submodels present in the committee window.

• Reset Default Committee:

  Use this to reset the committee window to its initial state. If you have switched to TE, the new default list is sorted by increasing TE.

• Calculate Cumulated Test Error:

  This will start the calculation of the combined test error for all the committees built with the first, first and second, etc submodels.

• Validate:

  If you are satisfied with the proposed committee (here, the 6 first model of the list), click on this button.
  You can then leave, but you can also modify the committee again. Always remember to validate at the end.

• Saving:

  You can save the graph for your committee selection as a postscript or GIF file (on some version, GIF is not available). When you do so, a text file containing the list of models corresponding to your graph is also saved. If you saved many graphs, they are followed by a number which differenciate them.

In a hurry..  Top

• Click on "Calculate Cumulated Test Error"
• Click on validate
• Quit

Model Manager Tutorial Copyright Neuromat Ltd, All rights reserved Author: Thomas Sourmail, Neuromat Ltd

Back to main

Top