A question concerning the perceptron training is how many iterations over the training examples gives the best output accuracy.
Some criteria can be used. For example, the number of iterations can be fixed from the development tests. Alternatively, a set of examples can be taken out of the training data from the test set, and using those for testing and the rest for training, the number of iterations determined by the corresponding observations. Still an alternative is to check the training error. When the training error is below a certain proportion of the training data, the learning process stops. The last method assumes that the perceptron model converges.
I have seen some example where the first method is far from optimal. This happens in a case where the size difference between the development data and the test data is huge. It seems that the second method is a comparatively sensible choice.
Subscribe to:
Post Comments (Atom)
3 comments:
One possibility is to use 3 data partitions: train, validation ("tuning") and testing:
1. Train the perceptron on the training data for many iterations.
2. Track the validation performance at each training iteration.
3. Select the number of iterations which corresponds to the optimal validation performance.
4. Construct the final model by training a new perceptron on the combined training and validation data. (Discard the original perceptron.)
5. Estimate the true performance using the testing data.
-Will Dwinnell
Data Mining in MATLAB
I have done some experiments to compare the methods.
Method 1 and method 3 in the original post require a separate set of development data, but are much easier to do than method 2. However, method 2 generally performs better than both method 1 and method 3. Method 3 gave higher accuracy than method 1 in my experiments.
I also did some different splittings for method 2. For example, the first 10% of the training data, or the first of every 10 sentences in the training data, were as the development test data. It turned out that the splitting did not affect the result.
To Will Dwinnell:
Many thanks for commenting. This method is essentially the same as the method 2 from my original post, though it is much better formulated!
Oh I must mention that method 2 has been used in some publications, such as
Projective Dependency Parsing with Perceptron, Xavier Carreras, Mihai Surdeanu and Lluís Màrquez, Shared Task system, in Proceedings of CoNLL-X, New York 2006.
Post a Comment