10.1. Progressive Validation Technique

Progressive validation is a technique which allows you to use almost 1 2 of the data in a holdout set for training purposes while still providing the same guarantee as the holdout bound. It first appeared in [3] and is discussed in a more refined and detailed form here.

Suppose that you have a training set of size mtrain and test set of size mpv. Progressive validation starts by first learning a hypothesis on the training set and then testing on the first example of the test set. Then, we train on training set plus the first example of the test set and test on the second example of the test set. The process continues mtest iterations. Let m abbreviate mpv. Then, we have m hypotheses, h1,...,hm and m error observations, ê1,...,êm. The hypothesis output by progressive validation is the randomized hypothesis which chooses uniformly from h1,...,hm and evaluates to get an estimated output. Note that this protocol is similar to those in [36] and the new thing here is an analysis of performance.

Since we are randomizing over hypotheses trained on mtrain to mtrain + mpv 1 examples, the expected number of examples used by any hypothesis is mtrain + mpv1 2 . Given that training can exhibit phase transitions, the extra few examples can greatly improve the accuracy of the trained example.

Viewed as an interactive proof of learning, the progressive validation technique follows the protocol of figure 10.1.1.


PIC

Figure 10.1.1: The progressive validation protocol has a learner repeatedly commit to a hypothesis before it is given a new example. Based upon the test errors, a bound on the true error rate of the metahypothesis which chooses randomly from each of (h1,...,hm) before each evaluation is provided.

The true error rate of this randomized hypothesis will be: epv = 1 mpv i=1mpve(h i) where e(hi) = PrD(hi(x)y) and the empirical error estimate of this randomized hypothesis will be: êpv = 1 mpv i=1mpvê i