“Assumption” is another word to be careful with in machine learning because it is used in several ways.
- Assumption = Bias There are several ways to see that some form of ‘bias’ (= preferring of one solution over another) is necessary. This is obvious in an adversarial setting. A good bit of work has been expended explaining this in other settings with “no free lunch” theorems. This is a usage specialized to learning which is particularly common when talking about priors for Bayesian Learning.
- Assumption = “if” of a theorem The assumptions are the ‘if’ part of the ‘if-then’ in a theorem. This is a fairly common usage.
- Assumption = Axiom The assumptions are the things that we assume are true, but which we cannot verify. Examples are “the IID assumption” or “my problem is a DNF on a small number of bits”. This is the usage which I prefer.
One difficulty with any use of the word “assumption” is that you often encounter “if assumption then conclusion so if not assumption then not conclusion“. This is incorrect logic. For example, with variant (1), “the assumption of my prior is not met so the algorithm will not learn”. Or, with variant (3), “the data is not IID, so my learning algorithm designed for IID data will not work”. In each of these cases “will” must be replaced with “may” for correctness.
On a related topic, check out some work on ‘chaotic’ learning robots using bias, it does not have to be an “assumption” to help, merely a weighting, or just an off-centre mass…
http://www.newscientist.com/article.ns?id=dn6582&print=true
[2nd thought]A bias or yearning for the perfect all-solving algorithm may exist, but the goal itself may not be achievable, however the bias itself might be useful in the way sexually selected traits are, like peacocks tails, the yearning for the platonic ideal selects better and better wide-solving algorithms sets, its a metabias about learning, its an epistemological position reified into the ontological (plato’s ideals at least), while others, think its all flux…