Information theory is the basis of Gaussian adaptation

December 30, 2007

According to Kjellström, 1969, (see reference in the list
http://en.wikipedia.org/wiki/Gaussian_adaptation#References)
a connection between Gaussian adaptation, GA, and information theory is the average speed of stepwise random walks inside a high dimensional simplex region. It turns out that the speed is asymptotically proportional to (see also the point 7 in blog “Gaussian adaptation as a model of evolution”)
- P log(P),
where P is the probability that a random step will lead to a new feasible position inside the simplex. Maximum speed is obtained when P = 1/e = 0.37.

A plausible interpretation of this is that 1/P is proportional to the time/work needed to find a step leading to a feasible position, while –log(P) is the self-information obtained when such a step may be taken. Thus, – P log(P) may be seen as a measure of efficiency; information divided by the work/time needed to get the information.

More generally, Kjellström, 1991, shows that the same definition of efficiency is valid for a large class of measures fulfilling some postulates below. These postulates seem to limit the applicability of the theorem, but in combination with GA the postulates may be approximately fulfilled also when the processes are not statistically independent in different parameters, because GA may make linear transformations of the whole process.

Postulate 1. P is the average hitting probability on some region of acceptability, A or s(x). The time/work needed to test a point x for membership in A is equal for all points.

Postulate 2. The process is statistically independent and separable in the n different parameters and equally efficient in all parameters, i. e. P = p^n (raised to n), where p is the average hitting probability as far as one parameter is concerned.

postulate 3. The efficiency E is a continuous function of P in the interval 0 £ P £ q £ 1, (where q > 0 is the maximum attainable P-value) and is positive in the P-interval, except at the end points where E = 0. The derivative [dE(P)/dP] at P=q is limited negative, i. e. -infinity < [dE(P)/dP] at P=q < 0.
Theorem. All measures of efficiency, that satisfy the postulates above, are asymptotically proportional to – P log(P/q) when n increases infinitely. The maximum value of E is obtained with P = q/e. And in the special case when q = 1 we have P = 1/e = 0.37.
Similar results have been reported by Gaines, 1997, and Taxén, 2003.

According to point 6 in the blog “Gaussian adaptation as a model of evolution”, another connection between GA and information theory is that GA may maximize the average information (disorder, diversity) of a Gaussian distribution keeping P constant. The necessary maximizing conditions are m* = m and M* proportional to M.

Permalink