Maximum entropy spectrum

The entropy of a probability distribution function p(x) with xin mathbb{R}^n,

 int -p(x) log(p(x)) dx,

quantifies the uncertainty of the corresponding random variable. When p(x) is a multi-dimensional Gaussian distribution with zero mean and covariance matrix M, the entropy becomes

 frac{1}{2}log |M|+frac{n}{2}-frac{n}{2}log (2pi).

For a zero-mean stationary Gaussian process {y(t), tin mathbb{Z}}, corresponding to infinite-sized Toeplitz structured covariance, the entropy “diverges”. Thus, one uses instead the ‘‘entropy rate". Let T_{N-1} be a N dimensional principle sub-matrix of the covariance matrix

 T_{N-1}=left[           begin{array}{cccc}             r(0) & r(1) & ldots & r(N-1)              r(-1)  & r(0) & ldots & r(N-2)              vdots & vdots & ddots & cdots              r(-N+1) & r(-N+2) & r(-1) & r(0)            end{array}         right],

Then the entropy rate is

 lim_{Nrightarrow infty} frac{1}{N} left( frac{1}{2}log |T_{N-1}|+frac{N}{2}-frac{N}{2}log (2pi)  right)= lim_{Nrightarrow infty} frac{1}{2} log |T_{N-1}|^{1/N}+frac{1}{2}-frac{1}{2}log (2pi).

The limit of |T_{N-1}|^{1/N} converges to the optimal one-step prediction error variance, and from the Szegddot{o}-Kolmogorov formula

 lim_{Nrightarrow infty} |T_{N-1}|^{1/N} = frac{1}{2pi} e^{ int_{-pi}^{pi}  f(theta)frac{dtheta}{2pi}}.

Given state statistics, the maximum entropy power spectrum is

 f_{rm me}(theta)=argmax_{f(theta)} left{ int_{-pi}^{pi} log f(theta) dtheta ~mid~ int_{-pi}^{pi} G(e^{itheta})  f(theta)G(e^{itheta})^*dtheta = R right}.

The solution to this problem (see [2]) is

 f_{rm me}(theta)= P(e^{itheta})^{-1}Omega (P(e^{itheta})^{-1})^*


 begin{array}{rl}Gamma:=& (B^*R^{-1} B)^{-1} B^* R^{-1} Omega:=& Gamma RGamma^* P(z):=& Gamma G(z) end{array}

and G(z) is the input-to-state filter defined above. Then the maximum entropy spectrum is precisely the solution given above.

This formula subsumes the classical Burg method/AR modeling where the R is a Toeplitz matrix and G(z) is lag-delay filter bank.

The maximum entropy spectrum is obtained using the routine me.m. For the example discussed above, the maximum entropy spectrum is shown in blue. There are two peaks detected inside the window. Burg's spectrum is shown in green. The resolution of Burg's solution is not sufficient to distinguish the two peaks.

% maximum entropy spectrum and Burg spectrum
hold on;