Approximation via Subspaces: Analysis and Geometry

If such a system of o.n. basis vectors has been obtained, it is natural to ask: Does the set of such orthonormal elements constitute a basis for the whole space ${\cal H}$ ? In other words, for any $f\in{\cal H}$ , can one, in some sense, claim that

$$f=c_1u_1+c_2u_2+\cdots~~\qquad~~\textrm{(Generalized~Fourier~series)}$$

for an appropriate choice of $c_i$ ? This is an existence question. Furthermore, given that such constants have been constructed, are they unique?

For a finite dimensional vector space these questions have an implicit affirmative answer because of the finiteness of the dimension. However, for an infinite dimensional vector space there is cause to worry about existence. The vector $f$ might point into a direction which is so peculiar that not even the given infinite number of basis vectors is sufficient to give a 100% accurate representation of $f$ in terms of these vectors. There is a sense in which this worry is justified if the vector is a discontinuous function. This fact is highlighted by the Fourier theorem in the next chapter. However, there is another sense in which the representation is always 100% accurate, with the result that the answer to the above questions is in the affirmative.

A significant step towards establishing the geometrical framework for formulating the existence and uniqueness questions is obtained by the method of least squares approximation via finite dimensional subspaces. It is expressed by the following theorem.

Theorem 15.1 (Least squares approximation via subspaces)

Given an o.n. system

$$u_1,u_2,\dots,u_k,\dots$$

in the Hilbert space ${\cal H}$ , let $f$ be an arbitrary element of ${\cal H}$ . Then (i)the expression

$$\left\Vert f-\sum^N_{k=1} a_k u_k\right\Vert^2\equiv E^2_N (a_1,\dots ,a_N)$$

has a minimum, for

$$a_k=\langle u_k,f\rangle \equiv c_k~\quad k=1,\cdots,N~.$$

(ii)This minimum equals

$$\Vert f\Vert^2-\sum^N_{k=1}\vert c_k\vert^2\,.$$

(iii) Moreover,

$$\sum^\infty_{k=1}\vert c_k\vert^2\le \Vert f\Vert^2\,,$$

a result known as Bessel's inequality.

Comment:

1. The coefficients
   $$c_k=\langle u_k,f\rangle$$
   are called the (generalized) Fourier coefficients.
2. The function $E^2_N(a_1,\dots ,a_N)$ is called Gauss's mean squared error. Minimizing it by setting
   $$\frac{\partial E^2_N}{\partial \overline{a}_k} = 0~~\qquad~~\qquad~~k= 1,\dots ,N$$
   yields the $N$ Fourier coefficients
   $$a_k=\langle u_k,f\rangle \equiv c_k ~~\qquad~~\qquad~~k=1,\dots ,N$$
   as the solution to this equation (try it!). The word ``mean'' in Gauss's mean squared error arises from its defining property,
   $$E^2_N=\int_a^b \vert f(x)-\sum^N_{k=1}a_k u_k(x) \vert^2 \rho(x)\,dx ~.$$
   The integrand $\vert f(x)-\sum^N_{k=1}a_k u_k(x) \vert^2$ is the error at $x$ , while the integral is $(b-a)$ times the (weighted) ``mean'' of this quantity, in compliance with the mean value theorem of integral calculus.
3. The set of linear combinations
   $$\textrm{span}\{ u_1,\dots ,u_N\}\equiv W_N\subset{\cal H}$$
   is a subspace of ${\cal H}$ , and
   $$\sum^N_{k=1} c_k u_k\equiv w^\ast _N$$
   is the orthogonal projection of $f$ onto $W_N$ .

Proof: The Gaussian mean squared error function

$$\langle f-\sum^N_{k=1} a_k u_k\,,~f-\sum^N_{\ell =1} a_\ell u_\ell\rangle\equiv E^2_N$$

is a quadratic expression in the complex unknowns $a_k$ . As usual, in such expressions completing the square will yield the minimum value at a glance. Multiplying out the inner product yields

$$E^2_n = \Vert f\Vert^2-\sum^N_{k=1} \overline{a}_k\langle u_k,f\r... ...sum^N_{k=1}\sum^N_{\ell =1}\overline{a}_k a_\ell \langle u_k,u_\ell\rangle\,.$$

Introducing the Fourier coefficients $c_k$ and using the orthonormality of the $u_k$ 's yields, upon adding and subtracting $\sum\limits_k \vert c_k\vert^2$ ,

$$\begin{array}{lclclclcl} E^2_n &= &\Vert f\Vert^2 &- &\sum^N... ...rt c_k\vert^2 &+ &\sum^N_{k=1}\vert a_k -c_k\vert^2 \end{array}$$

Conclusions:

1. The mean squared error is minimized by
   $$a_k = c_k\equiv \langle u_k,f\rangle\,,$$
   and its minimum is
   $$\textrm{Min}\Vert f-\sum^N_{k=1} a_k u_k\Vert^2 = \Vert f\Vert^2 -\sum^N_{k=1} \vert c_k\vert^2\,.$$
   For this reason one says that $\sum_{k=1}^N u_k\langle u_k,f\rangle$ is the best approximation of $f$ in the subspace $W_N$ , or more precisely, $\sum_{k=1}^N u_k\langle u_k,f\rangle$ is the $W_N$ -induced least squares approximation of $f$ .
2. Furthermore,
   $$0\le \Vert f-\sum^N_{k=1} a_ku_k\Vert^2$$
   implies
   $$\sum^N_{k=1}\vert c_k\vert^2 \le \Vert f\Vert^2\,.$$
   This is Bessel's inequality, which is true for all integers $N$ , and which evidently is equivalent to
   $$\sum^\infty_{k=1}\vert c_k\vert^2 \le \Vert f\Vert^2\,.$$

   
   

   Lecture 7

   
   

3. If instead of an inequality, one has the equality
   $$\sum^\infty_{k=1}\vert c_k\vert^2\equiv \sum^\infty_{k=1}\langle f,u_k\rangle \langle u_k,f\rangle =\Vert f\Vert^2\,,~\quad~\forall~f\in{\cal H}$$
   then it is known as Parseval's relation or the completeness relation for the set of orthonormal basis elements.
4. This theorem also has the following geometrical interpretation: the set of linear combinations
   $$\textrm{span}\,\{ u_1,\dots ,u_N\}\equiv W_N\subset {\cal H}$$
   is a subspace of ${\cal H}$ , and the $N^{\textrm{th}}$ partial Fourier sum

   $$\sum^N_{k=1} c_k u_k\equiv w^\ast _N$$ (17)

   
   
   is the orthogonal projection of $f$ onto $W_N$ : It is linear, it is given by

   $$w^*_N=\sum_{k=1}^N u_k\langle u_k,f\rangle\equiv P_{W_N}f\quad (\in W_N)~,$$ (18)

   
   
   and it has the property that
   $$P_{W_N} P_{W_N}f=P_{W_N}f\quad \textrm{for all }f\in\mathcal{H}~.$$
   This expresses the fact that $P_{W_N}$ is the identity operator on $W_N$ , but that in light of Bessel;'s inequality $P_{W_N}$ shortens $f$ if $f\not\in W_N$ .

Figure 1.6: The $N$ -dimensional subspace $W_N$ of the ambient Hilbert space $\cal H$ . The least squares approximation $w^*_N$ is the orthogonal projection of the vector $f$ onto $W_N$ . The difference between the given vector $f$ and its projection onto the subspace is the error vector $h^*_N$ .

Comments:

1. Suppose we calculate the squared length of the difference $h^\ast_N$ between the given vector $f\in{\cal H}$ and its $N^{\textrm{th}}$ partial sum approximation $w^\ast_N(= P_{W_N}f)\in W_N$ . The vector $h^\ast_N$ is called the error vector. We find that
   

   $$\Vert h^\ast_N\Vert^2 = \langle f-\sum^N_{k=1} c_ku_k\,,~f- \sum^N_{\ell=1} c_\ell u_\ell\rangle$$

   $$= \Vert f\Vert^2 - \sum^N_{k=1}\vert c_k\vert^2~~.$$ (19)

   
   In other words, Bessel's inequality,
   $$0\le\Vert f\Vert^2-\sum^N_{k=1}\vert c_k\vert^2=\Vert h^\ast_N\Vert^2\equiv \Vert\textrm{error~vector}\Vert^2=\hbox{Min}~E^2_N~,$$
   is an expression of the fact that the minimum error has non-negative squared norm.
2. Suppose we calculate the inner product between the error vector $h^\ast_N$ and any one of the spanning vectors of the subspace $W_N$ . We find
   $$\langle u_\ell ,h^\ast_N\rangle =\langle u_\ell ,f-\sum^N_{k=1} c_k u_k\rangle =0~\quad~\ell =1,\dots ,N\,.$$
   In other words, $h^\ast_N$ is orthogonal to $W_N$ . In particular
   $$\langle w^\ast_N,h^\ast_N\rangle = 0\,.$$
   Consequently, $w^\ast_N$ and $h^\ast_N$ are the two sides of a right triangle whose hypotenuse is $f$ .
3. Consider the fact that the norm of the $N^{\textrm{th}}$ partial Fourier sum, Eq.(1.7), is
   $$\Vert w^\ast_N\Vert^2 = \sum^N_{k=1}\vert c_k\vert^2\,.$$
   This is Pythagoras's theorem in the subspace $W_N$ . Combine this with Eq. (1.9) and obtain
   $$\Vert h_N^\ast\Vert^2+\Vert w^\ast_N\Vert^2=\Vert f\Vert^2\,,$$
   which is Pythagoras's theorem applied to the right triangle $(h_N^\ast ,w^\ast_N, f)$ .