Incremental Formula for LSE Error Measures

Let E_k and E_k+1 be the optimal error measures of the least-squares problems involving k and k+1input-output data pairs, respectively. In symbol:

$$E_k = \Vert{\bf y}-{\bf A}\mbox{\boldmath$\theta$ }_k\Vert^2 = {\bf y}^T ({\bf y}-{\bf A}\mbox{\boldmath$\theta$ }_k)$$

$$E_{k+1} = \left\Vert \left[ \begin{array}{c} {\bf y}\\ y \en... ...^T \end{array} \right] \mbox{\boldmath$\theta$ }_{k+1} \right)$$

(Note that for simplicity, we have left out the $\tilde{\;}$ for the least-squares estimator $\mbox{\boldmath$\theta$ }_k$ and $\mbox{\boldmath$\theta$ }_{k+1}$.) Our mission is to find an incremental formula between E_k and E_k+1, as follows:

$$\begin{array}{rcl} E_{k+1} & = & \left[ \begin{array}{c} {\bf... ...}_k)(y-{\bf a}^T\mbox{\boldmath$\theta$ }_{k+1})\\ \end{array}$$ (1)

This concludes our derivation.

Note that in the above derivation, we have used a formula

$$\mbox{\boldmath$\theta$ }_{k+1}-\mbox{\boldmath$\theta$ }_{k}={\bf P}_k{\bf a}(y-{\bf a}^T\mbox{\boldmath$\theta$ }_{k+1}),$$

which can be derived as a side note in the following:

$$\begin{array}{rcl} \mbox{\boldmath$\theta$ }_k & = & {\bf P}_... ...{\bf a}(y-{\bf a}^T\mbox{\boldmath$\theta$ }_{k+1}) \end{array}$$