Recursive LSE in the Number of Parameters

Before delving into the problem of recrusive LSE in the number of parameters, we need to know the lemma that expresses the matrix inverse in a block form:

$$\begin{array} {rcl} \left[ \begin{array} {cc} {\bf A}& {\bf ... ...bf B}^T{\bf A}^{-1} & {\bf K}^{-1}\end{array}\right]\end{array}$$

where ${\bf K}= {\bf C}-{\bf B}^T{\bf A}^{-1}{\bf B}$.

A set of over-determined linear equations can be expressed as

$${\bf A}\mbox{\boldmath$\theta$}= {\bf y}.$$

The LSE (least-squares estimator) to the above question is

$$\hat{\mbox{\boldmath$\theta$}} = ({\bf A}^T{\bf A})^{-1}{\bf A}^T{\bf y}.$$

When extra parameters are introduced, the vector $\hat{\mbox{\boldmath$\theta$}}$ will have more components and the matrix ${\bf A}$ will have additional columns. We shall derive a recursive LSE formula in the number of parameters. The new set of over-determined linear equations can be expressed as

$$[{\bf A}\; {\bf B}] \left[ \begin{array} {c} \mbox{\boldmath$... ...a$}_A\\ \mbox{\boldmath$\theta$}_B\end{array}\right] = {\bf y}.$$

where $\mbox{\boldmath$\theta$}_B$ is a vector of newly added parameters and ${\bf B}$ is corresponding additional columns. The corresponding LSE can be expressed as

$$\begin{array} {rcl} \hat{\mbox{\boldmath$\theta$}}_{new} & =... ...{\mbox{\boldmath$\theta$}}) \end{array} \right],\\ \end{array}$$

where ${\bf K}= {\bf B}^T{\bf B}- {\bf B}^T{\bf A}({\bf A}^T{\bf A})^{-1}{\bf A}^T{\bf B} = {\bf B}^T({\bf I}- {\bf A}({\bf A}^T{\bf A})^{-1}{\bf A}^T){\bf B}$.(Note that if ${\bf B}= {\bf a}$ is a column vector, then ${\bf K}\geq 0$ and is equal to the error measure of fitting ${\bf A}\mbox{\boldmath$\theta$}={\bf a}$.)

$$\begin{array} {rcl} E_{new} & = & {\bf y}^T \left({\bf y}-... ...{\bf A}\hat{\mbox{\boldmath$\theta$}})\\ & \leq & E\end{array}$$

(Note that ${\bf K}^{-1}$ is symmetric.)