General parameterizations

Approximate solutions of the error minimization problem are obtained by restricting the search (trial) space for $h(x,y)$ = $\phi(x,y)$ (or $h(x)$ in regression). Functions $\phi$ which are in the considered search space are called trial functions. Solving a minimization problem in some restricted trial space is also called a variational approach [97,106,29,36,27]. Clearly, minimal values obtained by minimization within a trial space can only be larger or equal than the true minimal value, and from two variational approximations that with smaller error is the better one.

Alternatively, using parameterized functions $\phi$ can also be interpreted as implementing the a priori information that $\phi$ is known to have that specific parameterized form. (In cases where $\phi$ is only known to be approximately of a specific parameterized form, this should ideally be implemented using a prior with a parameterized template and the parameters be treated as hyperparameters as in Section 5.) The following discussion holds for both interpretations.

Any parameterization $\phi$ = $\phi (\{\xi_l\})$ together with a range of allowed values for the parameter vector $\xi$ defines a possible trial space. Hence we consider the error functional

$$E_{\phi (\xi)} = -(\,\ln P (\xi),\, N\,) +\frac{1}{2} (\,\phi (\xi),\, {{\bf K}}\,\phi (\xi)\,) + (\,P( \xi ),\,\Lambda_X\,),$$ (352)

for $\phi$ depending on parameters $\xi$ and $p(\xi )$ = $p(\,\phi (\xi)\,)$. In the special case of Gaussian regression this reads

$$E_{h (\xi)} = \frac{1}{2} (\,h (\xi)-t_D,\, {{\bf K}_D}\,h (\xi)-t_D\,) +\frac{1}{2} (\,h (\xi),\, {{\bf K}}\,h (\xi)\,) .$$ (353)

Defining the matrix

$$\Phi^\prime (l;x,y) = \frac{\partial \phi (x,y)}{\partial \xi_l}$$ (354)

the stationarity equation for the functional (352) becomes

$$0 = \Phi^\prime {\bf P}^\prime {\bf P}^{-1} N - \Phi^\prime {{\bf K}}\phi -\Phi^\prime {\bf P}^\prime \Lambda_X .$$ (355)

Similarly, a parameterized functional $E_\phi$ with non-zero template $t$ as in (226) would give

$$0 = \Phi^\prime {\bf P}^\prime {\bf P}^{-1} N - \Phi^\prime... ...left( \phi - t\right) - \Phi^\prime {\bf P}^\prime \Lambda_X .$$ (356)

To have a convenient notation when solving for $\Lambda_X$ we introduce

$${\bf P}^\prime_\xi = \Phi^\prime (\xi) {\bf P}^\prime(\phi),$$ (357)

i.e.,

$${\bf P}^\prime_\xi (l;x,y) = \frac{\partial P(x,y)}{\partia... ...i_l} \frac{\delta P(x,y)}{\delta \phi (x^\prime ,y^\prime )} ,$$ (358)

and

$$G_{\phi(\xi)} ={\bf P}^\prime_\xi {\bf P}^{-1} N - \Phi^\prime {{\bf K}}\phi ,$$ (359)

to obtain for Eq. (355)

$${\bf P}^\prime_\xi \Lambda_X = G_{\phi(\xi)} .$$ (360)

For a parameterization $\xi$ restricting the space of possible $P$ the matrix ${\bf P}^\prime_\xi$ is not square and cannot be inverted. Thus, let $({\bf P}^\prime_\xi)^{\char93 }$ be the Moore-Penrose inverse of ${\bf P}^\prime_\xi$, i.e.,

$$({\bf P}^\prime_\xi)^{\char93 } {\bf P}^\prime_\xi ({\bf P}... ...ar93 } {\bf P}^\prime_\xi = ({\bf P}^\prime_\xi)^{\char93 } ,$$ (361)

and symmetric $({\bf P}^\prime_\xi)^{\char93 } {\bf P}^\prime_\xi$ and ${\bf P}^\prime_\xi ({\bf P}^\prime_\xi)^{\char93 }$. A solution for $\Lambda_X$ exists if

$${\bf P}^\prime_\xi ({\bf P}^\prime_\xi)^{\char93 } G_{\phi(\xi)} =G_{\phi(\xi)} .$$ (362)

In that case the solution can be written

$$\Lambda_X = ({\bf P}^\prime_\xi)^{\char93 } G_{\phi(\xi)} +... ...({\bf P}^\prime_\xi)^{\char93 } {\bf P}^\prime_\xi V_\Lambda ,$$ (363)

with arbitrary vector $V_\Lambda$ and

$$\Lambda_X^0 = V_\Lambda - ({\bf P}^\prime_\xi)^{\char93 }{\bf P}^\prime_\xi V_\Lambda$$ (364)

from the right null space of ${\bf P}^\prime_\xi$, representing a solution of

$${\bf P}_\xi^\prime \Lambda_X^0 = 0 .$$ (365)

Inserting for $\Lambda_X(x) \ne 0$ Eq. (363) into the normalization condition $\Lambda_X$ = ${\bf I}_X {\bf P} \Lambda_X$ gives

$$\Lambda_X = {\bf I}_X {\bf P} \left( ({\bf P}^\prime_\xi)^{... ...^\prime_\xi)^{\char93 } {\bf P}^\prime_\xi V_\Lambda \right) .$$ (366)

Substituting back in Eq. (355) $\Lambda_X$ is eliminated yielding as stationarity equation

$$0 = \left( {\bf I} - {\bf P}^\prime_\xi {\bf I}_X {\bf P} ({... ...^\prime_\xi)^{\char93 } {\bf P}^\prime_\xi V_\Lambda \right) ,$$ (367)

where $G_{\phi(\xi)}$ has to fulfill Eq. (362). Eq. (367) may be written in a form similar to Eq. (193)

$${{\bf K}}_{\phi(\xi)}( \xi) = T_{\phi(\xi)}$$ (368)

with

$$T_{\phi(\xi)} (\xi) = {\bf P}_\xi^\prime {\bf P}^{-1}N -{\bf P}_\xi^\prime \Lambda_X ,$$ (369)

but with

$${{\bf K}}_{\phi(\xi)} (\xi) = \Phi^\prime {{\bf K}} \Phi(\xi) ,$$ (370)

being in general a nonlinear operator.