Loss functions for approximation

Log-loss: A typical loss function for density estimation problems is the log-loss

$$l(x,y,a) = -b_1(x)\ln p(y\vert x,a)+b_2(x,y)$$ (46)

with some $a$-independent $b_1(x)>0$, $b_2(x,y)$ and actions $a$ describing probability densities

$$\int \!dy\, p(y\vert x,a) = 1, \,\,\forall x\in X, \forall a\in A .$$ (47)

Choosing $b_2(x,y)$ = $\ln p(y\vert x,f)$ and $b_1(x)$ = $1$ gives

$$r(a,f) = \int\! dx\, dy\, p(x)p(y\vert x,f) \ln \frac{p(y\vert x,f)}{p(y\vert x,a)}$$ (48)

$$= < \ln \frac{p(y\vert x,f)}{p(y\vert x,a)} >_{X,Y\vert f}$$ (49)

$$= <{\rm KL}( \,{p(y\vert x,f)},\, {p(y\vert x,a)}\, )>_{X} ,$$ (50)

which shows that minimizing log-loss is equivalent to minimizing the ($x$-averaged) Kullback-Leibler entropy ${\rm KL}( \, {p(y\vert x,f)},\, {p(y\vert x,a)}\, )$[122,123,13,46,53].

While the paper will concentrate on log-loss we will also give a short summary of loss functions for regression problems. (See for example [16,201] for details.) Regression problems are special density estimation problems where the considered possible actions are restricted to $y$-independent functions $a(x)$.

Squared-error loss: The most common loss function for regression problems (see Sections 3.7, 3.7.2) is the squared-error loss. It reads for one-dimensional $y$

$$l(x,y,a) = b_1(x) \left( y-a(x) \right)^2 +b_2(x,y) ,$$ (51)

with arbitrary $b_1(x)>0$ and $b_2(x,y)$. In that case the optimal function $a(x)$ is the regression function of the posterior which is the mean of the predictive density

$$a^*(x) = \int \!dy \, y \, p(y\vert x,f) = \,\, <y>_{Y\vert x,f} .$$ (52)

This can be easily seen by writing

$$\left( y-a(x) \right)^2 = \left( y\;-\!<y>_{Y\vert x,f}+<y>_{Y\vert x,f}-\;a(x) \right)^2$$ (53)

$$= \left( y\;-\!<y>_{Y\vert x,f} \right)^2 +\left( a(x)\;- <y>_{Y\vert x,f} \right)^2$$

$$- 2 \left( y\,-<y>_{Y\vert x,f} \right) \left( a(x)\;- <y>_{Y\vert x,f} \right),$$ (54)

where the first term in (54) is independent of $a$ and the last term vanishes after integration over $y$ according to the definition of $<y>_{Y\vert x,f}$. Hence,

$$r(a,f)=\int\!dx\,b_1(x) p(x)\left( a(x)\,-\!<y>_{Y\vert x,f} \right)^2+{\rm const.}$$ (55)

This is minimized by $a(x) = <y>_{Y\vert x,f}$. Notice that for Gaussian $p(y\vert x,a)$ with fixed variance log-loss and squared-error loss are equivalent. For multi-dimensional $y$ one-dimensional loss functions like Eq. (51) can be used when the component index of $y$ is considered part of the $x$-variables. Alternatively, loss functions depending explicitly on multidimensional $y$ can be defined. For instance, a general quadratic loss function would be

$$l(x,y,a)= \sum_{k,k^\prime }(y_k-a_k){\bf K}(k,k^\prime)(y_{k^\prime } -a_{k^\prime }(x)).$$ (56)

with symmetric, positive definite kernel ${\bf K}(k,k^\prime )$.

Absolute loss: For absolute loss

$$l(x,y,a) = b_1(x) \vert y-a(x)\vert +b_2(x,y) ,$$ (57)

with arbitrary $b_1(x)>0$ and $b_2(x,y)$. The risk becomes

$$r(a,f) = \int \!dx\, b_1(x) p(x) \int_{-\infty}^{a(x)}\!dy \left (a(x)-y\right) p(y\vert x,f)$$

$$+ \int \!dx\, b_1(x)p(x)\int_{a(x)}^\infty\!dy \left (y-a(x)\right) p(y\vert x,f) +{\rm const.}$$ (58)

$$= 2\int \!dx\, b_1(x) p(x) \int_{m(x)}^{a(x)}\!dy \left (a(x)-y\right) p(y\vert x,f) +{\rm const.}^\prime ,$$ (59)

where the integrals have been rewritten as $\int_{-\infty}^{a(x)}$ = $\int_{-\infty}^{m(x)}$ + $\int_{m(x)}^{a(x)}$ and $\int_{a(x)}^\infty$ = $\int_{a(x)}^{m(x)}$ + $\int_{m(x)}^{\infty}$ introducing a median function $m(x)$ which satisfies

$$\int_{-\infty}^{m(x)} \!dy\, p(y\vert x,f) = \frac{1}{2},\, \forall x\in X ,$$ (60)

so that

$$a(x) \left( \int_{-\infty}^{m(x)} \!dy\, p(y\vert x,f) -\in... ...}^\infty \!dy\, p(y\vert x,f) \right) = 0,\, \forall x\in X .$$ (61)

Thus the risk is minimized by any median function $m(x)$.

$\delta$-loss and $0$-$1$ loss : Another possible loss function, typical for classification tasks (see Section 3.8), like for example image segmentation [153], is the $\delta$-loss for continuous $y$ or $0$-$1$-loss for discrete $y$

$$l(x,y,a) = - b_1(x) \delta \left( y-a(x) \right) +b_2(x,y),$$ (62)

with arbitrary $b_1(x)>0$ and $b_2(x,y)$. Here $\delta$ denotes the Dirac $\delta$-functional for continuous $y$ and the Kronecker $\delta$ for discrete $y$. Then,

$$r(a,f) = -\int\!dx\, b_1(x) p(x) \, p(\,y\!=\!a(x)\,\vert x,f) +{\rm const.} ,$$ (63)

so the optimal $a$ corresponds to any mode function of the predictive density. For Gaussians mode and median are unique, and coincide with the mean.