There exists a variety of well developed numerical methods for unconstraint as well as for constraint optimization [190,57,88,196,89,12,19,80,193]. Popular examples include conjugate gradient, Newton, and quasi-Newton methods, like the variable metric methods DFP (Davidon-Fletcher-Powell) or BFGS (Broyden-Fletcher-Goldfarb-Shanno).
All of them correspond to the choice of specific,
often iteration dependent, learning matrices
defining the learning algorithm.
Possible simple choices are:
On one hand, density estimation is a rather general problem requiring the solution of constraint, inhomogeneous, nonlinear (integro-)differential equations. On the other hand, density estimation problems are, at least for Gaussian specific priors and non restricting parameterization, typically ``nearly'' linear and have only a relatively simple non-negativity and normalization constraint. Furthermore, the inhomogeneities are commonly restricted to a finite number of discrete training data points. Thus, we expect the inversion of to be the essential part of the solution for density estimation problems. However, is not necessarily invertible or may be difficult to calculate. Also, inversion of is not exactly what is optimal and there are improved methods. Thus, we will discuss in the following basic optimization methods adapted especially to the situation of density estimation.