We consider stochastic gradient descent (SGD) methods with adaptive learning learning rates that are changed dynamically during the training process. For a particular class of coercive target functions we show convergence of such SGD methods to global minima. Afterwards, we specialize to the situation of optimizing artificial neural networks (ANNs) and discuss convergence results for the gradient flow, the continuous-time analogue of gradient descent.