Rmsprop

RMSProp. Per-coordinate adaptive learning rate via an exponential moving average of squared gradients: the same shape as Adagrad but with a sliding window instead of a monotone accumulator.

Reach for Rmsprop when Adagrad's effective learning rate decays faster than you want; non-stationary streams, online problems where the data distribution drifts over the lifetime of the optimizer. rho near 1 gives a long memory (close to Adagrad); rho near 0 gives a short memory.