Adagrad

Adagrad. Per-coordinate adaptive learning rate via accumulated squared gradients: w[i] -= lr * grad[i] / sqrt(sumG2[i] + epsilon).

Reach for Adagrad when feature occurrence is sparse and uneven; power-law-distributed categorical features, rarely-seen tokens, anything where you want rare features to take big steps and common features to settle into small ones. The accumulating denominator makes Adagrad's effective learning rate monotonically non-increasing, which is the limitation Rmsprop addresses.