Basic RNNs

Recurrent neural nets. Vanishing gradient problem, naively, RNNs don't give you long term memory.. so you have Long short-term memory networks

Recurrent neural networks -- Schmidhuber

Recurrent Neural Network Writes Music and Shakespeare Novels - Two Minute Papers

The vanishing gradients problem

Hochreiter1991 – Bengio1994

Long short-term memory

Proposed to solve the vanishing gradients problem

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

See also Neural networks with memory, Deep learning

Gated recurrent unit

A variation of the Long short-term memory network

More Augmented RNNs

Nice curated list of RNNs: https://github.com/kjw0612/awesome-rnn

Higher Order Recurrent Neural Networks . We propose to use more memory units to keep track of more preceding states in recurrent neural networks (RNNs), which are all recurrently fed to the hidden layers as feedback through different weighted path