Awesome Recurrent Neural Networks

A curated list of resources dedicated to recurrent neural networks (closely related to deep learning).

Maintainers - Jiwon Kim, Myungsub Choi

We have pages for other topics: awesome-deep-vision, awesome-random-forest

Contributing

Please feel free to pull requests, email Myungsub Choi ([email protected]) or join our chats to add links.

Sharing

• [Share on Twitter](http://twitter.com/home?status=http://jiwonkim.org/awesome-rnn%0AResources for Recurrent Neural Networks)
• Share on Facebook
• Share on Google Plus
• Share on LinkedIn

Table of Contents

• Codes
• Theory
  • Lectures
  • Books / Thesis
  • Network Variants
  • Surveys
• Applications
  • Language Modeling
  • Speech Recognition
  • Machine Translation
  • Conversation Modeling
  • Image Captioning
  • Video Captioning
  • Question Answering
  • Image Generation
  • Turing Machines
  • Robotics
  • Other
• Datasets
• Online Demos

Codes

• Theano - Python
  • Simple IPython tutorial on Theano
  • Deep Learning Tutorials
    • RNN for semantic parsing of speech
    • LSTM network for sentiment analysis
  • Pylearn2 : Library that wraps a lot of models and training algorithms in deep learning
  • Blocks : modular framework that enables building neural network models
  • Keras : Theano-based deep learning library similar to Torch, but in Python
  • Lasagne : Lightweight library to build and train neural networks in Theano
  • theano-rnn by Graham Taylor
  • Passage : Library for text analysis with RNNs
  • Theano-Lights : Contains many generative models
• Caffe - C++ with MATLAB/Python wrappers
  • LRCN by Jeff Donahue
• Torch - Lua
  • char-rnn by Andrej Karpathy : multi-layer RNN/LSTM/GRU for training/sampling from character-level language models
  • LSTM by Wojciech Zaremba : Long Short Term Memory Units to train a language model on word level Penn Tree Bank dataset
  • Oxford by Nando de Freitas : Oxford Computer Science - Machine Learning 2015 Practicals
  • rnn by Nicholas Leonard : general library for implementing RNN, LSTM, BRNN and BLSTM (highly unit tested).
• Etc.
  • Chainer : new, flexible deep learning library in Python
  • CGT(Computational Graph Toolkit) : replicates Theano's API, but with very short compilation time and multithreading
  • RNNLIB by Alex Graves : C++ based LSTM library
  • RNNLM by Tomas Mikolov : C++ based simple code
  • neuraltalk by Andrej Karpathy : numpy-based RNN/LSTM implementation
  • gist by Andrej Karpathy : raw numpy code that implements an efficient batched LSTM
  • Recurrentjs by Andrej Karpathy : a beta javascript library for RNN
  • RNN by Andrej Karpathy: a good brief introduction for RNN.

Theory

Lectures

• Stanford NLP (CS224d) by Richard Socher
  • Lecture Note 3 : neural network basics
  • Lecture Note 4 : RNN language models, bi-directional RNN, GRU, LSTM
• Stanford vision (CS231n) by Andrej Karpathy
  • About NN basic, and CNN
• Oxford Machine Learning by Nando de Freitas
  • Lecture 12 : Recurrent neural networks and LSTMs
  • Lecture 13 : (guest lecture) Alex Graves on Hallucination with RNNs

Books / Thesis

• Alex Graves (2008)
  • Supervised Sequence Labelling with Recurrent Neural Networks
• Tomas Mikolov (2012)
  • Statistical Language Models based on Neural Networks
• Ilya Sutskever (2013)
  • Training Recurrent Neural Networks
• Richard Socher (2014)
  • Recursive Deep Learning for Natural Language Processing and Computer Vision

Network Variants

• Bi-directional RNN [Paper]
  • Mike Schuster and Kuldip K. Paliwal, Bidirectional Recurrent Neural Networks, Trans. on Signal Processing 1997
• LSTM [Paper]
  • Sepp Hochreiter and Jurgen Schmidhuber, Long Short-Term Memory, Neural Computation 1997
• Multi-dimensional RNN [Paper]
  • Alex Graves, Santiago Fernandez, and Jurgen Schmidhuber, Multi-Dimensional Recurrent Neural Networks, ICANN 2007
• GRU (Gated Recurrent Unit) [Paper]
  • Kyunghyun Cho, Bart van Berrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio, Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, arXiv:1406.1078 / EMNLP 2014
• GFRNN [Paper-arXiv] [Paper-ICML] [Supplementary]
  • Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio, Gated Feedback Recurrent Neural Networks, arXiv:1502.02367 / ICML 2015
• Tree-Structured LSTM [Paper]
  • Kai Sheng Tai, Richard Socher, and Christopher D. Manning, arXiv:1503.00075 / ACL 2015
• Grid LSTM [Paper]
  • Nal Kalchbrenner, Ivo Danihelka, and Alex Graves, Grid Long Short-Term Memory, arXiv:1507.01526

Surveys

• Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, Deep Learning, Nature 2015
• Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnik, Bas R. Steunebrink, Jurgen Schmidhuber, LSTM: A Search Space Odyssey, arXiv:1503.04069
• Zachary C. Lipton, A Critical Review of Recurrent Neural Networks for Sequence Learning, arXiv:1506.00019
• Andrej Karpathy, Justin Johnson, Li Fei-Fei, Visualizing and Understanding Recurrent Networks, arXiv:1506.02078
• Rafal Jozefowicz, Wojciech Zaremba, Ilya Sutskever, An Empirical Exploration of Recurrent Network Architectures, ICML, 2015.

Applications

Language Modeling

• Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan "Honza" Cernocky, Sanjeev Khudanpur, Recurrent Neural Network based Language Model, Interspeech 2010 [Paper]
• Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan "Honza" Cernocky, Sanjeev Khudanpur, Extensions of Recurrent Neural Network Language Model, ICASSP 2011 [Paper]
• Stefan Kombrink, Tomas Mikolov, Martin Karafiat, Lukas Burget, Recurrent Neural Network based Language Modeling in Meeting Recognition, Interspeech 2011 [Paper]
• Jiwei Li, Minh-Thang Luong, and Dan Jurafsky, A Hierarchical Neural Autoencoder for Paragraphs and Documents, ACL 2015 [Paper], [Code]
• Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, and Richard S. Zemel, Skip-Thought Vectors, arXiv:1506.06726 [Paper]

Speech Recognition

• Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, and Brian Kingsbury, Deep Neural Networks for Acoustic Modeling in Speech Recognition, IEEE Signam Processing Magazine 2012 [Paper]
• Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton, Speech Recognition with Deep Recurrent Neural Networks, arXiv:1303.5778 / ICASSP 2013 [Paper]
• Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio, Attention-Based Models for Speech Recognition, arXiv:1506.07503 [Paper]

Machine Translation

• Oxford [Paper]
  • Nal Kalchbrenner and Phil Blunsom, Recurrent Continuous Translation Models, EMNLP 2013
• Univ. Montreal
  • Kyunghyun Cho, Bart van Berrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio, Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, arXiv:1406.1078 / EMNLP 2014 [Paper]
  • Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio, On the Properties of Neural Machine Translation: Encoder-Decoder Approaches, SSST-8 2014 [Paper]
  • Jean Pouget-Abadie, Dzmitry Bahdanau, Bart van Merrienboer, Kyunghyun Cho, and Yoshua Bengio, Overcoming the Curse of Sentence Length for Neural Machine Translation using Automatic Segmentation, SSST-8 2014
  • Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, arXiv:1409.0473 / ICLR 2015 [Paper]
  • Sebastian Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio, On using very large target vocabulary for neural machine translation, arXiv:1412.2007 / ACL 2015 [Paper]
• Univ. Montreal + Middle East Tech. Univ. + Univ. Maine [Paper]
  • Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio, On Using Monolingual Corpora in Neural Machine Translation, arXiv:1503.03535
• Google [Paper]
  • Ilya Sutskever, Oriol Vinyals, and Quoc V. Le, Sequence to Sequence Learning with Neural Networks, arXiv:1409.3215 / NIPS 2014
• Google + NYU [Paper]
  • Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba, Addressing the Rare Word Problem in Neural Machine Transltaion, arXiv:1410.8206 / ACL 2015
• Stanford [Paper]
  • Minh-Thang Luong, Hieu Pham, and Christopher D. Manning, Effective Approaches to Attention-based Neural Machine Translation, arXiv:1508.04025

Conversation Modeling

• Lifeng Shang, Zhengdong Lu, and Hang Li, Neural Responding Machine for Short-Text Conversation, arXiv:1503.02364 / ACL 2015 [Paper]
• Oriol Vinyals and Quoc V. Le, A Neural Conversational Model, arXiv:1506.05869 [Paper]
• Ryan Lowe, Nissan Pow, Iulian V. Serban, and Joelle Pineau, The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems, arXiv:1506.08909 [Paper]

Image Captioning

• UCLA + Baidu [Web] [Paper-arXiv1], [Paper-arXiv2]
  • Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan L. Yuille, Explain Images with Multimodal Recurrent Neural Networks, arXiv:1410.1090
  • Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan L. Yuille, Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN), arXiv:1412.6632 / ICLR 2015
• Univ. Toronto [Paper] [Web demo]
  • Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel, Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models, arXiv:1411.2539 / TACL 2015
• Berkeley [Web] [Paper]
  • Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell, Long-term Recurrent Convolutional Networks for Visual Recognition and Description, arXiv:1411.4389 / CVPR 2015
• Google [Paper]
  • Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan, Show and Tell: A Neural Image Caption Generator, arXiv:1411.4555 / CVPR 2015
• Stanford [Web] [Paper]
  • Andrej Karpathy and Li Fei-Fei, Deep Visual-Semantic Alignments for Generating Image Description, CVPR 2015
• Microsoft [Paper]
  • Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Srivastava, Li Deng, Piotr Dollar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, Lawrence Zitnick, and Geoffrey Zweig, From Captions to Visual Concepts and Back, arXiv:1411.4952 / CVPR 2015
• CMU + Microsoft [Paper-arXiv], [Paper-CVPR]
  • Xinlei Chen, and C. Lawrence Zitnick, Learning a Recurrent Visual Representation for Image Caption Generation
  • Xinlei Chen, and C. Lawrence Zitnick, Mind’s Eye: A Recurrent Visual Representation for Image Caption Generation, CVPR 2015
• Univ. Montreal + Univ. Toronto [Web] [Paper]
  • Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio, Show, Attend, and Tell: Neural Image Caption Generation with Visual Attention, arXiv:1502.03044 / ICML 2015
• Idiap + EPFL + Facebook [Paper]
  • Remi Lebret, Pedro O. Pinheiro, and Ronan Collobert, Phrase-based Image Captioning, arXiv:1502.03671 / ICML 2015
• UCLA + Baidu [Paper]
  • Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan L. Yuille, Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images, arXiv:1504.06692
• MS + Berkeley
  • Jacob Devlin, Saurabh Gupta, Ross Girshick, Margaret Mitchell, and C. Lawrence Zitnick, Exploring Nearest Neighbor Approaches for Image Captioning, arXiv:1505.04467 (Note: technically not RNN) [Paper]
  • Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, and Margaret Mitchell, Language Models for Image Captioning: The Quirks and What Works, arXiv:1505.01809 [Paper]
• Adelaide [Paper]
  • Qi Wu, Chunhua Shen, Anton van den Hengel, Lingqiao Liu, and Anthony Dick, Image Captioning with an Intermediate Attributes Layer, arXiv:1506.01144
• Tilburg [Paper]
  • Grzegorz Chrupala, Akos Kadar, and Afra Alishahi, Learning language through pictures, arXiv:1506.03694
• Univ. Montreal [Paper]
  • Kyunghyun Cho, Aaron Courville, and Yoshua Bengio, Describing Multimedia Content using Attention-based Encoder-Decoder Networks, arXiv:1507.01053
• Cornell [Paper]
  • Jack Hessel, Nicolas Savva, and Michael J. Wilber, Image Representations and New Domains in Neural Image Captioning, arXiv:1508.02091

Video Captioning

• Berkeley [Web] [Paper]
  • Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell, Long-term Recurrent Convolutional Networks for Visual Recognition and Description, arXiv:1411.4389 / CVPR 2015
• UT Austin + UML + Berkeley [Paper]
  • Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko, Translating Videos to Natural Language Using Deep Recurrent Neural Networks, arXiv:1412.4729
• Microsoft [Paper]
  • Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui, Joint Modeling Embedding and Translation to Bridge Video and Language, arXiv:1505.01861
• UT Austin + Berkeley + UML [Paper]
  • Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko, Sequence to Sequence--Video to Text, arXiv:1505.00487
• Univ. Montreal + Univ. Sherbrooke [Paper]
  • Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville, Describing Videos by Exploiting Temporal Structure, arXiv:1502.08029
• MPI + Berkeley [Paper]
  • Anna Rohrbach, Marcus Rohrbach, and Bernt Schiele, The Long-Short Story of Movie Description, arXiv:1506.01698
• Univ. Toronto + MIT [Paper]
  • Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler, Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books, arXiv:1506.06724
• Univ. Montreal [Paper]
  • Kyunghyun Cho, Aaron Courville, and Yoshua Bengio, Describing Multimedia Content using Attention-based Encoder-Decoder Networks, arXiv:1507.01053

Question Answering

• FAIR [Web] [Paper]
  • Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov, and Alexander M. Rush, Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks, arXiv:1502.05698
• Virginia Tech. + MSR [Web] [Paper]
  • Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh, VQA: Visual Question Answering, arXiv:1505.00468 / CVPR 2015 SUNw:Scene Understanding workshop
• MPI + Berkeley [Web] [Paper]
  • Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz, Ask Your Neurons: A Neural-based Approach to Answering Questions about Images, arXiv:1505.01121
• Univ. Toronto [Paper] [Dataset]
  • Mengye Ren, Ryan Kiros, and Richard Zemel, Exploring Models and Data for Image Question Answering, arXiv:1505.02074 / ICML 2015 deep learning workshop
• Baidu + UCLA [Paper] [Dataset]
  • Hauyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu, Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering, arXiv:1505.05612
• DeepMind + Oxford [Paper]
  • Karl M. Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom, Teaching Machines to Read and Comprehend, arXiv:1506.03340
• MetaMind [Paper]
  • Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Mohit Iyyer, Ishaan Gulrajani, and Richard Socher, Ask Me Anything: Dynamic Memory Networks for Natural Language Processing, arXiv:1506.07285

Image Generation

• Karol Gregor, Ivo Danihelka, Alex Graves, Danilo J. Rezende, and Daan Wierstra, DRAW: A Recurrent Neural Network for Image Generation, ICML 2015 [Paper]
• Angeliki Lazaridou, Dat T. Nguyen, R. Bernardi, and M. Baroni, Unveiling the Dreams of Word Embeddings: Towards Language-Driven Image Generation, arXiv:1506.03500 [Paper]
• Lucas Theis and Matthias Bethge, Generative Image Modeling Using Spatial LSTMs, arXiv:1506.03478 [Paper]

Turing Machines

• A.Graves, G. Wayne, and I. Danihelka., Neural Turing Machines, arXiv preprint arXiv:1410.5401 [Paper]
• Jason Weston, Sumit Chopra, Antoine Bordes, Memory Networks, arXiv:1410.3916 [Paper]
• Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus, End-To-End Memory Networks, arXiv:1503.08895 [Paper]
• Wojciech Zaremba and Ilya Sutskever, Reinforcement Learning Neural Turing Machines, arXiv:1505.00521 [Paper]
• Baolin Peng and Kaisheng Yao, Recurrent Neural Networks with External Memory for Language Understanding, arXiv:1506.00195 [Paper]

Robotics

• Hongyuan Mei, Mohit Bansal, and Matthew R. Walter, Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences, arXiv:1506.04089 [Paper]
• Marvin Zhang, Sergey Levine, Zoe McCarthy, Chelsea Finn, and Pieter Abbeel, Policy Learning with Continuous Memory States for Partially Observed Robotic Control, arXiv:1507.01273. [Paper]

Other

• Alex Graves, Generating Sequences With Recurrent Neural Networks, arXiv:1308.0850 [Paper]
• Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu, Recurrent Models of Visual Attention, NIPS 2014 / arXiv:1406.6247 [Paper]

Datasets

• Speech Recognition
  • OpenSLR (Open Speech and Language Resources)
    • LibriSpeech ASR corpus
  • VoxForge
• Image Captioning
  • Flickr 8k
  • Flickr 30k
  • Microsoft COCO
• Image Question Answering
  • DAQUAR - built upon NYU Depth v2 by N. Silberman et al.
  • VQA - based on MSCOCO images
  • Image QA - based on MSCOCO images
  • [Multilingual Image QA] - built from scratch by Baidu - in Chinese, with English translation
• Action Recognition
  • THUMOS : Large-scale action recognition dataset
  • MultiTHUMOS : Extension of THUMOS '14 action detection dataset with dense multilabele annotation

Online Demos

• Alex graves, hand-writing generation [link]
• Ink Poster: Handwritten post-it notes [link]