Imitation learning and learning from observation

Definition - Imitation learning

• Imitation learning techniques aim to mimic human behavior in a given task.
• An agent (a learning machine) is trained to perform a task from demonstrations by learning a mapping between observations and actions.
• 
• Bildunterschrift: imitation is learning?

Imitation learning flowchart

Relation with RL

• 
• A task may be formulated as an optimization problem, but it has become widely accepted that having prior knowledge provided by an expert is more effective and efficient than searching for a solution from scratch
• [Schaal 1999; Schaal 1997; Bakker and Kuniyoshi 1996; Billard et al. 2008].
• Bildunterschrift: Imitation learning and RL

Imitation learning - Observational

• State-of-the-art:
• 2017 June 20, Diana Borsa, Bilal Piot, Rémi Munos, and Olivier Pietquin. Observational Learning by Reinforcement Learning. arXiv:1706.06617.
• 2017 July 7, Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv:1707.02201. video.
• 2017 May 15, Kareem Amin, Nan Jiang, and Satinder Singh. Repeated Inverse Reinforcement Learning for AI Safety. arXiv:1705.05427.
• 2017 March 22, Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-Shot Imitation Learning. arXiv:1703.07326. blog. video
• 2017 March 6, Bradly C. Stadie, Pieter Abbeel, and Ilya Sutskever. Third-Person Imitation Learning. arXiv:1703.01703.
• 2016 June 10, Jonathan Ho and Stefano Ermon. Generative Adversarial Imitation Learning. arXiv:1606.03476.
• 2016 March 1, Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization. arXiv:1603.00448.
• Observational Learning
• In observational learning, a student agent can observe a teacher agent, but is not required to learn anything from it.
• In simple scenarios where a student agent seeks a sometimes hidden goal,
• Borsa et al. (2017) demonstrate that a reinforcement learning agent can distill useful information contained in observed teacher behavior, and learns useful policies such as seeking the teacher then simply following it to the goal.

Deep Learning

• Many of the successes in deep learning can be attributed to the availability of computing capacity and data. As of July 2017, performance of deep learning models in vision tasks still increases with orders of magnitude of training data size (Sun et al., 2017).
• Shalev-Shwartz et al. (2017) described four types of simple problems that are difficult for common deep learning methods.
• References
• 2017 July 10, Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. arXiv:1707.02968.
• 2017 June 5, Alessandro Achille and Stefano Soatto. On the Emergence of Invariance and Disentangling in Deep Representations. arXiv:1706.01350.
• 2017 April 29, Ravid Shwartz-Ziv and Naftali Tishby. Opening the Black Box of Deep Neural Networks via Information. arXiv:1703.00810. video. news.
• 2017 April 26, Shai Shalev-Shwartz, Ohad Shamir, and Shaked Shammah. Failures of Gradient-Based Deep Learning. arXiv:1703.07950.
• Information Theory
• In the plane of the mutual information (MI) between input and representation and the MI between representation and output (called the Information Plane), Schwartz-Ziv & Tishby (2017) visualized SGD in two stages, error minimization and representation compression. The compression stage covers most training epochs and begins after training errors get small.
• References
• 2017 June 16, Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A Closer Look at Memorization in Deep Networks. arXiv:1706.05394.
• 2017 May 15, Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp Minima Can Generalize For Deep Nets. arXiv:1703.04933.
• 2017 April 21, Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing Gradient Descent Into Wide Valleys. arXiv:1611.01838.
• 2017 February 26, Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv:1611.03530.
• 2017 February 9, Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. arXiv:1609.04836.
• 2017 February 20, David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S. Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep Nets Don’t Learn via Memorization. OpenReview.

Continual Learning

• Continual learning focuses on an agent’s ability to learn new tasks without forgetting old ones. Skills learned from old tasks may or may not transfer to new tasks, but must be largely retained.
• One approach is to penalize learning for weights that are important for remembering old t tasks.
• Zenke & Poole & Ganguli (2017) introduce intelligent synapses that compute importance measures of weights online.
• Another is to design novel network architectures. PathNets (Fernando et al., 2017) are large neural networks where the learning of a particular task evolves towards a subset of parameters, these parameters are then fixed before learning moves on to the next task.
• 2017 October 2, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Lifelong Learning with Dynamically Expandable Networks. arXiv:1708.01547.
• 2017 May 31, Tom Veniat and Ludovic Denoyer. Learning Time-Efficient Deep Architectures with Budgeted Super Networks. arXiv:1706.00046.
• 2017 May 23, Ari Seff, Alex Beatson, Daniel Suo, and Han Liu. Continual Learning in Generative Adversarial Nets. arXiv:1705.08395.
• 2017 May 20, Roby Velez and Jeff Clune. Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural networks. arXiv:1705.07241.
• 2017 May 11, Amir Rosenfeld and John K. Tsotsos. Incremental Learning Through Deep Adaptation. arXiv:1705.04228.
• 2017 March 13, Friedemann Zenke, Ben Poole, and Surya Ganguli. Improved multitask learning through synaptic intelligence. arXiv:1703.04200. code.
• 2017 January 30, Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. PathNet: Evolution Channels Gradient Descent in Super Neural Networks. arXiv:1701.08734.

Few-Shot Learning

• Few-shot learning is about learning to solve tasks using only a small number of samples.
• In classification models, for example, the objective is to perform well after seeing very few samples from each class.
• Finn et al. (2017) propose a model-agnostic algorithm to learn a good weight initialization so that subsequent learning converges rapidly.
• This technique is later used in a continuous adaptation ( Al-Shedivat et al., 2017) problem, where an agent learns to solve a single but nonstationary task or environment.
• References
• 2017 October 10, Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments. arXiv:1710.03641. site.
• 2017 July 18, Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv:1703.03400. blog.
• 2017 March 15, Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical Networks for Few-shot Learning. arXiv:1703.05175.

Meta learning - learning-to-learn

• Learning to Optimize
• A standard learning architecture can be made more flexible and efficient when its internal dynamics is also learned from data.
• Conceptually, this dynamics learning occurs at a higher level than the learning implemented by the standard architecture.
• Wichrowska et al. (2017) introduce a hierarchical RNN optimizer and achieve better generalization to new problems and large number of optimization steps.
• Bello & Zoph et al. (2017) train a RNN controller to search the space of mathematical expressions of update rules and discover rules that perform better than commonly used optimizers such as Adam and RMSProp.
• .
• An agent can be trained across many different tasks to meta-learn a reinforcement learning procedure that rapidly and efficiently adapts to a new task;
• Works have been done focusing on relatively structured (Wang et al., 2016) and unstructured (Duan et al., 2016) distributions of tasks.
• References:
• * 2017 September 12, Ke Li. Learning to Optimize with Reinforcement Learning. Berkeley Artificial Intelligence Research.
• 2017 September 21, Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V. Le. Neural Optimizer Search with Reinforcement Learning. arXiv:1709.07417.
• 2017 July 11, Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. Meta-Learning with Temporal Convolutions. arXiv:1707.03141.
• 2017 March 14, Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned Optimizers that Scale and Generalize. arXiv:1703.04813.

Memory

• Memory is a significant idea inspired by neuroscience, from which we know that there are different types of memory systems in the brain, such as short-term working memory and episodic memory. Traditionally neural networks don’t have much memory and adding memory to them hasn’t yet shown widespread evidence of promise. Consequently, this is a relatively neglected area in the science towards AGI.
• Episodic Memory
• 2017 March 6, Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural Episodic Control. arXiv:1703.01988.
• 2016 June 14, Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-Free Episodic Control. arXiv:1606.04460.
• References
• 2017 February 27, Emilio Parisotto and Ruslan Salakhutdinov. Neural Map: Structured Memory for Deep Reinforcement Learning. arXiv:1702.08360.
• 2017 January 30, Caglar Gulcehre, Sarath Chandar, and Yoshua Bengio. Memory Augmented Neural Networks with Wormhole Connections. arXiv:1701.08718.
• 2016 October 27, Jack W Rae, Jonathan J Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, and Timothy P Lillicrap. Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes. arXiv:1610.09027.
• 2016 January 5, Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471-476.

Network Architecture

• Modular Networks
• Hafner et al. (2017) propose ThalNet, a modular neural network inspired by neocortical communication via the thalamus.
• It comprises a tuple of computation modules that route their respective features into a shared center vector, which is then read selectively by each module to obtain its recurrent context input. It is shown that the model learns to route information hierarchically.
• References
• 2017 June 18, Danijar Hafner, Alex Irpan, James Davidson, and Nicolas Heess. Learning Hierarchical Information Flow with Recurrent Neural Modules. arXiv:1706.05744.
• Conditional Computation
• Shazeer et al. (2017) introduced a Sparsely-Gated Mixture-of-Experts (MoE) layer where each sample only activates a sparse subset of many expert networks, determined by a gating network. The layer is embedded in a larger architecture that is overall trainable by backpropagation.
• Evolutionary Algorithms
• In an experiment evolving networks to produce patterns, Huizinga et al. (2017) showed that network weights encoded dimensions of image variations that are sensible to humans, possibly due to the diversity of environmental selection.

Deep Feedforward Networks

• Greff et al. (2016) argued that the functional blocks of Highway Networks (Srivastava et al., 2015) and Residual Networks (He et al., 2015) engage in an unrolled iterative estimation of representations. Gomez & Ren et al. (2017) present the Reversible Residual Network (RevNet) where each layer’s activations can be reconstructed from the next layer’s, eliminating the need to store activations in memory during backpropagation.
• A deep feedforward network can be interrupted at an intermediate layer to make anytime predictions. Hu et al. (2017) add uniformly spaced auxiliary predictions and losses to the residual bottleneck units of Resnets, and achieve competitive results.
• References
• 2017 August 23, Hanzhang Hu, Debadeepta Dey, J. Andrew Bagnell, and Martial Hebert. Anytime Neural Networks via Joint Optimization of Auxiliary Losses. arXiv:1708.06832.
• 2017 July 14, Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. The Reversible Residual Network: Backpropagation Without Storing Activations. arXiv:1707.04585.
• 2017 April 18, Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to Reason: End-to-End Module Networks for Visual Question Answering. arXiv:1704.05526.
• 2017 April 17, Joost Huizinga, Kenneth O. Stanley, and Jeff Clune. The Emergence of Canalization and Evolvability in an Open-Ended, Interactive Evolutionary System. arXiv:1704.05143. code.
• 2017 March 10, Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabas Poczos, Ruslan Ssalakhutdinov, and Alexander Smola. Deep Sets. arXiv:1703.06114.
• 2017 January 30, Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. PathNet: Evolution Channels Gradient Descent in Super Neural Networks. arXiv:1701.08734.
• 2017 January 23, Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538.

Adversarial Learning

• Optical illusions are like adversarial examples for the human brain. Meiji University’s Koichi Sugihara has made quite a few stunning 3D illusions.
• Lu et al. (2017) show that when the pictures of physical adversarial examples are taken from different distances and angles, they are correctly classified by trained neural networks.
• Evtimov et al. (2017) propose Robust Physical Perturbations, an algorithm that can generate robust adversarial examples. They physically realize an adversarial example of a Stop sign that is misclassified under varying conditions, such as distances, angles, and resolutions.
• Adversarial Training Examples
• Koh & Liang (2017) demonstrated that modern neural networks are vunerable to adversarial training-set attacks. They successfully flipped the classification output of an Inception network by making a visually-indistinguishable small change to just 1 training image.
• References
• 2017 July 27, Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song. Robust Physical-World Attacks on Machine Learning Models. arXiv:1707.08945.
• 2017 July 12, Jiajun Lu, Hussein Sibai, Evan Fabry, and David Forsyth. NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles. arXiv:1707.03501.
• 2017 April 11, Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The Space of Transferable Adversarial Examples. arXiv:1704.03453.
• 2017 March 27, Aran Nayebi and Surya Ganguli. Biologically inspired protection of deep networks from adversarial attacks. arXiv:1703.09202.
• 2017 April 5, Wieland Brendel and Matthias Bethge. Comment on “Biologically inspired protection of deep networks from adversarial attacks”. arXiv:1704.01547.
• 2017 March 14, Pang Wei Koh and Percy Liang. Understanding Black-box Predictions via Influence Functions. arXiv:1703.04730.
• 2017 February 6, Jonathan C. Brant and Kenneth O. Stanley. Minimal Criterion Coevolution: A New Approach to Open-Ended Search. The Genetic and Evolutionary Computation Conference (GECCO).
• 2017 January 27, Anjuli Kannan and Oriol Vinyals. Adversarial Evaluation of Dialogue Models. arXiv:1701.08198.
• 2017 January 26, Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv:1701.07875.
• 2017 January 23, Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial Learning for Neural Dialogue Generation. arXiv:1701.06547. code
• 2017 January 17, Martin Arjovsky and Léon Bottou. Towards Principled Methods for Training Generative Adversarial Networks. arXiv:1701.04862.
• 2017 January 4, Dmitry Krotov and John J Hopfield. Dense Associative Memory is Robust to Adversarial Inputs. arXiv:1701.00939.

Generative Models

• Generative models are likely what humans and some other animals rely on for planning.
• Variational Autoencoders
• The variational autoencoders (VAE) framework combines a neural-network based inference model with a neural-network based generative model. For an overview, see
• 2017 September 30, Diederik P. Kingma. Variational Inference & Deep Learning: A New Synthesis (PDF). Dropbox.
• A subset of latent variables in a VAE can be made interpretable by applying semi-supervised learning, where labels are supplied for these variables (Siddharth et al., 2017).
• References
• 2017 June 1, N. Siddharth, Brooks Paige, Jan-Willem Van de Meent, Alban Desmaison, Frank Wood, Noah D. Goodman, Pushmeet Kohli, and Philip H. S. Torr. Learning Disentangled Representations with Semi-Supervised Deep Generative Models. arXiv:1706.00400.
• Generative Adversarial Networks
• Hu et al. (2017) establish formal connections between GANs and VAEs, and note that GANs are also related to reinforcement learning, teacher-student distillation and wake-sleep algorithm.
• Memory-Augmented Generative Models
• 2017 September 21, Jörg Bornschein, Andriy Mnih, Daniel Zoran, and Danilo J. Rezende. Variational Memory Addressing in Generative Models. arXiv:1709.07116.

Learning in Other science Fields

• Physics
• Lin et al. (2017) explore how physical properties frequently translate to simple neural networks, then argue that a deep neural network could be more efficient than a shallow one when the data is generated by a hierarchical statistical process.
• They prove it is exponentially more efficient to approximate the product of input variables using a deep network than a shallow one. This theorem is extended in Rolnick & Tegmark (2017) to include broad natural classes of multivariate polynomials.
• References
• 2017 May 16, David Rolnick and Max Tegmark. The power of deeper networks for expressing natural functions. arXiv:1705.05502.
• 2017 May 2, Henry W. Lin, Max Tegmark, and David Rolnick. Why does deep and cheap learning work so well?. arXiv:1608.08225.
• 2017 April 5, Yoav Levine, David Yakira, Nadav Cohen, and Amnon Shashua. Deep Learning and Quantum Physics : A Fundamental Bridge. arXiv:1704.01552.
• Topology
• 2014 January 6, Monica Bianchini and Franco Scarselli. On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures. IEEE Transactions on Neural Networks and Learning Systems, 25(8):1553-1565.
• Transparency
• Weller (2017) provides a brief survey on transparency and reviews settings where it may be unhelpful.
• Gu et al. (2017) is able to create BadNet, a maliciously trained neural network that incorporates a backdoor. In a somewhat realistic setting, a maliciously trained U.S. street sign classifier identifies stop signs as speed limits when a special ticker is added to the stop sign.
• Cognitive Psychology
• Ritter & Barrett et al. (2017) use cognitive psychology to study the one-shot word learning Matching Networks and an Inception Baseline model. They find that the networks prefer to categorize objects according to shape rather than color.
• References
• 2017 August 22, Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv:1708.06733.
• 2017 July 4, Adrian Weller. Challenges for Transparency. OpenReview.
• 2017 June 26, Samuel Ritter, David G. T. Barrett, Adam Santoro, and Matt M. Botvinick. Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study. arXiv:1706.08606.

Computer Vision

• Li et al. (2017) added an additional continue/stop action to the Recurrent Visual Attention Model (Mnih et al., 2014), and saved the average computation time.
• Datasets
• 2017 September 20. Announcing the Matterport3D Research Dataset. Matterport.
• References
• 2017 April 18, Joel Janai, Fatma Güney, Aseem Behl, and Andreas Geiger. Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art. arXiv:1704.05519.
• 2017 March 30, Zhichao Li, Yi Yang, Xiao Liu, Shilei Wen, and Wei Xu. Dynamic Computational Time for Visual Attention. arXiv:1703.10332. code.
• 2016 December 3, Gao Huang, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. Densely Connected Convolutional Networks. arXiv:1608.06993.
• 2014 June 24, Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent Models of Visual Attention. arXiv:1406.6247.

Natural Language Processing

• Chatbot
• Lewis et al. (2017) train an end-to-end chatbot that can negotiate with humans in natural language. In a simple negotiation task involving three item types with assigned numerical values, the chatbot is pre-trained with supervised learning, then fine-tuned by reinforcement learning against a version of itself.
• eural Machine Translation
• 2017 September 22, Philipp Koehn. Neural Machine Translation. arXiv:1709.07809.
• Abstractive Summarization
• Paulus et al. (2017) proposed a reinforcement learning model with intra-decoder attention, and achieved state-of-the-art results on the CNN/Daily Mail dataset.
• Chatbot References
• 2017 June 15, Mike Lewis, Denis Yarats, Yann N. Dauphin, Devi Parikh, and Dhruv Batra. Deal or No Deal? End-to-End Learning for Negotiation Dialogues. Facebook Artificial Intelligence Research. blog. code.
• NMT References
• 2017 May 11, Romain Paulus, Caiming Xiong, and Richard Socher. Your tl;dr by an ai: a deep reinforced model for abstractive summarization. Salesforce MetaMind.
• Further Reading
• 2017 Hilary, University of Oxford. Deep Learning for Natural Language Processing. Phil Blunsom.
• 2017 March 5, arXiv. Neural Machine Translation and Sequence-to-sequence Models: A Tutorial. Graham Neubig.
• 2017 Winter, Stanford University. CS 224N / Ling 284: Natural Language Processing. Chris Manning and Richard Socher.

Robotics

• In Heess et al. (2017), agents successfully learn complex locomotion behavior such as running, jumping, crouching and turning.
• In Merel et al. (2017), agents’ sub-skill policies learned from motion capture data can be reused to solve tasks when controlled by a higher level controller.
• In Wang & Merel et al. (2017), agents learn semantically meaningful embeddings of demonstrations, which allow for smooth policy interpolation.
• References
• 2017 July 10, Ziyu Wang, Josh Merel, Scott Reed, Greg Wayne, Nando de Freitas, and Nicolas Heess. Robust Imitation of Diverse Behaviors. arXiv:1707.02747.
• 2017 July 7, Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, and David Silver. Emergence of Locomotion Behaviours in Rich Environments. arXiv:1707.02286.
• 2017 July 10, Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv:1707.02201.
• 2016 June 10, Jonathan Ho and Stefano Ermon. Generative Adversarial Imitation Learning. arXiv:1606.03476.
• Real World
• Rahmatizadeh et al. (2017) propose an approach that enables inexpensive robots to learn multiple manipulation tasks from relatively few demonstrations. They expect that as their dataset expands with more tasks, only a handful of additional demonstrations might be needed for leaning each new behavior.
• References
• 2017 July 10, Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration. arXiv:1707.02920.

Automated Theorem Proving

• From counting to Clay Mathematics Institute’s Millennium Problems, the mathematics trench is incredibly deep.
• Automated Theorem Proving (ATP) is about proving math theorems automatically.
• In June 2016, it was demonstrated for the first time (Alemi et al., 2016) that neural networks are useful in large scale ATP. Two months later, Whalen (2016) proposed a complete architecture combining deep learning and tree search for the Metamath system. Deep learning was incorporated by Loos et al. (2017) into the proof search procedure of the E Theorem Prover in January 2017.
• Resources
• Conferences: AITP (2017, 2016), LPAR (2017).
• The Deepmath project is a collaboration between Google Research and several universities that seeks to improve automated theorem proving using deep learning and other machine learning techniques.
• References
• 2017 September 28, Mingzhe Wang, Yihe Tang, Jian Wang, and Jia Deng. Premise Selection for Theorem Proving by Deep Graph Embedding. arXiv:1709.09994.
• 2017 March 1, Cezary Kaliszyk, François Chollet, and Christian Szegedy. HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving. arXiv:1703.00426. code. site.
• 2017 January 24, Sarah Loos, Geoffrey Irving, Christian Szegedy, and Cezary Kaliszyk. Deep Network Guided Proof Search. arXiv:1701.06972.
• 2016 August 10, Daniel Whalen. Holophrasm: a neural Automated Theorem Prover for higher-order logic. arXiv:1608.02644. code (release).
• 2016 June 14, Alex A. Alemi, Francois Chollet, Niklas Een, Geoffrey Irving, Christian Szegedy, and Josef Urban. DeepMath - Deep Sequence Models for Premise Selection. arXiv:1606.04442.

Deep Learning Implementation

• A typical approach to address the computational requirements of large-scale neural networks is to use a heterogeneous distributed environment with a mixture of many CPUs and GPUs.
• Mirhoseini & Pham et al. (2017) use reinforcement learning to optimize device placement for TensorFlow computational graphs and find non-trivial device placements for Inception-V3 and RNN LSTM.
• References
• Hardware
• 2017 April 5, Norm Jouppi. Quantifying the performance of the TPU, our first machine learning chip. Google Cloud Platform Blog.
• Benchmarks
• 2017 September 14, Yang You, Zhao Zhang, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. ImageNet Training in 24 Minutes. arXiv:1709.05011.
• 2017 June 13, Azalia Mirhoseini, Hieu Pham, Quoc V. Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. Device Placement Optimization with Reinforcement Learning. arXiv:1706.04972.
• 2016 April 22, Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking Deep Reinforcement Learning for Continuous Control. arXiv:1604.06778.

Forcast Next!! (Not far)

• Milestones
• 2018 March 17 (± 3 months). Groundbreaking Results from DeepMind.
• 2019 March 6 (± 3 months). Multi-Player No-Limit Poker.
• 2021 July 19 (± 12 months). Proving Mathematical Theorems.
• 2022 September 24 (± 12 months). StarCraft.
• DeepMind
• Fanancial Times reported on March 17, 2017 that DeepMind hoped to publish six more papers in highly regarded scientific
• journals Nature or Science “within the next year”. DeepMind is an Alphabet subsidiary that has published over two hundred papers, including three Nature papers since February 2015. Given its track record, and the existence of six high-quality preprint papers on arXiv, we are convinced of this estimate and assign a 90% confidence interval of ± 3 months that six groundbreaking results will come from DeepMind by March 17, 2018.

My Research Project

• Learning from the observation
• Goal: Taking video from experts action, and let the machine to learn from this video.
• The video sources come from different contexts and translated (mapped) into a target context.
• 

Applying (Deep) RL

• 
• Reinforcement learning are applied for learning from video by defining proper reward signals.
• Reward signals are based on distance measure from target images..

PyTorch-loading data

• Input pipline for custom dataset
• 1. Initialize file path or list of file names.
• 2. Lead one data from file (e.g. using numpy. fromfile, PIL.Image.open).
• 3. Preprocess the data (e.g. torchvision.Transform).
• 4. Return a data pair (e.g. image and label).
• class CustomDataset(data.Dataset):
• def __init__(self):
• pass
• def __getitem__(self, index):
• pass
• def __len__(self):
• return 0
• custom_dataset = CustomDataset()
• train_loader = torch.utils.data.DataLoader(dataset=custom_dataset,
• batch_size=100,
• shuffle=True,
• num_workers=2)

Working with CIFAR-10 / MNIST data

• Implementing the input pipline
• > train_dataset = dsets.CIFAR10(root='../data/',
• train=True,
• transform=transforms.ToTensor(),
• download=True)
• >train_dataset = dsets.MNIST(root='./data',
• train=True,
• transform=transforms.ToTensor(),
• download=True)
• Select one data pair (read data from disk).
• > image, label = train_dataset[0]
• Data Loader (this provides queue and thread in a very simple way).
• > train_loader= torch.utils.data.DataLoader(dataset=train_dataset,
• batch_size=100,
• shuffle=True,
• num_workers=2)
• > data_iter = iter(train_loader)
• > images, labels = data_iter.next()
• > for images, labels in train_loader:
• pass

Working with Complex Nets

• resnet = torchvision.models.resnet18(pretrained=True)
• finetune only top layer of the model.
• > for param in resnet.parameters():
• param.requires_grad = False
• finetuning
• > resnet.fc = nn.Linear(resnet.fc.in_features, 100)
• function autoencoder:initialize() local pool_layer1 = nn.SpatialMaxPooling(2, 2, 2, 2) local pool_layer2 = nn.SpatialMaxPooling(2, 2, 2, 2) self.net = nn.Sequential() self.net:add(nn.SpatialConvolution(3, 12, 3, 3, 1, 1, 0, 0)) self.net:add(nn.ReLU()) self.net:add(nn.SpatialConvolution(12, 12, 3, 3, 1, 1, 0, 0)) self.net:add(nn.ReLU()) self.net:add(pool_layer1) self.net:add(nn.SpatialConvolution(12, 24, 3, 3, 1, 1, 0, 0)) self.net:add(nn.ReLU()) self.net:add(pool_layer2) self.net:add(nn.Reshape(24 * 14 * 14)) self.net:add(nn.Linear(24 * 14 * 14, 1568)) self.net:add(nn.Linear(1568, 24 * 14 * 14)) self.net:add(nn.Reshape(24, 14, 14)) self.net:add(nn.SpatialConvolution(24, 12, 3, 3, 1, 1, 0, 0)) self.net:add(nn.ReLU()) self.net:add(nn.SpatialMaxUnpooling(pool_layer2)) self.net:add(nn.SpatialConvolution(12, 12, 3, 3, 1, 1, 0, 0)) self.net:add(nn.ReLU()) self.net:add(nn.SpatialMaxUnpooling(pool_layer1)) self.net:add(nn.SpatialConvolution(12, 3, 3, 3, 1, 1, 0, 0)) self.net = self.net:cuda() end

Folie 26

• Thank you