Downloaded 92 times
Uploaded byYan Xu
HML: Historical View and Trends of Deep Learning
The document provides a historical view and trends of deep learning. It discusses that deep learning models have evolved in several waves since the 1940s, with key developments including the backpropagation algorithm in 1986 and deep belief networks with pretraining in 2006. Current trends include growing datasets, increasing numbers of neurons and connections per neuron, and higher accuracy on tasks involving vision, NLP and games. Research trends focus on generative models, domain alignment, meta-learning, using graphs as inputs, and program induction.
More Related Content
Similar to HML: Historical View and Trends of Deep Learning
HML: Historical View and Trends of Deep Learning
- 1. Historical View andTrends of Deep Learning "DEEP LEARNING“ CHAPTER 1 1
- 2.
- 3. Survey: Topics youwant to learn 3 Deep Learning Reinforc ement Learning NLPForeca sting Ensem ble
- 4. HML 2018 Roadmap 1.Introduction (Chapter 1), Historical view and trends of deep learning – Yan Xu 2. Linear algebra and probability (Chapter 2&3) – Cheng Zhan 3. Numerical Computation and machine learning basics (Chapter 4&5) – Linda MacPhee-Cobb 4. Deep forward neural nets and regularization (Chapter 6&7) – Licheng Zhang 5 Quantum Machine Learning - Nicholas Teague 6. Optimization for training models (Chapter 8) 7. Convolutional Networks (Chapter 9) 8. Sequence modeling I (Chapter 10) 9. Sequence modeling II (Chapter 10) ...... 4
- 5. Outline • Representation Learning •Historical Waves • Current Trends of Deep Learning • Research Trends 5
- 6.
- 7. Illustration of DeepLearning Nested simple mappings 7
- 8. Computational Graphs Depth =3 Depth = 1 8
- 9.
- 10.
- 11. Historical Waves • Along and rich history. • The amount of available training data has increased. • Deep learning models have grown in size over time. • Deep learning has solved increasingly complicated applications with increasing accuracy. 11
- 12.
- 13.
- 14. Historical Waves McCulloch-Pitts neuron(1943) The perceptron (1958, 1962) ADALINE, stochastic gradient descent (1960) Neocognitron (1980) Distributed representation (1986) Back-propagation algorithm (1986) Convolutional neural network (1998) Sequence models (1991, 1994) Long Short Term Memory (LSTM) (1997) Deep belief network, pretraining (2006) Using GPUs for Deep Learning (2005, 2009) 14
- 15.
- 16. Current Trends: GrowingDatasets 16
- 17.
- 18.
- 19.
- 20.
- 21. SQuAD Challenge Stanford Question AnsweringD ataset (SQuAD) • the answer to every question is a segment of text from the corresponding reading passage from Wiki. • 100,000+ question-answer pairs on 500+ articles. ExactMatch 21
- 22.
- 23. Research Trends • Generativemodels • Domain alignment • Learning to Learn (Meta-Learning) • Neural networks and graphs • Program Induction Source: “Deep Learning: Practice and Trends”, NIPS 2017 23
- 24. Generative Models Generative ModelDiscriminative Model Naïve bayes Gaussian mixture Latent dirichlet allocation Generative adversarial networks Logistic regression Support vector machines Boosting Neural networks Deep Generative Models: Tutorial UAI 2017 https://danilorezendedotco m.files.wordpress.com/201 7/09/deepgenmodelstutori al.pdf 24
- 25.
- 26.
- 27. Neural Network andGraphs 27
- 28. Message Passing NeuralNetworks Predicting DFT with MPNNs (Gilmer et al, ICML 17) 13 properties DFT : Density functional theory 28
- 29. Program Induction RobustFill: Neural Program Learningunder Noisy I/O, 2017 29
- 30. Summary • Representation Learning •Historical Waves o ADALINE, stochastic gradient descent (1960) o Back-propagation algorithm (1986) o Deep belief network, pretraining (2006) • Current Trends of Deep Learning o Increasing data sets o Increasing number of neurons and number of connections per neuron o Increasing accuracy on various tasks in vision, NLP and game etc. • Research Trends o Generative models o Domain alignment o Meta learning o Graph as input o Program induction 30
- 31. References Deep Learning BookChapter 1 http://www.deeplearningbook.org/ NIPS 2017 slides and videos (Deep Learning: Practice and Trends): https://github.com/hindupuravinash/nips2017 Andrew L. Beam https://beamandrew.github.io/deeplearning/2017/02/23/deep_learnin g_101_part1.html 31
- 32. Thank You Slides: https://www.slideshare.net/xuyangela https://www.meetup.com/Houston-Machine-Learning/ Feel freeto message me if you want to lead a session! 32