Download as PDF, PPTX
![• The tanh(z) function is a rescaled version of
the sigmoid, and its output range is [ − 1,1]
instead of [0,1].
69](https://image.slidesharecdn.com/neuralnetworkanddeeplearning-150915170327-lva1-app6892/75/From-neural-networks-to-deep-learning-69-2048.jpg)
![• The tanh(z) function is a rescaled version of
the sigmoid, and its output range is [ − 1,1]
instead of [0,1].
69](https://crownmelresort.com/image.slidesharecdn.com/neuralnetworkanddeeplearning-150915170327-lva1-app6892/75/From-neural-networks-to-deep-learning-69-2048.jpg)
Uploaded byViet-Trung TRAN
From neural networks to deep learning
The document discusses the history and development of artificial neural networks and deep learning. It describes early neural network models like perceptrons from the 1950s and their use of weighted sums and activation functions. It then explains how additional developments led to modern deep learning architectures like convolutional neural networks and recurrent neural networks, which use techniques such as hidden layers, backpropagation, and word embeddings to learn from large datasets.
More Related Content
Similar to From neural networks to deep learning
From neural networks to deep learning
- 1. From Artificial Neural Networksto Deep learning Viet-Trung Tran 1
- 2.
- 3.
- 4.
- 5.
- 6. Perceptron • Rosenblatt 1957 • input signals x1, x2, • bias x0 = 1 • Net input = weighted sum = Net(w,x) • Activation/transfer func = f(Net(w,x)) • output weighted sum step func1on 6
- 7. Weighted Sum andBias • Weighted sum • Bias 7
- 8.
- 9. Hard-limiter function • Hard-limiter – Thresholdfunction – Discontinuous function – Discontinuous derivative 9
- 10. Threshold logic function • Saturating linear function • Contiguous function • Discontinuous derivative 10
- 11. Sigmoid function • Mostpopular • Output (0,1) • Continuous derivatives • Easy to differentiate 11
- 12. Artificial neural network– ANN structure • Number of input/output signals • Number of hidden layers • Number of neurons per layer • Neuron weights • Topology • Biases 12
- 13. Feed-forward neural network • connections between the units do not form a directed cycle 13
- 14. Recurrent neural network • A class of artificial neural network where connections between units form a directed cycle 14
- 15.
- 16. Neural network learning • 2 types of learning – Parameter learning • Learn neuron weight connections – Structure learning • Learn ANN structure from training data 16
- 17. Error function • Consideran ANN with n neurons • For each learning example (x,d) – Training error caused by current weight w • Training error caused by w for entire learning examples 17
- 18.
- 19.
- 20. Parameter learning: back propagationof error • Calculate total error at the top • Calculate contributions to error at each step going backwards 20
- 21. Back propagation discussion • Initial weights • Learning rate • Number of neurons per hidden layers • Number of hidden layers 21
- 22.
- 23.
- 24.
- 25.
- 26.
- 27. Learning from taggeddata • @Andrew Ng 27
- 28. 2006 breakthrough • Moredata • Faster hardware: GPU’s, multi-core CPU’s • Working ideas on how to train deep architectures 28
- 29.
- 30.
- 31.
- 32. Deep Learning trends • @Andrew Ng 32
- 33.
- 34.
- 35. AI will transformthe internet • @Andrew Ng • Technology areas with potential for paradigm shift: – Computer vision – Speech recognition & speech synthesis – Language understanding: Machine translation; Web search; Dialog systems; …. – Advertising – Personalization/recommendation systems – Robotics • All this is hard: scalability, algorithms. 35
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42. Convolution • Convolution isa mathematical operation on two functions f and g, producing a third function that is typically viewed as a modified version of one of the original functions, 42
- 43. Convolutional neural networks • Conv Nets is a kind of neural network that uses many identical copies of the same neuron – Large number of neurons – Large computational models – Number of actual weights (parameters) to be learned fairly small 43
- 44. A 2D ConvolutionalNeural Network • a convolutional neural network can learn a neuron once and use it in many places, making it easier to learn the model and reducing error. 44
- 45. Structure of ConvNets • Problem – predict whether a human is speaking or not • Input: audio samples at different points in time 45
- 46. Simple approach • justconnect them all to a fully-connected layer • Then classify 46
- 47. A more sophisticatedapproach • Local properties of the data – frequency of sounds (increasing/decreasing) • Look at a small window of the audio sample – Create a group of neuron A to compute certain features – the output of this convolutional layer is fed into a fully- connected layer, F 47
- 48.
- 49.
- 50.
- 51. 2D convolutional neuralnetworks 51
- 52.
- 53.
- 54.
- 55. Group of neurons:A • Bunch of neurons in parallel • all get the same inputs and compute different features. 55
- 56. Network in Network(Lin et al. (2013) 56
- 57. Conv Nets breakthroughsin computer vision • Krizehvsky et al. (2012) 57
- 58. Diferent Levels ofAbstraction 58
- 59.
- 60.
- 61.
- 62. Recurrent Neural Networks(RNN) have loops • A loop allows information to be passed from one step of the network to the next. 62
- 63. Unroll RNN • recurrentneural networks are intimately related to sequences and lists. 63
- 64. Examples • predict thelast word in “the clouds are in the sky" • the gap between the relevant information and the place that it’s needed is small • RNNs can learn to use the past information 64
- 65. • “I grewup in France… I speak fluent French.” • As the gap grows, RNNs become unable to learn to connect the information. 65
- 66. LONG SHORT TERMMEMORY NETWORKS LSTM Networks 66
- 67. LSTM networks • ASpecial kind of RNN • Capable of learning long-term dependencies • Structure in the form of a chain of repeating modules of neural network 67
- 68. RNN • repeating modulehas a very simple structure, such as a single tanh layer 68
- 69. • The tanh(z)function is a rescaled version of the sigmoid, and its output range is [ − 1,1] instead of [0,1]. 69
- 70. LSTM networks • Repeatingmodule consists of four neuron, interacting in a very special way 70
- 71. Core idea behindLSTMs • The key to LSTMs is the cell state, the horizontal line running through the top of the diagram. • The cell state runs straight down the entire chain, with only some minor linear interactions • Easy for information to just flow along it unchanged 71
- 72. Gates • The abilityto remove or add information to the cell state, carefully regulated by structures called gates • Sigmoid – How much of each component should be let through. – Zero means nothing through – One means let everything through • An LSTM has three of these gates 72
- 73. LSTM step 1 • decide what information we’re going to throw away from the cell state • forget gate layer 73
- 74. LSTM step 2 • decide what new information we’re going to store in the cell state • input gate layer 74
- 75. LSTMs step 3 • update the old cell state, Ct−1, into the new cell state Ct 75
- 76. LSTMs step 4 • decide what we’re going to output 76
- 77.
- 78.
- 79.
- 80.
- 81. RECURRENT NEURAL NETWORKS WITHWORD EMBEDDINGS 81
- 82.
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89.
- 90.
- 91.
- 92. • Inspired bythe architectural depth of the brain, researchers wanted for decades to train deep multi- layer neural networks. • No successful attempts were reported before 2006 …Exception: convolutional neural networks, LeCun 1998 • SVM: Vapnik and his co-workers developed the Support Vector Machine (1993) (shallow • architecture). • Breakthrough in 2006! 92
- 93. 2006 breakthrough • Moredata • Faster hardware: GPU’s, multi-core CPU’s • Working ideas on how to train deep architectures 93
- 94. • Beat stateof the art in many areas: – Language Modeling (2012, Mikolov et al) – Image Recognition (Krizhevsky won 2012 ImageNet competition) – Sentiment Classification (2011, Socher et al) – Speech Recognition (2010, Dahl et al) – MNIST hand-written digit recognition (Ciresan et al, 2010) 94
- 95. Credits • Roelof Pieters,www.graph-technologies.com • Andrew Ng • http://colah.github.io/ 95