Download as PDF, PPTX
![Data Analysis
46
Defines training vs validation
Class uint per class [1,26]](https://image.slidesharecdn.com/publishintroductiontodeeplearninginpythonandmatlab1-160502102437/75/Introduction-to-deep-learning-in-python-and-Matlab-46-2048.jpg)
![Optimization
SoftMax
Score (-∞,∞) → probabilities [0,1]
52https://classroom.udacity.com/courses/ud730](https://image.slidesharecdn.com/publishintroductiontodeeplearninginpythonandmatlab1-160502102437/75/Introduction-to-deep-learning-in-python-and-Matlab-52-2048.jpg)
![Data Analysis
46
Defines training vs validation
Class uint per class [1,26]](https://crownmelresort.com/image.slidesharecdn.com/publishintroductiontodeeplearninginpythonandmatlab1-160502102437/75/Introduction-to-deep-learning-in-python-and-Matlab-46-2048.jpg)
![Optimization
SoftMax
Score (-∞,∞) → probabilities [0,1]
52https://classroom.udacity.com/courses/ud730](https://crownmelresort.com/image.slidesharecdn.com/publishintroductiontodeeplearninginpythonandmatlab1-160502102437/75/Introduction-to-deep-learning-in-python-and-Matlab-52-2048.jpg)
More Related Content
![[251] implementing deep learning using cu dnn](https://cdn.slidesharecdn.com/ss_thumbnails/215implementingdeeplearningusingcudnn-150915052020-lva1-app6892-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC 2016] 系列活動：李宏毅 / 一天搞懂深度學習](https://cdn.slidesharecdn.com/ss_thumbnails/1-160521014039-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Introduction to deep learning in python and Matlab
Introduction to deep learning in python and Matlab
- 1. Introduction to Hands-on DeepLearning Imry Kissos Algorithm Researcher
- 2. Outline ● Problem Definition ●Motivation ● Training a Regression DNN ● Training a Classification DNN ● Open Source Packages ● Summary + Questions 2
- 3.
- 4. Tutorial ● Goal: Detectfacial landmarks on (normal) face images ● Data set provided by Dr. Yoshua Bengio ● Tutorial code available: https://github.com/dnouri/kfkd-tutorial/blob/master/kfkd.py 4
- 5. Flow 5 Predict Points on TestSet Train Model General Train Model “Nose Tip” Train Model “Mouth Corners”
- 6. Flow 6 Train Images Train Points FitTrained Net
- 7.
- 8. Python DL Framework Wrapperto Lasagne Theano extension for Deep Learning Define, optimize, and evaluate mathematical expressions Efficient Cuda GPU for DNN 8 Low Level High Level HW Supports: GPU & CPU OS: Linux, OS X, Windows
- 9. Training a DeepNeural Network 1. Data Analysis 2. Architecture Engineering 3. Optimization 4. Training the DNN 9
- 10. Training a DeepNeural Network 1. Data Analysis a. Exploration + Validation b. Pre-Processing c. Batch and Split 2. Architecture Engineering 3. Optimization 4. Training the DNN 10
- 11. Data Exploration +Validation Data: ● 7K gray-scale images of detected faces ● 96x96 pixels per image ● 15 landmarks per image (?) Data validation: ● Some Landmarks are missing 11 1
- 12.
- 13. Batch - - t -train batch - validation batch - - test batch ⇐One Epoch’s data 13train/valid/test splits are constant
- 14. Train / ValidationSplit 14 Classification - Train/Validation preserve classes proportion
- 15. Training a DeepNeural Network 1. Data Analysis 2. Architecture Engineering a. Layers Definition b. Layers Implementation 3. Optimization 4. Training 15
- 16. Architecture 16 X Y Conv PoolDense Output
- 17.
- 18.
- 19.
- 20.
- 21.
- 22. Training a DeepNeural Network 1. Data Analysis 2. Architecture Engineering 3. Optimization a. Back Propagation b. Objective c. SGD d. Updates e. Convergence Tuning 4. Training the DNN 22
- 23. Back Propagation Forward Path 23 ConvDense X Y Output Points
- 24. Back Propagation Forward Path 24 XY Conv Output PointsDense X Y Training Points
- 25. Back Propagation Backward Path 25 XY Conv Dense
- 26. Back Propagation Update 26 Conv Dense ForAll Layers:
- 27.
- 28. S.G.D 28Updates the networkafter each batch
- 29.
- 30. Adjusting Learning Rate& Momentum 30 Linear in epoch
- 31. Convergence Tuning 31 stops accordingto validation loss returns best weights
- 32. Training a DeepNeural Network 1. Data Analysis 2. Architecture Engineering 3. Optimization 4. Training the DNN a. Fit b. Fine Tune Pre-Trained c. Learning Curves 32
- 33. Fit 33 Loop over testbatchs Forward Loop over train batchs Forward+BackProp
- 34. Fine Tune Pre-Trained fgd 34 changeoutput layer load pre-trained weight fine tune specialist
- 35. Learning Curves Loop over6 Nets: 35 Epochs
- 36. Learning Curves Analysis 36 Net1 Net 2 OverfittingConvergence Jittering EpochsEpochs RMSE RMSE
- 37. Part 1 Summary Traininga DNN: 37
- 38. Python ● Rich eco-system ●State-of-the-art ● Easy to port from prototype to production 38 https://github.com/yoavram/Py4Eng
- 39. Python DL Framework 39 Theanobased Packages
- 40.
- 41.
- 42. Outline ● Problem Definition ●Motivation ● Training a regression DNN ● Training a classification DNN ● Improving the DNN ● Open Source Packages ● Summary 42
- 43. Matlab DL Framework OpenSource CNN Toolbox by Numerical computing using Parallel Computing Toolbox Efficient Cuda GPU for DNN 43 Low Level High Level HW Supports: GPU & CPU OS: Linux, OS X, Windows
- 44. Problem Statement Classify a,b, …, z images into 26 classes: 44http://www.robots.ox.ac.uk/~vgg/practicals/cnn/ Bonus - OCR:
- 45. Training a DeepNeural Network 1. Data Analysis 2. Training the DNN 3. Architecture Engineering 4. Optimization 45
- 46. Data Analysis 46 Defines trainingvs validation Class uint per class [1,26]
- 47.
- 48.
- 49. Customized Batch Loading 49 Howwould you add Data Augmentation ?
- 50. trainOpts 50 Start from lastiter if interrupted
- 51. initializeCharCnn() Net Architecture Layers: ● Conv ●Pool ● Conv ● Pool ● Conv ● Relu ● Conv ● SoftMaxLoss 51 %f is the W initial std
- 52. Optimization SoftMax Score (-∞,∞) →probabilities [0,1] 52https://classroom.udacity.com/courses/ud730
- 53. One Hot Encoding Encodeclass labels 53
- 54. Cross Entropy Distance measurebetween S(Y) and Labels 54
- 55. Cross Entropy Distance measurebetween S(Y) and Labels 55 D(S,L) is a positive scalar t - index of ground-truth class
- 56. Cross Entropy Distance measurebetween S(Y) and Labels 56 In vl_nnloss.m:
- 57.
- 58. Minimize Loss Loss=average crossentropy 58
- 59. Minimize loss - learningrate 59
- 60. Error Rate TopK -Target label is one of the top K predictions The Error Rate is: 60
- 61. Loss & ErrorConvergence 61 Loss Error Rate
- 62.
- 63.
- 64.
- 65. Beyond Training 1. Traininga classification DNN 2. Improving the DNN a. Analysis Capabilities b. Augmentation 3. Open Source Packages 4. Summary 65
- 66. Basic VS AdvancedMode 66 Basic Advance
- 67. Improving the DNN Verytempting: ● >1M images ● >1M parameters ● Large gap: Theory ↔ Practice ⇒Brute force experiments?! 67
- 68. Analysis Capabilities 1. Theoreticalexplanation a. Eg. dropout/augmentation decrease overfit 2. Empirical claims about a phenomena a. Eg. normalization helps convergence 3. Numerical understanding a. Eg. exploding / vanishing updates 68
- 69. Reduce Overfitting Solution: Data Augmentation 69 Net1 Net 2 Overfitting Epochs
- 70. Data Augmentation Horizontal FlipPerturbation 70 1
- 71. Convergence Challenges 71 Need tomonitor forward + backward path EpochsEpochs RMSE Data ErrorNormalization
- 72. Deal with NaN 1.If in first 100 iterations a. Learning rate is too high 2. Beyond 100 iterations a. Gradient explosion i. Consider gradient clipping b. Illegal math operation i. SoftMax: inf/inf ii. Division by zero by one of your customized layers 72http://russellsstewart.com//notes/0.html
- 73. The Net Doesn’tLearn Anything 1. Training loss does not reduce after first 100 iterations a. Reduce the training size to 10 instances (images) to overfit it i. Achieve 100% training accuracy on a small portion of data b. Change batch size to 1 to and monitor the error per batch c. Solve the simplest version of your problem 73 http://russellsstewart.com//notes/0.html
- 74. Beyond Training 1. Traininga classification DNN 2. Improving the DNN 3. Open Source Packages a. DL Open Source Packages b. Effort Estimation 4. Summary 74
- 75. Tips from OtherPackages Torch code organization Caffe’s separation configuration ↔code NeuralNet → YAML text format defining experiment’s configuration 75
- 76. DL Open SourcePackages 76 Caffe & MatConvNet for applications Torch, TensorFlow and Theano for research on DL http://fastml.com/torch-vs-theano/ Simple dnnComplex dnn
- 77. Disruptive Effort Estimation FeatureEng Deep Learning 77Modest SW Infra Huge SW Infra
- 78. Summary ● Dove intoTraining a DNN ● Presented Analysis Capabilities ● Reviewed Open Source Packages 78
- 79. References Hinton Coursera NeuronalNetwork https://www.coursera.org/course/neuralnets Udacity Tensor Flow course https://classroom.udacity.com/courses/ud730 Technion Deep Learning course http://moodle.technion.ac.il/course/view.php?id=4128 Oxford Deep Learning course https://www.youtube.com/playlist?list=PLE6Wd9FR--EfW8dtjAuPoTuPcqmOV53Fu CS231n CNN for Visual Recognition http://cs231n.github.io/ Deep Learning Book http://www.deeplearningbook.org/ 79
- 80.