Download as PDF, PPTX
![Deep Learning - Basics
Recurrent Neural Nets (RNN) – Generating Text
To generate text, we feed a character into the trained RNN and get a distribution
over what characters are likely to come next (red = likely). We sample from this
distribution, and feed it right back in to get the next letter.
This highlighted neuron gets very excited (green = excited, blue = not excited) when
the RNN is inside the [[ ]] markdown environment and turns off outside of it.
The RNN is likely using this neuron to remember if it is inside a URL or not.](https://image.slidesharecdn.com/8bc3f575-c905-4999-a322-79d6db9f8755-160622071903/75/Deep-Learning-42-2048.jpg)
![Deep Learning - Basics
Recurrent Neural Nets (RNN) – Generating Text
To generate text, we feed a character into the trained RNN and get a distribution
over what characters are likely to come next (red = likely). We sample from this
distribution, and feed it right back in to get the next letter.
This highlighted neuron gets very excited (green = excited, blue = not excited) when
the RNN is inside the [[ ]] markdown environment and turns off outside of it.
The RNN is likely using this neuron to remember if it is inside a URL or not.](https://crownmelresort.com/image.slidesharecdn.com/8bc3f575-c905-4999-a322-79d6db9f8755-160622071903/75/Deep-Learning-42-2048.jpg)
Deep Learning
- 1. Deep Learning The Past,Present and Future of Artificial Intelligence
- 2. Interest Google NGRAM &Google Trends
- 3. Superintelligence A superintelligence isany intellect that vastly outperforms the best human brains in practically every field, including scientific creativity, general wisdom, and social skills
- 4. Hype or Reality? Hawking,Musk and Gates Superintelligence may be the last invention humans ever need to make, provided that the machine is docile enough to tell us how to keep it under control – Nick Bostrom (University of Oxford – AI Philosopher) The pace of progress in artificial general intelligence is incredible fast. With artificial intelligence we’re summoning the demon – Elon Musk (CEO Tesla & SpaceX) First the machines will do a lot of jobs for us and a few decades after that though the intelligence is strong enough to be a concern – Bill Gates The advent of super intelligent AI would be either the best or the worst thing ever to happen to humanity – Stephen Hawking (Physicist)
- 5. Hype or Reality? AIresearches There's a big difference between intelligence and sentience. Our software is becoming more intelligent, but that does not imply it is about to become sentient – Andrew Ng (Stanford & Baidu) The A.I. community as a whole is a long way away from building anything that could be a concern to the general public – Dileep George (Co-Founder Vicarious) Our systems can outperform humans in very narrow domains, but human-level general A.I. is several decades away – Yann LeCun (New York University & Facebook) For a very long time it will be a complementary tool that human scientists and human experts can use to help them with the things that humans are not naturally good – Demis Hassabis (Co-Founder DeepMind)
- 6.
- 7. Hype or Reality? Google Machinelearning is a core transformative way by which we are rethinking everything we are doing – Sundar Pichai (CEO Google)
- 8.
- 9. Hype or Reality? NIPS(Computational Neuroscience Conference) Growth
- 10. What is ArtificialIntelligence? Computers are already more intelligent than humans on many tasks, including remembering things, doing arithmetic, doing calculus, trading stocks, landing aircraft. – Thomas Dietterich (Oregon State University) Once something becomes commonplace, it's demystified, and it doesn't feel like the magical intelligence that we see in humans – Carlos Guestrin (University of Washington) You can have intelligent machines that have no self-conscience, no ego, and have no self-preservation instinct – Yoshua Bengio (University of Montreal) Intelligence is the ability to perform well across wide range of tasks, even some the system has never seen before. – Demis Hassabis (DeepMind)
- 11. What is ArtificialIntelligence? Artificial Intelligence Output: Movement Text Input: Sensors Data
- 12. What is ArtificialIntelligence? Artificial Narrow Intelligence (ANI): Machine intelligence that equals or exceeds human intelligence or efficiency at a specific task. Artificial General Intelligence (AGI): A machine with the ability to apply intelligence to any problem, rather than just one specific problem (human-level intelligence). Artificial Superintelligence (ASI): An intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.
- 13. Machine Learning -Basics Introduction Machine Learning is a type of Artificial Intelligence that provides computers with the ability to learn without being explicitly programmed. Machine Learning Algorithm Learned Model Data Prediction Labeled Data Training Prediction Provides various techniques that can learn from and make predictions on data
- 14. Machine Learning -Basics Learning Approaches Supervised Learning: Learning with a labeled training set Example: email spam detector with training set of already labeled emails Unsupervised Learning: Discovering patterns in unlabeled data Example: cluster similar documents based on the text content Reinforcement Learning: learning based on feedback or reward Example: learn to play chess by winning or losing
- 15. Machine Learning -Basics Problem Types Regression (supervised – predictive) Classification (supervised – predictive) Anomaly Detection (unsupervised– descriptive) Clustering (unsupervised – descriptive)
- 16. Machine Learning -Basics Algorithms Comparison - Classification
- 17. What is DeepLearning? Part of the machine learning field of learning representations of data. Exceptional effective at learning patterns. Utilizes learning algorithms that derive meaning out of data by using a hierarchy of multiple layers that mimic the neural networks of our brain. If you provide the system tons of information, it begins to understand it and respond in useful ways.
- 18. Inspired by theBrain The first hierarchy of neurons that receives information in the visual cortex are sensitive to specific edges while brain regions further down the visual pipeline are sensitive to more complex structures such as faces. Our brain has lots of neurons connected together and the strength of the connections between neurons represents long term knowledge. One learning algorithm hypothesis: all significant mental algorithms are learned except for the learning and reward machinery itself.1
- 19. Why Deep Learning? Applications Speech Recognition Computer Vision NaturalLanguage Processing
- 20. A brief History Along time ago… 1974 Backpropagation awkward silence (AI Winter) 1995 SVM reigns Convolution Neural Networks for Handwritten Recognition 1998 2006 Restricted Boltzmann Machine 1958 Perceptron 1969 Perceptron criticized Google Brain Project on 16k Cores 2012 2012 AlexNet wins ImageNet
- 21. A brief History TheBig Bang aka “One net to rule them all” ImageNet: The “computer vision World Cup”
- 22. A brief History TheBig Bang aka “One net to rule them all” Deep Learning in Speech Recognition
- 23. What changed? Old winein new bottles Big Data (Digitalization) Computation (Moore’s Law, GPUs) Algorithmic Progress
- 24. The Big Players SuperstarResearchers Geoffrey Hinton: University of Toronto & Google Yann LeCun: New York University & Facebook Andrew Ng: Stanford & Baidu Yoshua Bengio: University of Montreal Jürgen Schmidhuber: Swiss AI Lab & NNAISENSE
- 25.
- 26.
- 27. Deep Learning -Basics No more feature engineering Feature Engineering Traditional Learning AlgorithmInput Data Costs lots of time Deep Learning AlgorithmInput Data
- 28. Deep Learning -Basics Architecture A deep neural network consists of a hierarchy of layers, whereby each layer transforms the input data into more abstract representations (e.g. edge -> nose -> face). The output layer combines those features to make predictions.
- 29. Deep Learning -Basics What did it learn? Edges Nose, Eye… Faces
- 30. Deep Learning -Basics Artificial Neural Networks Consists of one input, one output and multiple fully-connected hidden layers in- between. Each layer is represented as a series of neurons and progressively extracts higher and higher-level features of the input until the final layer essentially makes a decision about what the input shows. The more layers the network has, the higher- level features it will learn.
- 31. Deep Learning -Basics The Neuron An artificial neuron contains a nonlinear activation function and has several incoming and outgoing weighted connections. Neurons are trained to filter and detect specific features or patterns (e.g. edge, nose) by receiving weighted input, transforming it with the activation function und passing it to the outgoing connections.
- 32. Deep Learning -Basics Non-linear Activation Function Non-linearity is needed to learn complex (non-linear) representations of data, otherwise the NN would be just a linear function. Most deep networks use ReLU - max(0,x) - nowadays for hidden layers, since it trains much faster, is more expressive than logistic function and prevents the gradient vanishing problem.
- 33. Deep Learning -Basics The Training Process Forward it trough the network to get predictionsSample labeled data Backpropagate the errors Update the connection weights Learns by generating an error signal that measures the difference between the predictions of the network and the desired values and then using this error signal to change the weights (or parameters) so that predictions get more accurate.
- 34. Deep Learning -Basics Gradient Descent Gradient Descent finds the (local) the minimum of the cost function (used to calculate the output error) and is used to adjust the weights.
- 35. Deep Learning -Basics Data transformation in other dimensions A neural network is transforming the data into other dimensions to solve the specified problem.
- 36. Deep Learning -Basics Deep Autoencoders Composed of two symmetrical deep-belief networks. The encoding network learns to compresses the input to a condensed vector (dimensionality reduction). The decoding network can be used to reconstruct the data. Topic Modeling: Document in a collection is converted to a Bag-of- Words and transformed to a compressed feature vector using an autoencoder. The distance from every other document-vector can be measured and nearby document-vectors fall under the same topic.
- 37. Deep Learning -Basics Convolutional Neural Nets (CNN) Convolutional Neural Networks learn a complex representation of visual data using vast amounts of data. They are inspired by the human visual system and learn multiple layers of transformations, which are applied on top of each other to extract a progressively more sophisticated representation of the input. Every layer of a CNN takes a 3D volume of numbers and outputs a 3D volume of numbers. E.g. Image is a 224*224*3 (RGB) cube and will be transformed to 1*1000 vector of probabilities.
- 38. Deep Learning -Basics Convolutional Neural Nets (CNN) Convolution layer is a feature detector that automagically learns to filter out not needed information from an input by using convolution kernel. Pooling layers compute the max or average value of a particular feature over a region of the input data (downsizing of input images). Also helps to detect objects in some unusual places and reduces memory size.
- 39. Deep Learning -Basics Recurrent Neural Nets (RNN) general computers which can learn algorithms to map input sequences to output sequences RNNs are general computers which can learn algorithms to map input sequences to output sequences (flexible-sized vectors). The output vector’s contents are influenced by the entire history of inputs. general computers which can learn algorithms to map input sequences to output sequences State-of-the-art results in time series prediction, adaptive robotics, handwriting recognition, image classification, speech recognition, stock market prediction, and other sequence learning problems. Everything can be processed sequentially.
- 40. Deep Learning -Basics Long Short-Term Memory RNN (LSTM) A Long Short-Term Memory (LSTM) network is a particular type of recurrent network that works slightly better in practice, owing to its more powerful update equation and some appealing back propagation dynamics. general computers which can learn algorithms to map input sequences to output sequences The LSTM units give the network memory cells with read, write and reset operations. During training, the network can learn when it should remember data and when it should throw it away. general computers which can learn algorithms to map input sequences to output sequences Well-suited to learn from experience to classify, process and predict time series when there are very long time lags of unknown size between important events.
- 41. Deep Learning -Basics Recurrent Neural Nets (RNN) – Generating Text To train the RNN, insert characters sequentially and predict the probabilities of the next letter. Backpropagate error and update RNN’s weights to increase the confidence of the correct letter (green) and decrease the confidence of all other letters (red). Trained on structured Wikipedia markdown. Network learns to spell English words completely from scratch and copy general syntactic structures.
- 42. Deep Learning -Basics Recurrent Neural Nets (RNN) – Generating Text To generate text, we feed a character into the trained RNN and get a distribution over what characters are likely to come next (red = likely). We sample from this distribution, and feed it right back in to get the next letter. This highlighted neuron gets very excited (green = excited, blue = not excited) when the RNN is inside the [[ ]] markdown environment and turns off outside of it. The RNN is likely using this neuron to remember if it is inside a URL or not.
- 43. Deep Learning -Basics Natural Language Processing – Embeddings Embeddings are used to turn textual data (words, sentences, paragraphs) into high- dimensional vector representations and group them together with semantically similar data in a vectorspace. Thereby, computer can detect similarities mathematically.
- 44. Deep Learning -Basics Natural Language Processing – Word2Vec Word2Vec is an unsupervised learning algorithm for obtaining vector representations for words. These vectors were trained for a specific domain on a very large textual data set. GloVe is a better performing alternative. It detects similarities mathematically by grouping the vectors of similar words together. All it needs is words co-occurance in the given corpus.
- 45. Deep Learning -Basics Natural Language Processing – Word2Vec Woman – Man ≈ Aunt - Uncle King - Male + Female ≈ Queen Human - Animal ≈ Ethics
- 46. Deep Learning -Basics Natural Language Processing – Thought Vectors Thought vectors is a way of embedding thoughts in vector space. Their features will represent how each thought relates to other thoughts. By reading every document on the web, computers might be able to reason like humans do by mimicking the thoughts expressed in content. A neural machine translation is trained on bilingual text using a encoder and decoder RNN. For translation, the input sentence is transformed into a thought vector. This vector is used to reconstruct the given thought in another language.
- 47. Deep Learning -Basics Usage Requirements Large data set with good quality (input-output mappings) Measurable and describable goals (define the cost) Enough computing power (AWS GPU Instance) Excels in tasks where the basic unit (pixel, word) has very little meaning in itself, but the combination of such units has a useful meaning
- 48. Deep Learning inGoogle Products RankBrain (Search): In few months, RankBrain has become the third-most important signal contributing to the search result. Speech Recognition (Google Now): 30% reduction in Word Error Rate for English. Biggest single improvement in 20 years of speech research. Visual Translation (Google Translate): Real-time visual translation of 20 more languages all on the phone and without an Internet connection using a deep neural net.
- 49. Deep Learning inGoogle Products Video Thumbnails (YouTube): Compared to the previous thumbnails, the DNN- powered model selects frames with much better quality Hotword Detection (Google Now): 25% better than old system in noisy conditions.
- 50. Deep Learning inGoogle Products Voicemail transcriptions (Google Voice / Project Fi): Using a long short-term memory deep recurrent neural network the transcription errors was cut by 49%. Photo Search (Google Photos): Error rate of just 5% which is as good as humans performing the same task. House number Recognition (Street View): Performance is comparable to, and in some cases exceeds, that of human operators.
- 51. Deep Learning inGoogle Products Spam Filter (Gmail): Rejects 99.9% of spam, while mistaking legitimate messages for spam only 0.05% of the time. a neural net to suggest automatic replies to emails that you can send with a single click. Smart Reply (Inbox): A neural net to suggest automatic replies to emails that you can send with a single click.
- 52. Deep Learning inGoogle Research DeepMind Deep Q-Learning Outperforms humans in over 30 Atari games just by receiving the pixels on the screen with the goal to maximize the score (Reinforcement Learning)
- 53. Deep Learning inGoogle Research DeepMind Deep Q-Learning Deep Q-Learning (DQN) is a model-free approach to reinforcement learning using deep networks in environments with discrete action choices
- 54. Deep Learning inGoogle Research DeepMind Deep Q-Learning Policy distillation: Extracts the learned state (policy) of a reinforcement learning agent (teacher) and trains a new network (student) that performs at the expert level while being dramatically smaller and more efficient. Single-task policy distillation Multi-task policy distillation
- 55. Deep Learning inGoogle Research DeepStereo – 2D to 3D image conversion DeepStereo: Turns images from Street View into a 3D space that shows unseen views from different angels by figuring out the depth and color of each pixel.
- 56. Deep Learning inGoogle Research Neural Image Caption Generator Neural Image Caption Generator generates fitting natural-language captions only based on the pixels by combining a vision CNN and a language-generating RNN. A man flying through the air while riding a skateboard Two pizzas sitting on top of a stove top oven A close up of a child holding a stuffed animal
- 57. Deep Learning inGoogle Research Neural Conversational Model Neural Chatbot: Predicts the next sentence given the previous sentences in a conversation. The LSTM remembers facts, understands contexts and performs common sense reasoning in the trained domain. Human: what is the purpose of life ? Machine: to serve the greater good . Human: what is the purpose of living ? Machine: to live forever Human: what is the purpose of existence? Machine: to find out what happens when we get to the planet earth Human: where are you now ? . Machine: i ’m in the middle of nowhere . Human: what is the purpose of dying ? Machine: to have a life Human: what is the purpose of being intelligent ? Machine: to find out what it is. Human: what is the purpose of emotions ? Machine: I don ’t know . Philosophical Q&A – Network trained with OpenSubtitles (movie subtitles)
- 58. Deep Learning inGoogle Research … and more Learns Grammar. Trained on a large synthetic corpus that was annotated using existing parser. Outperforms existing rule- based parsers. Learns Algorithms: combinatorial optimization problems Unconfirmed: Self-Driving Cars, Robots, Drones, Advertisement Placement, Medical Data
- 59. Deep Learning inGoogle Research DeepDream – Inceptionism Inceptionism helps to understand and visualize what a neural network has learned during training “No picture of a dumbbell is complete without a muscular weightlifter” Let the network over- interpret whatever it detects in a selected layer (e.g. edges)
- 60. Deep Learning inGoogle Research DeepDream – Inceptionism By choosing higher level layers, more sophisticated features or even whole objects tend to emerge.
- 61. Deep Learning inGoogle Research DeepDream – A Tool for Artists?
- 62. Deep Learning -Demos Deep Art – morph images into paintings
- 63. Deep Learning -Demos Handwriting & Music Generation This LSTM recurrent neural network is able to generate highly realistic cursive handwriting in a wide variety of styles, simply by predicting one data point at a time. Recurrent neural network generates the next notes of "Let It Go"
- 64. Deep Learning -Demos Image Segmentation & Machine Translation Recognize objects in image and recover the 2D outline of the object. Neural machine translation
- 65. Deep Learning -Demos Image Tagging & Captioning
- 66. Deep Learning -Demos Image Generation – Bedrooms
- 67. Deep Learning -Demos Image Generation – Album Covers
- 68. Deep Learning -Tools Its all Open Source
- 69. Deep Learning -Tools Computing is affordable AWS EC2 GPU Spot Instance: g2.2xlarge - $0.0782 per Hour The DIGITS DevBox combines the world’s best hardware (4 GPUs), software, and systems engineering for deep learning in a powerful solution that can fit under your desk. Cost: $15k
- 70. Outlook Goal-based AI Computer Program Data Output Traditional Programming Computer Output Data Program MachineLearning ComputerGoal Program Goal-based AI
- 71. Outlook NVIDIA Pascal NVIDIA’s PascalGPU architecture will accelerate deep learning applications up to 10X beyond the speed of its current-generation Maxwell processors.
- 72. Outlook Artificial Quantum Intelligence QuantumArtificial Intelligence Lab is a joint initiative of NASA and Google to study how quantum computing might advance machine learning. This type of computing may provide the most creative and parallelized problem-solving process under the known laws of physics. Quantum computers handle what are called quantum bits or qubits that can readily have a value of one or zero or anything in between. Quantum computing represents a paradigm shift, a radical change in the way we do computing and at a scale that has unimaginable power – Eric Ladizinsky (Co-founder D-Wave)
- 73. Outlook What’s the bestway to communicate with AIs? The AIs are going to end up needing to use some form of symbolic language that represents concepts abstractly, Such a language could be used as a bridge between human understanding and AI.
- 74. Outlook Neuromorphic Chips IBM TrueNorthis a brain-inspired computer chip that implements networks of integrate-and-fire spiking artificial neurons and uses only a tiny 70 mw of power –orders of magnitude less energy than traditional chips. The system is designed to be able to run deep-learning algorithms. 1 million Programmable Neurons 256 million Programmable Synapses 4096 Neurosynaptic Cores
- 75. Outlook The Enabler Many ofthe biggest problems facing humanity today, like curing diseases or addressing climate change, would be vastly easier with the help of AI. The Big Bang for Self-Driving Cars (10-15 years). Fully autonomous taxi systems will change the paradigm of the need to own a car. AI will fuel a medical revolution (5-10 years) by enabling far more efficient drug discovery, diagnoses and research.
- 76. Significant advances indeep reinforcement and unsupervised learning Bigger and more complex architectures based on various interchangeable modules/techniques Deeper models that can learn from much fewer training cases Harder problems such as video understanding and natural language processing will be successfully tackled by deep learning algorithms Outlook Deep Learning
- 77. Takeaways Machines that learnto represent the world from experience. Deep Learning is no magic! Just statistics in a black box, but exceptional effective at learning patterns. We haven’t figured out creativity and human-empathy. Transitioning from research to consumer products. Will make the tools you use every day work better, faster and smarter.
- 78.