Download as PDF, PPTX
![Atari[Nature,2015]
4](https://image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-4-2048.jpg)
![AlphaGo[Nature,2016]
5](https://image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-5-2048.jpg)
![Expectedfuturerewards
Any goal can be represented as a sum of intermediate rewards.
[ ∣ ] = [ + γ + + … ∣ ]∑
∞
t=0
γ
t
Rt St R0 R1 γ
2
R2 St
15](https://image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-16-2048.jpg)
![Updaterule
In rabbits, humans and machines we get the same algorithm:
while True:
Q[t] = Q[t-1] + alpha * (Q_target - Q[t-1])
27](https://image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-28-2048.jpg)
![Q-Learning[Watkins,1989]
The agent does not have a model of the environment.
Perform actions following a standard policy.
Predict using the target policy.
Which makes it an "o -policy", model-free method.
28](https://image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-29-2048.jpg)
![Network
Input: an image of shape [None, 42, 42, 4]
4 Conv2D 32 lters, 4x4 kernel
1 Hidden layer of size 256
1 Fully connected layer of size action_size
35](https://image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-36-2048.jpg)
![Atari[Nature,2015]
4](https://crownmelresort.com/image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-4-2048.jpg)
![AlphaGo[Nature,2016]
5](https://crownmelresort.com/image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-5-2048.jpg)
![Expectedfuturerewards
Any goal can be represented as a sum of intermediate rewards.
[ ∣ ] = [ + γ + + … ∣ ]∑
∞
t=0
γ
t
Rt St R0 R1 γ
2
R2 St
15](https://crownmelresort.com/image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-16-2048.jpg)
![Updaterule
In rabbits, humans and machines we get the same algorithm:
while True:
Q[t] = Q[t-1] + alpha * (Q_target - Q[t-1])
27](https://crownmelresort.com/image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-28-2048.jpg)
![Q-Learning[Watkins,1989]
The agent does not have a model of the environment.
Perform actions following a standard policy.
Predict using the target policy.
Which makes it an "o -policy", model-free method.
28](https://crownmelresort.com/image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-29-2048.jpg)
![Network
Input: an image of shape [None, 42, 42, 4]
4 Conv2D 32 lters, 4x4 kernel
1 Hidden layer of size 256
1 Fully connected layer of size action_size
35](https://crownmelresort.com/image.slidesharecdn.com/rlbasic-170706155846/75/Deep-Reinforcement-Learning-36-2048.jpg)
Uploaded byMeetupDataScienceRoma
Deep Reinforcement Learning
The document provides an overview of deep reinforcement learning (DRL), outlining its fundamental concepts such as Q-learning, policy, value functions, and the role of agents in decision-making. It discusses various aspects including the significance of delayed feedback, the use of neural networks, and methods for improving stability in learning processes. Additionally, it highlights tools, challenges, and resources available for further exploration in the field of DRL.
In this document
Powered by AI
Overview of Deep Reinforcement Learning (RL) and its significance in Machine Learning (ML). Outlined topics include framework, Q-Learning, and a demo.
Examples of RL applications: Atari games (Nature, 2015) and AlphaGo (Nature, 2016), showcasing RL's effectiveness.
Description of RL's decision-making process and its unique aspects: no supervisor, delayed feedback, context dependence.
Defining the environment and agent in RL, including example code for interaction with an environment using Gym.
Discusses assumptions regarding state transitions in RL, emphasizing the importance of current state info.
Concept of expected future rewards in RL where goals are described as a sum of intermediate rewards.
The three main tools in RL: Policy, Value function, and Model; defining agent behavior and environment representation.
Key design choices like balancing exploration and exploitation and finding the optimal policy to maximize rewards.
Different methods to approximate the value function in RL: Monte Carlo and Temporal Difference learning.
Introduction of Deep Q-Learning using neural networks, addressing issues such as data correlation and reward stability.
Innovative strategies from DeepMind for stable Q-learning, including neural network architecture for input images.
Key hyperparameters for RL models such as learning rate, reward clipping, and optimizer choice.
Various resources to learn more about RL and acknowledgment to the audience, concluding the presentation.
More Related Content
Similar to Deep Reinforcement Learning
![[Sponsored] C3.ai description](https://cdn.slidesharecdn.com/ss_thumbnails/c3deckmeetupnovember252019v2-191128092146-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Giovanni Galloro] How to use machine learning on Google Cloud Platform](https://cdn.slidesharecdn.com/ss_thumbnails/mlcapabilitiesongcp-190115085455-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Redis Released]- FalkorDB - Redis + Graph Agentic Memory’s Secret Sauce](https://cdn.slidesharecdn.com/ss_thumbnails/redisreleased-falkordbslidedeck-1125-251115194922-e1c0046b-thumbnail.jpg?width=640&height=640&fit=bounds)
Deep Reinforcement Learning
- 1.
- 2. Outline What, why andwhere it stands in ML? General framework Q-Learning Deep + ... Codeanddemo(hopefullyworking...) 2
- 3.
- 4.
- 5.
- 6. "RL tries tounderstand the optimal way to make decisions." David Silver - Research Scientist, Google DeepMind 6
- 7. Howdoesitwork? = ( ,, , , . . . , , , , )Ht s1 a1 r1 s2 st at rt sT 7
- 8. Whyitisdi erentfromotherMLsettings? No supervisor Delayedfeedback Time matters Data depends on the Agent policy 8
- 9.
- 10.
- 11.
- 12. Environment import Gym env =Gym.make('SpaceInvaders-v0') s = env.step() # take an action terminal = False while not terminal: next_state, reward, terminal, _ = env.step(action) 11
- 13. s_1 -> s_2-> s_3 12
- 14. Keyassumptions 1. The probabilityof the next state depends only on the current state 2. Each state contains all the relevant information 13
- 15. It'sMe,Mario! Mario wants tobreak bricks and free the princess! 14
- 16. Expectedfuturerewards Any goal canbe represented as a sum of intermediate rewards. [ ∣ ] = [ + γ + + … ∣ ]∑ ∞ t=0 γ t Rt St R0 R1 γ 2 R2 St 15
- 17. Tools 1. Policy: 2. Valuefunction: 3. Model: We have to pick at least 1 of the 3. π(a|s) Q(s, a) (P, R) 16
- 18. Policy A policy denes how the agent behaves. It takes as input a state and output an action. It can be stochastic, or deterministic. 17
- 19. Valuefunction A value functionestimates how much reward the agent can achieve. It takes as input a (state,action), and output values. One for each possible action. 18
- 20. Model A model isthe Agent representation of the environment. Takes as input a state and output (next_state,reward). 19
- 21. Designchoice Balance learning andplanning Explore new actions and exploit good ones Assign credits for correct actions 20
- 22.
- 23. Howtosolveit? The goal isto nd the optimalpolicy that maximize the futureexpected rewards. 22
- 24. Repeat 1. Prediction: Computethe value of the expected reward from until the terminal state. 2. Control: Act greedly with respect to the predicted values. st 23
- 25. Approximationofthevaluefunction Monte Carlo (usedin Alpha Go) Temporal Di erence (used in Atari) 24
- 26.
- 27.
- 28. Updaterule In rabbits, humansand machines we get the same algorithm: while True: Q[t] = Q[t-1] + alpha * (Q_target - Q[t-1]) 27
- 29. Q-Learning[Watkins,1989] The agent doesnot have a model of the environment. Perform actions following a standard policy. Predict using the target policy. Which makes it an "o -policy", model-free method. 28
- 30. Lossfunction Building on whatwe learn from the rabbit. The learning goal is to minimize the following loss function: Putting all together we get... Q_target = r + gamma * np.argmax( Q(s, A)) Loss = 1/n * np.sum( (Q_target - Q(s,a))^2) 29
- 31. DeepQ-Learning Let's add NeuralNetworks and we are good to go right? 30
- 32.
- 33. Notice... 1. Data arehighly correlated 2. The target values are not robust 3. Wild rewards make the value function freaks out 32
- 34. Wewish... A stable Q_target,a robust Q and predictable rewards. But how? 33
- 35. DeepMindideas 1. Di erentneural networks for Q and Q_target 2. Estimate Q_target using past experiences 3. Update Q_target every C steps 4. Clip rewards between -1 and 1 34
- 36. Network Input: an imageof shape [None, 42, 42, 4] 4 Conv2D 32 lters, 4x4 kernel 1 Hidden layer of size 256 1 Fully connected layer of size action_size 35
- 37. Hyperparams Learning rate: 0.001 Rewardclip: (-1, 1) Gradient clip: 40 Optimizer: AdamOptimizer 36
- 38.
- 39. Tools Challenges Demo OpenAI Tensor ow General AIChallenge Stanford 38
- 40. Resources: Papers RL - DavidSilver Introduction to RL Patacchiola Blog Human Level control Async Method for DRL 39
- 41.
- 42.