What is Reinforcement Learning and How Does It Work (Updated 2024)

Introduction

Reinforcement Learning, seems intriguing, right? Here in this article, we will see what it is and why is it so much talked about these days. This acts as a guide to learn the fundamentals of reinforcement learning for beginners. Reinforcement Learning is definitely one of the evident research areas at present which has a good boom to emerge in the coming future and its popularity is increasing day by day. Lets, get it started.

It is basically the concept where machines can teach themselves depending upon the results of their own actions. Without further delay, let’s start this tutorial.

This article was published as a part of the Data Science Blogathon.

What is Reinforcement Learning?

Reinforcement Learning is a part of machine learning. Here, agents are self-trained on reward and punishment mechanisms. It’s about taking the best possible action or path to gain maximum rewards and minimum punishment through observations in a specific situation. It acts as a signal to positive and negative behaviors. Essentially an agent (or several) is built that can perceive and interpret the environment in which is placed, furthermore, it can take actions and interact with it.

To know the meaning of reinforcement learning, let’s go through the formal definition.

Reinforcement learning, a type of machine learning, in which agents take actions in an environment aimed at maximizing their cumulative rewards – NVIDIA

Reinforcement learning (RL) is based on rewarding desired behaviors or punishing undesired ones. Instead of one input producing one output, the algorithm produces a variety of outputs and is trained to select the right one based on certain variables – Gartner

It is a type of machine learning technique where a computer agent learns to perform a task through repeated trial and error interactions with a dynamic environment. This learning approach enables the agent to make a series of decisions that maximize a reward metric for the task without human intervention and without being explicitly programmed to achieve the task – Mathworks

The above definitions are technically provided by experts in that field however for someone who is starting with reinforcement learning, but these definitions might feel a little bit difficult. As this is a reinforcement learning guide for beginners, let’s create our reinforcement learning definition in an easier way.

Simplified Definition of Reinforcement Learning

Through a series of Trial and Error methods, an agent keeps learning continuously in an interactive environment from its own actions and experiences. The only goal of it is to find a suitable action model which would increase the total cumulative reward of the agent. It learns via interaction and feedback.

Well, that’s the definition of reinforcement learning. Now how we come to this definition, how a machine learns and how it can solve complex problems in the world through reinforcement learning, is something we are going to see further.

How Does Reinforcement Learning Work?

By exploring the environment and trying different actions, the agent gradually learns the best course of action for different situations. These learned behaviors are like a set of guidelines, or a policy, that helps the agent choose its next action. The goal is to maximize its total reward over time. However, the agent faces a dilemma: should it keep exploring new possibilities to discover potentially even better rewards, or should it stick with actions that have already proven successful? This is known as the exploration-exploitation trade-off.

Here what do you see?

You can see a dog and a master. Let’s imagine you are training your dog to get the stick. Each time the dog gets a stick successfully, you offered him a feast (a bone let’s say). Eventually, the dog understands the pattern, that whenever the master throws a stick, it should get it as early as it can to gain a reward (a bone) from a master in a lesser time.

Terminologies used in Reinforcement Learning

• Agent – Agent or Reinforcement learning agent or Learning agent all are same. It is the sole decision-maker and learner
• Environment – a physical world where an agent learns and decides the actions to be performed
• Action Space – a list of action which an agent can perform
• Action -An agent’s single choice (move left, pick up object) in the environment.
• State – the current situation of the agent in the environment
• Reward – For each selected action by agent to solve reinforcement learning problem, the environment gives a reward. It’s usually a scalar value and nothing but feedback from the environment
• Reward Function: This is a predefined function within the RL framework that determines how rewards are assigned based on the state of the environment and the agent’s actions.
• Policy – the agent prepares strategy(decision-making) to map situations to actions.
• Value Function – The value of state shows up the reward achieved starting from the state until the policy is executed
• Model – Every RL agent doesn’t use a model of its environment. The agent’s view maps state-action pairs probability distributions over the states.

Characteristics of Reinforcement Learning

• No supervision, only a real value or reward signal
• Decision making is sequential
• Time plays a major role in reinforcement problems
• Feedback isn’t prompt but delayed
• The following data it receives is determined by the agent’s actions

How is Reinforcement Learning different from Supervised Learning?

Data and Feedback

• Supervised Learning: Relies on labeled data. Each data point has a pre-defined output or label (e.g., classifying emails as spam or not spam). The model learns the mapping between the input data and the desired output.
• Unsupervised Learning: Deals with unlabeled data. The goal is to identify patterns or structures within the data itself (e.g., grouping customers with similar purchase history). No pre-defined output is provided.
• Reinforcement Learning: Doesn’t use labeled data. The agent interacts with the environment and receives feedback in the form of rewards (positive, negative, or neutral). The agent learns through trial and error to maximize future rewards.

Learning Process

• Supervised Learning: The model is like a student directly taught by a teacher (training data) what the correct output should be for a given input.
• Unsupervised Learning: The model is like an explorer trying to find patterns and relationships within uncharted territory (data) with minimal guidance.
• Reinforcement Learning: The model resembles an athlete learning through trial and error in a competition (environment). It receives feedback (rewards) but needs to figure out the best strategy on its own.

Goal

• Supervised Learning: Aims to learn a function that maps inputs to desired outputs accurately.
• Unsupervised Learning: Focuses on uncovering hidden structures or patterns within the data.
• Reinforcement Learning: The objective is to learn a policy or strategy that maximizes long-term rewards within an environment.

In supervised learning, the model is trained with a training dataset that has a correct answer key. The decision is done on the initial input given as it has all the data that’s required to train the machine. The decisions are independent of each other so each decision is represented through a label.

Example: Object Recognition

Approaches to Implement Reinforcement Learning Algorithms

The world of reinforcement learning (RL) offers a diverse toolbox of algorithms. Some popular examples include Q-learning, policy gradient methods, and Monte Carlo methods, along with temporal difference learning. Deep RL takes things a step further by incorporating powerful deep neural networks into the RL framework. One such deep RL algorithm is Trust Region Policy Optimization (TRPO).

However, despite their variety, all these algorithms can be neatly categorized into two main groups:

Value-Based

• Focuses on learning a value function that estimates the expected future reward for an agent in a given state under a specific policy.
• The agent aims to maximize this value function to achieve long-term reward.
• Popular algorithms in this category include Q-Learning, SARSA, and Deep Q-Networks (DQN).

Policy-Based

• Directly learns the policy function, which maps states to actions.
• The goal is to find the optimal policy that leads to the highest expected future rewards.
• Examples of policy-based methods include REINFORCE, Proximal Policy Optimization (PPO), and Actor-Critic methods.

Model-Based

• Attempts to learn a model of the environment dynamics. This model predicts the next state and reward for a given state-action pair.
• The agent can then use this model to plan and simulate actions in a virtual environment before taking them in the real world.
• While conceptually appealing, this approach can be computationally expensive for complex environments and often requires additional assumptions about the environment’s behavior.

How to Choose the Right Approach:

The choice of approach depends on several factors, including:

• The complexity of the environment: For simpler environments, value-based methods might be sufficient. Complex environments might benefit from policy-based or model-based approaches (if feasible).
• Availability of computational resources: Model-based approaches can be computationally expensive.
• The desired level of interpretability: Value-based methods often offer more interpretability compared to policy-based methods.

Types of Reinforcement Learning

There are two types :

1. Positive Reinforcement

Positive reinforcement is defined as when an event, occurs due to specific behavior, increases the strength and frequency of the behavior. It has a positive impact on behavior.

Advantages

• Maximizes the performance of an action
• Sustain change for a longer period

Disadvantage

• Excess reinforcement can lead to an overload of states which would minimize the results.

2. Negative Reinforcement

Negative Reinforcement is represented as the strengthening of a behavior. In other ways, when a negative condition is barred or avoided, it tries to stop this action in the future.

Advantages

• Maximized behavior
• Provide a decent to minimum standard of performance

Disadvantage

• It just limits itself enough to meet up a minimum behavior

Widely Used Models for Reinforcement Learning

Reinforcement learning (RL) tackles problems where an agent interacts with an environment, learning through trial and error to maximize rewards. Two main categories of models are used:

1. Traditional RL Models: Suitable for smaller environments and rely on simpler function approximation.
2. Deep Reinforcement Learning Models: Leverage deep learning techniques (like neural networks) for complex, high-dimensional environments.

Traditional RL Models

Markov Decision Process (MDP’s)

Markov Decision Process (MDP’s) are mathematical frameworks for mapping solutions in RL. The set of parameters that include Set of finite states – S, Set of possible Actions in each state – A, Reward – R, Model – T, Policy – π. The outcome of deploying an action to a state doesn’t depend on previous actions or states but on current action and state.

Q Learning

It’s a value-based model free approach for supplying information to intimate which action an agent should perform. It revolves around the notion of updating Q values which shows the value of doing action A in state S. Value update rule is the main aspect of the Q-learning algorithm.

SARSA (State-Action-Reward-State-Action)

Similar to Q-Learning but focuses on learning the value of the specific action taken in the current state, considering the next state reached. This can be computationally more efficient than Q-Learning in some cases.

These models often use techniques like Monte Carlo methods to estimate the value of states or state-action pairs. Monte Carlo methods involve simulating multiple playthroughs of the environment to gather reward information and update the agent’s policy accordingly.

Deep Reinforcement Learning Models

• Deep Q-Learning (DQL): Combines Q-Learning with a deep neural network to approximate the Q-value function. This allows DQL to handle complex environments with many states and actions, where traditional function approximation methods might struggle. DQL has been a major breakthrough in deep rl.
• Policy Gradient Methods: These methods directly train the policy function, which maps states to actions. One approach is REINFORCE, which uses Monte Carlo methods to estimate the gradient of the expected reward with respect to the policy parameters. This gradient is then used to update the policy in a direction that increases the expected reward. More advanced methods like Proximal Policy Optimization (PPO) address limitations of REINFORCE to improve stability and performance.
• Actor-Critic Methods: Combine an actor (policy network) and a critic (value network) for joint policy learning and value estimation. The actor learns the policy, while the critic evaluates the value of states or state-action pairs. This combined approach can improve learning efficiency and stability.

Practical Applications of reinforcement learning

• Robotics for Industrial Automation
• Text summarization engines, dialogue agents (text, speech), gameplays
• Autonomous Self Driving Cars
• Machine Learning and Data Processing
• Training system which would issue custom instructions and materials with respect to the requirements of students
• AI Toolkits, Manufacturing, Automotive, Healthcare, and Bots
• Aircraft Control and Robot Motion Control
• Building artificial intelligence for computer games

Conclusion

Reinforcement learning guides us in determining actions that maximize long-term rewards. However, it may struggle in partially observable or non-stationary environments. Moreover, its effectiveness diminishes when ample supervised learning data is available. A key challenge lies in managing parameters to optimize learning speed.

Hope now you got the feel and certain level of the description on Reinforcement Learning. Thanks for your time.

Frequently Asked Questions

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

Prathima 26 May, 2024