Getting ready for AI based gaming agents – Overview of Open Source Reinforcement Learning Platforms

jalFaizy
jalFaizy 05 Jul, 2020
7 min read

Introduction

We are living in exciting times. We are all set to create an army of smart machines and robots. Creating these machines has been a dream and one of the biggest challenges humans have faced. What adds to this excitement is that no one knows how these smart machines and robots will impact us in return. Will they end up taking people out of their jobs? Or Will they create new avenues and opportunities, which we humans can’t think of as of now! One thing is for sure though – there is a lot of automation about to happen here!

Researchers have created a roadmap for machine intelligence research. Following suite, some major platforms have been built to give way to this research. In this article, I explain Reinforcement learning in simple terms and compare major platforms for testing reinforcement learning algorithms. Kindly note, I have included only those platforms which have projects dedicated for the environment and not those which have integrated support for reinforcement learning algorithms.

These platforms should enable a generation of research and new findings / developments in Artificial Intelligence and Machine Learning.

Table of Contents

  1. What is Reinforcement Learning?
  2. Examples of Reinforcement Learning
  3. What is a Reinforcement Learning Platform?
  4. Examples of Reinforcement Learning Platform
  5. Cheatsheet of Major Platforms
  6. A few other notable platforms

 

1. What is Reinforcement learning?

Let us start with a simple analogy. If you have a pet at home, you may have used this technique with your pet.

A clicker (or whistle) is a technique to let your pet know some treat is just about to get served! This is essentially “reinforcing” your pet to practice good behavior. You click the “clicker” and follow up with a treat. And with time, your pet gets accustomed to this sound and responds every time he/she hears the click sound. With this technique, you can train your pet to do “good” deeds when required.

Now let’s make these replacements in the example:

  • The pet becomes the artificial agent
  • The treat becomes the reward function
  • The good behavior is the resultant action

The above example explains what reinforcement learning looks like. This is actually a classic example of reinforcement learning.

To apply this on an artificial agent, you have a kind of a feedback loop to reinforce your agent. It rewards when the actions performed is right and punishes in-case it was wrong. Basically what you have in your kitty is:

  • an internal state, which is maintained by the agent to learn about the environment
  • a reward function, which is used to train your agent how to behave
  • an environment, which is a scenario the agent has to face
  • an action, which is done by the agent in the environment
  • and last but not the least, an agent which does all the deeds!

Source: UTCS RL Reading Group

2. Examples of Reinforcement Learning

Now, I am sure you must be thinking how the experiment conducted on animals can be relevant to people practicing machine learning. This is what I thought when I came across reinforcement learning first.

A lot of beginners tend to think that there are only 2 types of problems in machine learning – Supervised machine learning and Unsupervised machine learning. I don’t know where this notion comes from, but the world of machine learning is much more than the 2 types of problems mentioned above. Reinforcement learning is one such class of problems.

Let’s look at some real-life applications of reinforcement learning. Generally, we know the start state and the end state of an agent, but there could be multiple paths to reach the end state – reinforcement learning finds an application in these scenarios. This essentially means that driverless cars, self navigating vaccum cleaners, scheduling of elevators are all applications of Reinforcement learning.

Here is a video of a game bot trained to play flappy bird.

 

3. What is a Reinforcement Learning Platform?

Before we look into what a platform is, lets try to understand a reinforcement learning environment.

A reinforcement learning environment is what an agent can observe and act upon. The horizon of an agent is much bigger, but it is the task of the agent to perform actions on the environment which can help it maximize its reward. As per “A brief introduction to reinforcement learning” by Murphy (1998),

The environment is a modeled as a stochastic finite state machine with inputs (actions sent from the agent) and outputs (observations and rewards sent to the agent).

Let’s take an example,

This is a typical game of mario. Remember how you played this game. Now consider that you are the “agent” who is playing the game.

Now you have “access” to a land of opportunities, but you don’t know what will happen when you do something, say smash a brick. You can see a limited amount of “environment”, and until you traverse around the world you can’t see everything. So you move around the world, trying to perceive what entails ahead of you, and at the same time try to increase your chances to attain your goal.

This whole “story” is not created by itself. You have to “render” it first. And that is the main task of the platform, viz to create everything required for a complete experience – the environment, the agent and the rewards.

 

4. Major Reinforcement Learning Platforms

i) Deepmind Lab

DeepMind Lab is a fully 3D game-like platform tailored for agent-based AI research

A recent release by Google Deepmind, Deepmind lab is an integrated agent-environment platform for general artificial intelligence research with a focus on first person perspective games. It was built to accomodate the research done at DeepMind. Deepmind lab is based on an open-source engine ioquake3 , which was modified to be a flexible interface for integration with artificial systems.

Things I liked

  • It has richer and realistic visuals.
  • Closer integration with the gaming environment

Things I did not like

  • It still lacks variety in terms of a gaming environment, which would get built over time by open source contributions.
  • Also at the moment, it supports only Linux, but has been tested on different OS. Bazel (which is a dependency for deepmind lab) is experimental for windows. So windows support for Deepmind lab is still not guaranteed.

Resources to explore further:

 

ii) OpenAI Gym

(OpenAI Gym is) A toolkit for developing and comparing reinforcement learning algorithms

OpenAI Gym is a platform for creating, evaluating and benchmarking artificial agents in a game environment. The best thing I like about gym is that along with the toolkit, there is a community support built around it, viz an evaluation platform, code sharing platform and a discussion platform. Gym platform consists multiple categories of environment along with sample solutions provided by the community

Things I liked

  • Variety of game environments with considerable open-source support.

Things I did not like

  • Like Deepmind lab, Gym also has a limit on the number of environments it supports (This is essentially taken care by OpenAI Universe)

Resources to explore further:

 

iii) OpenAI Universe

Universe is a software platform for measuring and training an AI’s general intelligence across the world’s supply of games, websites and other applications

This is essentially an extension to OpenAI gym, with support for literally “anything” you can do on a computer. Universe is built to emulate how a human interacts with a computer. It uses Virtual Network Computing to access a computer remotely, packages any program and converts it into a gym environment.

Things I liked

  • Unlimited access to any game environment.
  • Universe is not only constrained to gaming environment, it can be used to replace things like manual testing and working on Amazon Mechanical Turk

Things I did not like

  • Initial release lacks many things which were promised, the significant one is integration with Windows.

Resources to explore further:

iv) Project Malmo

The Malmo platform is a sophisticated AI experimentation platform built on top of Minecraft, and designed to support fundamental research in artificial intelligence.

Project Malmo is a research initiative by Microsoft research, with an aim to build AI agents to do complex tasks. Minecraft is a perfect scenario for building AI agents, and that is why they chose it.

Things I liked

  • Integration with a sophisticated environment
  • Flexibility for customizing game environment

Things I did not like

  • The support is only for minecraft, and no other game environment unlike OpenAI universe.

Resources to explore further: 

 

v) VizDoom

Doom-based AI Research Platform for Reinforcement Learning from Raw Visual Information

I personally found this the most interesting platform to build AI agents, as you can have a multi-agent support with a competitive environment to test the agent. The platform runs on Doom, a first person shooting game, with a variety of levels and modes.

Things I liked

  • Variety of game modes with competitive environment

Things I did not like

  • Similar to above platform, VizDoom has only support for one environment.

Resources to explore further: 

5. Cheatsheet of Major Platforms

A few other notable platforms

  • RL-Glue
    • About: A standard interface for variety of languages to connect agents, environments and experiment programs together.
    • Resource: Main wiki Page
  • CommAI
    • About: A platform for training and testing AI systems based on communication tasks
    • Resource: Github repo
  • Burlap
    • About: BURLAP is a java code library for the use and development of single or multi-agent planning and learning algorithms and domains to accompany them.
    • Resource: Github repo
  • rlenvs
    • About: A platform similar to OpenAI Gym but for lua
    • Resource: Github repo

 

Acknowledgements

Thanks to AV community and reddit community for the helpful discussions. Special thanks to johny_cauchy, kendingpku and Kaixhin for their feedback.

 

End Notes

In this article, we briefly looked at what reinforcement learning is. I have listed all the major platforms for RL research. Most of these rely on gaming environments to simulate real life conditions. If you know other platforms for reinforcement learning, do let me know in the comments below!

Have your worked on any of these platforms? Share your experience by dropping in your comments. If you have any doubts / suggestions / feedback I would love you hear it. Feel free to post your comments.

You can test your skills and knowledge. Check out Live Competitions and compete with best Data Scientists from all over the world.

DeepMindDeepmind Labintelligent agentmachine learningOpenAIOpenAI GymOpenAI UniverseProject MalmoReinforcement Learningreinforcement learning platformsVizia
jalFaizy
jalFaizy 05 Jul, 2020

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naïve Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Proven Ways for improving the “Accuracyâ€_x009d_ of a Machine Learning Model
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

David Jung
David Jung 17 Oct, 2017

Nice overview.

123

Faizan Shaikh
Faizan Shaikh 16 Nov, 2017

Thanks!

123
abdul
abdul 19 Dec, 2017

wow clearly explained .. expecting more post like this

123

Erickson Liang
Erickson Liang 21 Dec, 2017

Hi, I'm a master student in Taiwan that study in Industrial Engineering, I want to apply this reinforcement learning for my Thesis. that is about inventory problem, that the reinforcement learning will be the one who makes the decision for example to order the items. Do you where I can study something like that? another question is do I need to make the model of reinforcement learning all by myself or actually in python there are already in the library that I can use? or if there any simulation way to do this reinforcement learning. Thank you very much.

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Rahul Shah img

Rahul Shah

27
Sion Chakrabarti img

Sion Chakrabarti

16
CHIRAG GOYAL img

CHIRAG GOYAL

87
Barney Darlington img

Barney Darlington

5
Suvojit Hore img

Suvojit Hore

9
Arnab Mondal img

Arnab Mondal

15
Prateek Majumder img

Prateek Majumder

68