Does The Popularity of AutoML Means the End of Data Science Jobs?

CHIRAG
CHIRAG 15 Apr, 2021
5 min read
This article was published as a part of the Data Science Blogathon.

Introduction

With the strike of the COVID-19 pandemic and urge people showed to understand the pattern of COVID-19 transmission, which was being formulated by the huge data being available in the healthcare community, people were more and more inclined to learn Machine learning and Data analysis than ever before and this urge is going to retain even post-pandemic.

While going through Data preprocessing for Data science projects you would have felt the thirst for an automated method which would have saved your time from the conceivable exhaustive process of preprocessing followed by trying different models which would properly fit your data.

The answer to this need is YES, there exist tools that would not only automate the preprocessing stage but would also choose the relevant model for predictive analysis.

“The need is for Automated Machine learning (AutoML)”

This article will give you a brief idea about the pros and cons of AutoML in present times and try to give you a view on “Does the Popularity of AutoML mean End of Data Science jobs?” 

 

AutoML mean End of Data Science jobs

What is AutoML?

Automated machine learning (AutoML) is the process of applying Machine Learning (ML) models to real-world problems by giving just the initial commands and the rest of the pipelines are taken care of by the program itself. More specifically, it automates the selection, composition, and parameterization of machine learning models. Internally, it creates different pipelines by choosing the different hyperparameters and choose that pipeline that gives maximum accuracy for our dataset.

AutoML has become the new trend in the field of machine learning. Its aim to automate the whole cycle of Machine learning and Deep learning projects. Machine learning is definitely the most thriving technology. For a newbie, it takes a lot of time to learn the coding stuff even after being thorough with the theoretical part. AutoML becomes the need of the hour for the researchers and enthusiasts from different domain who want to apply machine learning techniques to their work but does not have much time to spend on learning the coding part in depth.

“Automated machine learning (AutoML) aims to lower the need for skilled Data Scientists to build Machine learning and Deep learning models”. This is what is said by certain sections in the field of machine learning, but despite the ideology of having a negative impact on jobs, there are some people who are optimistic about the development in AutoML and believe that it will revolutionize machine learning sector.

Pros of AutoML

1. Accessibility: Professionals and researchers from other domains of specialization without much experience in Machine learning can use AutoML for their project without worrying much about the exhaustive and redundant processes of data preparation and other processing stages including model selection.

2. Efficiency: For regular Data scientists and Analysts, AutoML can save much of their time in redundant steps which could have been utilized in making the models more optimized by tuning the hyperparameters. Hence, it increases the efficiency of the work.

3. Less Errors: Codes are often prone to errors. AutoML helps to reduce human errors in the common steps. You wouldn’t have to worry about some errors in the former stages which would eventually ruin your future predictions. AutoML is analogous to using a calculator for multiplication, in which rather than doing operations and steps manually we can straight away use a calculator for multiplication to arrive at the same result.

4. Cost savings: This will be extremely useful for small companies or startups that can’t afford to hire an ML professional to build their recommendation or sales forecast systems. But for large projects, you’ll still need people to do the modeling.

5. Fulfill Industry Demands: AutoML will make the process of learning ML, too many other professionals from other domains easier, which would eventually attract people to switch to Machine learning and analyst jobs which would fulfill the ever-growing demand for human resource in this sector.

 

Shortcoming / Challenges to AutoML

1. Computational power:  It will require more computational power to choose the correct model for the data, which will require the data to go through every model and fit it to find the accuracy, but if we try to do it manually we can trade-off this problem by eliminating many models which will certainly not work well with our data.

2. Difficulty with varying Datasets: AutoML is usually generalized to different forms of the dataset at present. But every dataset can vary in the relevance of features, its structure, and datatypes present in features, AutoML can do a significantly satisfactory job with most of the datasets but it can not meet the accuracy and persistence which could be meet manually.

3. Black Box:  Although AutoML improves efficiency in producing results it can be difficult to track the flow of the algorithm that has delivered the respective output. Moreover, this also makes it difficult to choose the right model for a given problem, because it can be difficult to predict a result of the process selecting it, is a black box.

What AutoML Platforms are Available?

1. Google Cloud AutoML: Google Cloud AutoML gained popularity due to its user-friendly interface and high performance. Build your own custom machine learning model in minutes.

2. Microsoft Azure AutoML:  Azure AutoML offers a transparent model selection process to its users who are not that familiar with coding. It is a cloud-based service for creating and managing machine learning solutions. Azure as a platform can be learned without knowing any programming at all.

3. H2O.ai: It offers both an open-source package and a commercial AutoML service called Driverless AI. Since its inception, this platform has been widely adopted in industries, including financial services and retail. It enables organizations to rapidly build world-class AI models and applications.

4. TPOT: TPOT (Tree-based Pipeline Optimization Tool) is a Python package that is free to use. Albeit free, the package has achieved outstanding performance in various datasets: around 97% accuracy for the Iris dataset, 98% for MNIST digit recognition, and around 10 Mean squared error(MSE) for Boston Housing Prices prediction.

 

Does AutoML eliminate data science jobs?

The answer to the given question is “NO”. Here are the reasons that support my view.

 

 

Does AutoML eliminate data science jobs?

1.  While AutoMLs is quite satisfactory at choosing models most of the time, but they are still not capable of doing most of the work of a Data Scientist. We still need Data scientists/Analysts to apply their domain knowledge to generate more useful features(Feature Engineering)and information that impact the target outcome.

2.  AutoML will not replace most of the data science positions, instead, it can help professionals to fast the phase of their projects.

3. Machines are not intelligent enough and the algorithms often don’t generalize and understand the context of a problem.

4.  AutoML can help us find a suitable model for a given problem but it can’t come up with a new approach which is often required for emerging real-life problems altogether.

End Notes

This article introduces you to AutoML and makes great efforts to remove myths behind this emerging method. This article is based on my research, mythologies, and thoughts.

I hope all the things explained in this article are clear to you.

Please don’t forget to share your thoughts on this with me.

If you have any queries, please ask them in the comments below and I will do my best to resolve them.

About the Author

Chirag Goyal

Currently, I pursuing my Bachelor of Technology (B.Tech) in Computer Science and Engineering from the Indian Institute of Technology Jodhpur(IITJ). I am very enthusiastic about Machine learning, Deep Learning, and Artificial Intelligence. If you want to connect with me here is the link to my Linkedin profile.

http://linkedin.com/in/chirag-goyal-3419671a0

The media shown in this article are not owned by Analytics Vidhya and is used at the Author’s discretion. 

AutoMLblogathondata science jobs
CHIRAG
CHIRAG 15 Apr, 2021

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naïve Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Proven Ways for improving the “Accuracyâ€_x009d_ of a Machine Learning Model
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Stephen Cobb
Stephen Cobb 14 Apr, 2021

"The End of Data Science", don't you think you being a little dramatic with that statement. That can't be and you engineers are always right? You will have to forgive my poke at you. I have worked with engineers for the past 30 years, ego us not your amego

123

Stephen Cobb
Stephen Cobb 14 Apr, 2021

I'm sure you were offer ded, but so was I by the words you chose to describe me and all non-engineers. The Masses Masses. Grow up and smell the coffee

123

DHIRAJLATA
DHIRAJLATA 21 Apr, 2021

Hi Chirag , Great work . I am currently a student of Data science but i have learnt a lot about Auto ML with this article . Precisely explained the advantages and disadvantages it can bring , but moreover I see a huge shift to all other sectors also . Work will be more efficient in lesser time still that human touch will be required indeed . Technology is always to ease our work . I APPRECIATE your time and effort for sharing your point of view . THANKS

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Rahul Shah img

Rahul Shah

27
Sion Chakrabarti img

Sion Chakrabarti

16
CHIRAG GOYAL img

CHIRAG GOYAL

87
Barney Darlington img

Barney Darlington

5
Suvojit Hore img

Suvojit Hore

9
Arnab Mondal img

Arnab Mondal

15
Prateek Majumder img

Prateek Majumder

68