Step by Step process of Feature Engineering for Machine Learning Algorithms in Data Science

E
ELLURU PAVAN KUMAR 25 Mar, 2021
7 min read

Introduction

Data Science is not a field where theoretical understanding helps you to start a carrier. It totally depends on the projects you do and the practice you have done that determines your probability of success. Feature engineering is a very important aspect of machine learning and data science and should never be ignored. The main goal of Feature engineering is to get the best results from the algorithms.

feature_engineering

Table of Contents

  1. Why should we use Feature Engineering in data science?
  2. Feature Selection
  3. Handling missing values
  4. Handling imbalanced data
  5. Handling outliers
  6. Binning
  7. Encoding
  8. Feature Scaling

1. Why should we use Feature Engineering in data science?

In Data Science, the performance of the model is depending on data preprocessing and data handling. Suppose if we build a model without Handling data, we got an accuracy of around 70%. By applying the Feature engineering on the same model there is a chance to increase the performance from 70% to more.

Simply, by using Feature Engineering we improve the performance of the model.

 

2. Feature selection

Feature selection is nothing but a selection of required independent features. Selecting the important independent features which have more relation with the dependent feature will help to build a good model. There are some methods for feature selection:

2.1 Correlation Matrix with Heatmap

Heatmap is a graphical representation of 2D (two-dimensional) data. Each data value represents in a matrix.

Firstly, plot the pair plot between all independent features and dependent features. It will give the relation between dependent and independent features. The relation between the independent feature and the dependent feature is less than 0.2 then choose that independent feature for building a model.

Feature Engineering correlation heatmap

2.2 Univariate Selection

In this, Statistical tests can be used to select the independent features which have the strongest relationship with the dependent feature. SelectKBest method can be used with a suite of different statistical tests to select a specific number of features.

Univariate Selection
feature engineering steps univariate selection

 

  • Which feature has the highest score will be more related to the dependent feature and choose those features for the model.

2.3 ExtraTreesClassifier method

In this method, the ExtraTreesClassifier method will help to give the importance of each independent feature with a dependent feature. Feature importance will give you a score for each feature of your data, the higher the score more important or relevant to the feature towards your output variable.

ExtraTreesClassifier method
Feature engineering steps ExtraTreesClassifier method output

 

3. Handling Missing Values

In some datasets, we got the NA values in features. It is nothing but missing data. By handling this type of data there are many ways:

  • In the missing value places, to replace the missing values with mean or median to numerical data and for categorical data with mode.
Handling Missing Values

 

  • Drop NA values entire rows.
Drop NA

 

  • Drop NA values entire features. (it helps if NA values are more than 50% in a feature)
Drop NA values

 

  • Replace NA values with 0.
Replace NA values

If you choose to drop options, there is a chance to lose important information from them. So better to choose to replace options.

 

4. Handling imbalanced data

Why need to handle imbalanced data? Because of to reduce overfitting and underfitting problem.

suppose a feature has a factor level2(0 and 1). it consists of 1’s is 5% and 0’s is 95%. It is called imbalanced data.

Example:-

Handling imbalanced data 101

By preventing this problem there are some methods:

4.1 Under-sampling majority class

Under-sampling the majority class will resample the majority class points in the data to make them equal to the minority class.

Under-sampling majority class feature engineering steps

4.2 Over Sampling Minority class by duplication

Oversampling minority class will resample the minority class points in the data to make them equal to the majority class.

Over Sampling Minority class by duplication feature engineering steps

 

4.3 Over Sampling minority class using Synthetic Minority Oversampling Technique (SMOTE)

In this method, synthetic samples are generated for the minority class and equal to the majority class.

Over Sampling minority class using Synthetic Minority Oversampling Technique (SMOTE) feature engineering

5. Handling outliers

firstly, calculate the skewness of the features and check whether they are positively skewed, negatively skewed, or normally skewed. Another method is to plot the boxplot to features and check if any values are out of bounds or not. if there, they are called outliers.

Handling outliers fature engineering steps

how to handle these outliers: –

first, calculate quantile values at 25% and 75%.

 

how to handle these outliers code feature engineering steps

 

  • next, calculate the Interquartile range

IQR = Q3 – Q1

IQR = Q3 – Q1
  • next, calculate the upper extreme and lower extreme values

lower extreme=Q1 – 1.5 * IQR

upper extreme=Q3– 1.5 * IQRe

upper extreme and lower extreme values feature engineering

 

  • lastly, check the values will lie above the upper extreme or below the lower extreme. if it presents then remove them or replace them with mean, median, or any quantile values.
  • Replace outliers with mean
Replace outliers with mean feature engineering

 

  • Replace outliers with quantile values
Replace outliers with quantile values

 

  • Drop outliers
Drop outliers

6. Binning

Binning is nothing but any data value within the range is made to fit into the bin. It is important in your data exploration activity. We typically use it to transform continuous variables into discrete ones.

Suppose if we have AGE feature in continuous and we need to divide the age into groups as a feature then it will be useful.

AGE feature

7. Encoding:

Why this will apply? because in datasets we may contain object datatypes. for building a model we need to have all features are in integer datatypes. so, Label Encoder and OneHotEncoder are used to convert object datatype to integer datatype.

  • Label Encoding

 

Encoding feature engineering

Before applying Label Encoding

Label Encoding

 

Label Encoding feature engineering

After applying label encoding then apply the column transformer method to convert labels to 0 and 1

label encoding
  • One Hot Encoding:

By applying get_dummies we convert directly categorical to numerical

One Hot Encoding

 

8. Feature scaling

Why this scaling is applying? because to reduce the variance effect and to overcome the fitting problem. there are two types of scaling methods:

8.1 Standardization

When this method is used?. when all features are having high values, not 0 and 1.

It is a technique to standardize the independent features that present in a fixed range to bring all values to the same magnitudes.

Standardization feature engineering steps

In standardization, the mean of the independent features is 0 and the standard deviation is 1.

Method 1:

 

code
Method 1 output

Method2:

code Method2

 

After encoding feature labels are in 0 and 1. This may affect standardization. To overcome this, we use Normalization.

8.2 Normalisation

Normalization also makes the training process less sensitive by the scale of the features. This results in getting better coefficients after training.

formula

Method 1: -MinMaxScaler

It is a method to rescales the feature to a hard and fast range of [0,1] by subtracting the minimum value of the feature then dividing by the range.

MinMaxScaler feaure engineering

Method 2: – Mean Normalization

It is a method to rescales the feature to a hard and fast range of [-1,1] with mean=0.

Mean Normalization
Mean Normalization

 

Mean Normalization feature engineering output

 

End Notes:-

In this article, I covered step by step process of feature engineering. This is more helpful to increase prediction accuracy.

Keep in mind that there are no particular methods to increase your prediction accuracy. It all depends on your data and applies multiple methods.

As a next step, I encourage you to try out different datasets and analyze them. And don’t forget to share your insights in the comments section below!

About the Author:

I am Pavan Kumar Reddy Elluru. I completed my graduation at G.Pullareddy Engineering College in the year 2020. I am a certified data scientist in the year 2021 and passionate about Machine Learning and Deep Learning Projects.

Please ping me in case of any queries or just to say hi!

Email id:- [email protected]

Linkedin id: – www.linkedin.com/in/elluru-pavan-kumar-reddy-a1b183197

Github id: – pawankumarreddy1999 (Pavan Kumar Reddy Elluru) (github.com)

BinningblogathonCorrelationfeature engineeringFeature scalingfeature selectionheatmapimbalanced datasetMissing Valuesnormalising dataoutliers
E
ELLURU PAVAN KUMAR 25 Mar, 2021

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naïve Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Proven Ways for improving the “Accuracyâ€_x009d_ of a Machine Learning Model
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Sushant
Sushant 16 Dec, 2021

In correlation matrix, if the relation between independent and dependent variable is greater than 0.02 then it gives a good measure in prediction. Why are we doing the exact opposite thing. Please let me know if I might be wrong

123

R.vijayarahini
R.vijayarahini 26 Aug, 2022

Very easy for learning and useful for studys

123

Anand Shaw
Anand Shaw 11 Sep, 2023

It's is very good Example of Feature Engineering But i have a question this all steps are Full ??

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Rahul Shah img

Rahul Shah

27
Sion Chakrabarti img

Sion Chakrabarti

16
CHIRAG GOYAL img

CHIRAG GOYAL

87
Barney Darlington img

Barney Darlington

5
Suvojit Hore img

Suvojit Hore

9
Arnab Mondal img

Arnab Mondal

15
Prateek Majumder img

Prateek Majumder

68