Python Treatment for Outliers in Data Science

Shanthababu Pandian
Shanthababu Pandian 03 Aug, 2022
6 min read
This article was published as a part of the Data Science Blogathon.

Before we get started the discussion on Outliers, we should understand exactly what Feature Improvements mean under Feature Engineering.

What is Feature Engineering?

  • When we have a LOT OF FEATURES in the given dataset, feature engineering can become quite a challenging and interesting module.
  • The number of features could significantly impact the model considerably, So that feature engineering is an important task in the Data Science life cycle.
Family of Feature Engineering

Family of Feature Engineering

Feature Improvements

In the Feature Engineering family, we are having many key factors are there, let’s discuss Outlier here. This is one of the interesting topics and easy to understand in Layman’s terms.

 

Outlier

  • An outlier is an observation of a data point that lies an abnormal distance from other values in a given population. (odd man out)
    • Like in the following data point (Age)
      • 18,22,45,67,89,125,30
  • An outlier is an object(s) that deviates significantly from the rest of the object collection.
    • List of Cities
      • New York, Las Angles, London, France, Delhi, Chennai
  • It is an abnormal observation during the Data Analysis stage, that data point lies far away from other values.
    • List of Animals
      • cat, fox, rabbit, fish
  • An outlier is an observation that diverges from well-structured data.
  • The root cause for the Outlier can be an error in measurement or data collection error.
  • Quick ways to handling Outliers.
    • Outliers can either be a mistake or just variance. (As mentioned, examples)
    • If we found this is due to a mistake, then we can ignore them.
    • If we found this is due to variance, in the data, we can work on this.
outlier apple

In the picture of the Apples, we can find the out man out?? Is it? Hope can Yes!

But the huge list of a given feature/column from the .csv file could be a really challenging one for naked eyes.

First and foremost, the best way to find the Outliers are in the feature is the visualization method.

Visualization of outlier

Visualization of outlier

 

What are the Possibilities for an Outlier? 

Of course! It would be below quick reasons.

  • Incorrect data entry or error during data processing
  • Missing values in a dataset.
  • Data did not come from the intended sample.
  • Errors occur during experiments.
  • Not an errored, it would be unusual from the original.
  • Extreme distribution than normal.

That’s fine, but you might have questions about Outlier if you’re a real lover of Data Analytics, Data mining, and Data Science point of view.

Let’s have a quick discussion on those.

 Possibilities for an Outlier

 

Understand more about Outlier

  • Outliers tell us that the observations of the given data set, how the data point(s) differ significantly from the overall perspective. Simply saying odd one/many. this would be an error during data collection. 
  • Generally, Outliers affect statistical results while doing the EDA process, we could say a quick example is the MEAN and MODE of a given set of data set, which will be misleading that the data values would be higher than they really are.
  • the CORRELATION COEFFICIENT is highly sensitive to outliers. Since it measures the strength of a linear relationship between
    two variables. the relationship dependent of the data. correlation is a non-resistant measure and r (correlation coefficient) is strongly affected by
    outliers.

    • Positive Relationship 
      • When the correlation coefficient is closer to value 1
    •  Negative Relationship
      • When the correlation coefficient is closer to value -1
    • Independent
      • When X and Y are independent, then the correlation coefficient is close to zero (0)
  • We could understand the data collection process from the Outliers and its observations. An analysis of how it occurs and how to minimize and set the process in future data collection guidelines.
  • Even though the Outliers increase the inconsistent results in your dataset during analysis and the power of statistical impacts significant, there would challenge and roadblocks to remove them in few situations.
  • DO or DO NOT (Drop Outlier)
    • Before dropping the Outliers, we must analyze the dataset with and without outliers and understand better the impact of the results.
    • If you observed that it is obvious due to incorrectly entered or measured, certainly you can drop the outlier. No issues on that case.
    • If you find that your assumptions are getting affected, you may drop the outlier straight away, provided that no changes in the results.
    • If the outlier affects your assumptions and results. No questions simply drop the outlier and proceed with your further steps.

 

Finding Outliers

So far we have discussed what is Outliers, how it affects the given dataset, and Either can we drop them or NOT. Let see now how to find from the given dataset. Are you ready!

We will look at simple methods first, Univariate and Multivariate analysis.

  • Univariate method: I believe you’re familiar with Univariate analysis, playing around one variable/feature from the given data set. Here to look at the Outlier we’re going to apply the BOX plot to understand the nature of the Outlier and where it is exactly.
Finding Outlier

Let see some sample code. Just I am taking titanic.csv as a sample for my analysis, here I am considering age for my analysis.

plt.figure(figsize=(5,5))
sns.boxplot(y='age',data=df_titanic)

You can see the outliers on the top portion of the box plot visually in the form of dots.

  • Multivariate method: Just I am taking titanic.csv as a sample for my analysis, here I am considering age and passenger class for my analysis.
plt.figure(figsize=(8,5))
sns.boxplot(x='pclass',y='age',data=df_titanic)
Multivariate method

We can very well use Histogram and Scatter Plot visualization technique to identify the outliers.

On top of this, we have with
mathematically to find the Outliers as follows Z-Score and Inter Quartile Range (IQR) Score methods

Z-Score method: In which the distribution of data in the form mean is 0 and the standard deviation (SD) is 1 as Normal Distribution format.

Let’s consider below the age group of kids, which was collected during data science life cycle stage one, and proceed for analysis, before going into further analysis, Data scientist wants to remove outliers. Look at code and output, we could understand the essence of finding outliers using the Z-score method.

import numpy as np  
kids_age = [1, 2, 4, 8, 3, 8, 11, 15, 12, 6, 6, 3, 6, 7, 12,9,5,5,7,10,10,11,13,14,14] 
mean = np.mean(voting_age) 
std = np.std(voting_age) 
print('Mean of the kid''s age in the given series :', mean) 
print('STD Deviation of kid''s age in the given series :', std)
threshold = 3
outlier = [] 
for i in voting_age: 
    z = (i-mean)/std 
    if z > threshold: 
        outlier.append(i) 
print('Outlier in the dataset is (Teen agers):', outlier)

Output

Mean of the kid’s age in the given series: 2.6666666666666665
STD Deviation of kids age in the given series: 3.3598941782277745
The outlier in the dataset is (Teenagers): [15]

(IQR) Score method: In which data has been divided into quartiles (Q1, Q2, and Q3). Please refer to the picture Outliers Scaling above.  Ranges as below.

  • 25th percentile of the data – Q1
  • 50th percentile of the data – Q2
  • 75th percentile of the data – Q3

Let’s have the junior boxing weight category series from the given data set and will figure out the outliers.

import numpy as np  
import seaborn as sns
# jr_boxing_weight_categories
jr_boxing_weight_categories = [25,30,35,40,45,50,45,35,50,60,120,150] 
Q1 = np.percentile(jr_boxing_weight_categories, 25, interpolation = 'midpoint')
Q2 = np.percentile(jr_boxing_weight_categories, 50, interpolation = 'midpoint')  
Q3 = np.percentile(jr_boxing_weight_categories, 75, interpolation = 'midpoint')
IQR = Q3 - Q1
print('Interquartile range is', IQR)
low_lim = Q1 - 1.5 * IQR
up_lim = Q3 + 1.5 * IQR
print('low_limit is', low_lim)
print('up_limit is', up_lim)
outlier =[]
for x in jr_boxing_weight_categories:
    if ((x> up_lim) or (x<low_lim)):
         outlier.append(x)
print(' outlier in the dataset is', outlier)

Output

Interquartile range is 20.0
low_limit is 5.0
up_limit is 85.0
the outlier in the dataset is [120, 150]

sns.boxplot(jr_boxing_weight_categories)
Output

Loot at the boxplot we could understand where the outliers are sitting in the plot.

So far, we have discussed what is Outliers, how it looks like, Outliers are good or bad for data set, how to visualize using matplotlib /seaborn and stats methods.

Now, will conclude correcting or removing the outliers and taking appropriate decision. we can use the same Z- score and (IQR) Score with the condition we can correct or remove the outliers on-demand basis. because as mentioned earlier Outliers are not errors, it would be unusual from the original.

Hope this article helps you to understand the Outliers in the zoomed view in all aspects. let’s come up with another topic shortly. until then bye for now! Thanks for reading! Cheers!!

The media shown in this article are not owned by Analytics Vidhya and is used at the Author’s discretion.

blogathonoutlier
Shanthababu Pandian
Shanthababu Pandian 03 Aug, 2022

Shanthababu has 20+ years of experience in Information Technology (IT). Expertise in ML & Data Architect and Program Delivery. Analytics and AIML Solutions Lead, Data Architecting, Modelling, Data Quality, Data Governance, Data Privacy and Data Integration, Data Platform, Data analysis and Data Science program.

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naïve Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Proven Ways for improving the “Accuracyâ€_x009d_ of a Machine Learning Model
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Danimir
Danimir 10 Mar, 2021

Great explanation! Thanks

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Rahul Shah img

Rahul Shah

27
Sion Chakrabarti img

Sion Chakrabarti

16
CHIRAG GOYAL img

CHIRAG GOYAL

87
Barney Darlington img

Barney Darlington

5
Suvojit Hore img

Suvojit Hore

9
Arnab Mondal img

Arnab Mondal

15
Prateek Majumder img

Prateek Majumder

68