What’s the first thing you do when attempting to cross the road? We typically look left and right, take stock of the vehicles on the road, and decide. In milliseconds, Our brain can analyze what kind of vehicle (car, bus, truck, auto, etc.) is approaching us. Can machines do that?
The answer was an emphatic ‘no’ until a few years back. However, the rise and advancements in computer vision have changed the game. We can build computer vision models that can detect objects, determine their shape, predict the direction they will go in, and many other things. You might have guessed it—that’s the powerful technology behind self-driving cars!
There are multiple ways of dealing with computer vision challenges. The most popular approach I have encountered is based on identifying the objects present in an image, aka object detection. But what if we want to dive deeper? What if just detecting objects isn’t enough—we want to analyze our image at a much more granular level?
As data scientists, we are always curious to dig deeper into the data. Asking questions like these is why I love working in this field!
In this article, I will introduce you to image segmentation. It is a powerful computer vision algorithm that builds upon the idea of object detection and takes us to a whole new level of working with image data. This technique opens up so many possibilities – it has blown my mind. Also, in this article, you will learn about image segmentation in image processing, its benefits, and how it works. So, you will completely understand image segmentation and image segmentation in image processing. Also, in this article you will get to know about the types of image segmentation, What is image segmentation algorithms adn also image segmentation techniques.
Image segmentation is a computer vision method that divides a digital image into distinct pixel groups, known as image segments, to aid in object detection and similar tasks. By breaking down an image’s complex visual information into uniquely shaped segments, this technique facilitates quicker and more sophisticated image processing
Here’s a breakdown of what image segmentation is and what it does:
Let’s understand the image segmentation algorithm using a simple example. Consider the below image:
There’s only one object here – a dog. We can build a straightforward cat-dog classifier model and predict that there’s a dog in the given image. But what if we have a cat and a dog in a single image?
We can train a multi-label classifier, for instance. However, there’s another caveat—we won’t know the location of either animal or object in the image.
That’s where image localization comes into the picture (no pun intended!). It helps us identify a single object’s location in the given image. We rely on object detection (OD) if we have multiple objects present. We can predict the location and class for each object using OD.
Before detecting the objects and even before classifying the image, we need to understand what it consists of. Enter Image Segmentation.
We can divide or partition the image into various parts called segments. It’s not a great idea to process the entire image at the same time, as there will be regions in the image that do not contain any information. By dividing the image into segments, we can use the important segments to process the image. That, in a nutshell, is how image segmentation works.
An image is a collection or set of different pixels. We group the pixels that have similar attributes using image segmentation. Take a moment to go through the below visual (it’ll give you a practical idea of segmentation in image processing):
Object detection builds a bounding box corresponding to each class in the image. But it tells us nothing about the object’s shape—we only get the set of bounding box coordinates. We want more information—this is too vague for our purposes.
The image segmentation algorithm creates a pixel-wise mask for each object in the image. This technique gives us a far more granular understanding of the object(s) in the image.
Why do we need to go this deep? Can’t all image processing tasks be solved using simple bounding box coordinates? Let’s take a real-world example to answer this pertinent question.
Cancer has long been a deadly illness. Even in today’s age of technological advancements, cancer can be fatal if we don’t identify it at an early stage. Detecting cancerous cells as quickly as possible can save millions of lives.
The shape of the cancerous cells plays a vital role in determining the severity of the cancer. You might have put the pieces together, but object detection will not be very useful here. We will only generate bounding boxes, which will not help us identify the shape of the cells.
Image Segmentation techniques make a MASSIVE impact here. They help us approach this problem more granularly and get more meaningful results. A win-win for everyone in the healthcare industry.
Here, we can see the shapes of all the cancerous cells. There are many other applications where the Image segmentation algorithm is transforming industries:
There are even more applications where Image Segmentation algorithms are very useful. Feel free to share them with me in the comments section below this article – let’s see if we can build something together. 🙂
We can broadly divide image segmentation techniques into two types. Consider the below images:
Can you identify the difference between these two? Both images use image segmentation techniques to identify and locate the people present.
Let me quickly summarize what we’ve learned. If there are 5 people in an image, semantic segmentation will focus on classifying all the people as a single instance. Instance segmentation, however, will identify each of these people individually.
So far, we have delved into the theoretical concepts of image processing and segmentation. Let’s mix things up a bit – we’ll combine learning concepts with implementing them in Python. I believe that’s the best way to learn and remember any topic.
One simple way to segment different objects could be to use their pixel values. An important point to note – the pixel values will be different for the objects and the image’s background if there’s a sharp contrast between them.
In this case, we can set a threshold value. The pixel values falling below or above that threshold can be classified accordingly (as objects or backgrounds). This technique is known as Threshold Segmentation.
If we want to divide the image into two regions (object and background), we define a single threshold value. This is known as the global threshold.
If we have multiple objects along with the background, we must define multiple thresholds. These thresholds are collectively known as the local threshold.
Let’s implement what we’ve learned in this section. Download this image and run the below code. It will give you a better understanding of how thresholding works (you can use any image of your choice if you feel like experimenting!).
First, we’ll import the required libraries.
Let’s read the downloaded image and plot it:
It is a three-channel image (RGB). We need to convert it into grayscale so that we have only a single channel. Doing this will also help us better understand how the algorithm works.
Now, we want to apply a certain threshold to this image. This threshold should separate the image into two parts – the foreground and the background. Before we do that, let’s quickly check the shape of this image:
gray.shape
(192, 263)
The height and width of the image are 192 and 263, respectively. We will take the mean of the pixel values and use that as a threshold. If the pixel value exceeds our threshold, we can say it belongs to an object. The pixel value will be treated as the background if it is less than the threshold. Let’s code this:
Nice! The darker region (black) represents the background, and the brighter (white) region is the foreground. We can define multiple thresholds as well to detect multiple objects:
There are four different segments in the above image. You can set different threshold values and check how the segments are made. Some of the advantages of this method are:
However, this approach has some limitations. When there is no significant grayscale difference or an overlap of the grayscale pixel values, it becomes very difficult to get accurate segments.
What divides two objects in an image? An edge is always between two adjacent regions with different grayscale values (pixel values). The edges can be considered as the discontinuous local features of an image.
We can use this discontinuity to detect edges and hence define a boundary of the object. This helps us detect the shapes of multiple objects in a given image. Now, the question is, how can we detect these edges? This is where we can make use of filters and convolutions. Refer to this article if you need to learn about these concepts.
The below visual will help you understand how a filter convolves over an image :
Here’s the step-by-step process of how this works:
The values of the weight matrix define the output of the convolution. My advice: It helps to extract features from the input. Researchers have found that choosing some specific values for these weight matrices helps us detect horizontal or vertical edges (or even the combination of horizontal and vertical edges).
One such weight matrix is the Sobel operator. It is typically used to detect edges. The Sobel operator has two weight matrices—one for detecting horizontal edges and the other for detecting vertical edges. Let me show how these operators look, and we will then implement them in Python.
Sobel filter (horizontal) =
| 1 | 2 | 1 |
| 0 | 0 | 0 |
| -1 | -2 | -1 |
Sobel filter (vertical) =
| -1 | 0 | 1 |
| -2 | 0 | 2 |
| -1 | 0 | 1 |
Edge detection works by convolving these filters over the given image. Let’s visualize them on this article.
It should be fairly simple to understand how the edges are detected in this image. Let’s convert it into grayscale and define the sobel filter (both horizontal and vertical) that will be convolved over this image:
Now, convolve this filter over the image using the convolve function of the ndimage package from scipy.
Let’s plot these results:
Here, we can identify the horizontal and vertical edges. There is one more type of filter that can detect both horizontal and vertical edges simultaneously. This is called the laplace operator:
| 1 | 1 | 1 |
| 1 | -8 | 1 |
| 1 | 1 | 1 |
Let’s define this filter in Python and convolve it on the same image:
Next, convolve the filter and print the output:
Here, we can see that our method has detected both horizontal and vertical edges. I encourage you to try it on different images and share your results. Remember, the best way to learn is by practicing!
This idea might have come to you while reading about image segmentation techniques. Can’t we use clustering techniques to divide images into segments? We certainly can!
In this section, we’ll get an intuition of clustering (it’s always good to revise certain concepts!) and how to use it to segment images.
Clustering is dividing the population (data points) into many groups, such that data points in the same groups are more similar to other data points in that group than those in other groups. These groups are known as clusters.
One of the most commonly used clustering algorithms is k-means. Here, the k represents the number of clusters (not to be confused with k-nearest neighbor). Let’s understand how k-means works:
The key advantage of using the k-means algorithm is that it is simple and easy to understand. We are assigning the points to the clusters closest to them.
Let’s put our learning to the test and check how well k-means segment the objects in an image. We will be using this image, so download it, read it and, check its dimensions:
It’s a 3-dimensional image of shape (192, 263, 3). To cluster the image using k-means, we first need to convert it into a 2-dimensional array whose shape is (length* width* channels). In our example, this will be (192* 263, 3).
(50496, 3)
The image has been converted to a 2-dimensional array. Next, the k-means algorithm is fitted to this reshaped array to obtain the clusters. The cluster_centers_ function of k-means returns the cluster centers, and the labels _ function gives us the label for each pixel (it tells us which pixel of the image belongs to which cluster).
I have chosen 5 clusters for this article, but you can play around with this number and check the results. Now, let’s return the clusters to their original shape, a 3-dimensional image, and plot the results.
Amazing, isn’t it? We can segment the image pretty well using just 5 clusters. I’m sure you’ll be able to improve the segmentation by increasing the number of clusters.
K-means works well when we have a small dataset. It can segment the objects in the image and give impressive results. However, when applied to a large dataset (more images), the algorithm hits a roadblock.
It looks at all the samples at every iteration, so the time taken is too high. Hence, it’s also too expensive to implement. And since k-means is a distance-based algorithm, it only applies to convex datasets and is unsuitable for clustering non-convex clusters.
Finally, let’s look at a simple, flexible, and general approach for segmentation in image processing.
Data scientists and researchers at Facebook AI Research (FAIR) pioneered a deep learning architecture called Mask R-CNN that can create a pixel-wise mask for each object in an image. This is a cool concept so follow along closely!
Mask R-CNN is an extension of the popular Faster R-CNN object detection architecture. Mask R-CNN adds a branch to the already existing Faster R-CNN outputs. The Faster R-CNN method generates two things for each object in the image:
Mask R-CNN adds a third branch to this, which also outputs the object mask. Take a look at the below image to get an intuition of how Mask R-CNN works on the inside:
Mask R-CNN is the current state-of-the-art for image segmentation techniques and runs at 5 fps.
Image classification, object detection, and image segmentation techniques are all fundamental tasks in computer vision that analyze image content, but they answer different questions about the image:
I have summarized the different image segmentation algorithms in the below table.. I suggest keeping this handy next time you’re working on an image segmentation challenge or problem!
| Algorithm | Description | Advantages | Limitations |
| Region-Based Segmentation | Separates the objects into different regions based on some threshold value(s). | When there is no significant grayscale difference or an overlap of the grayscale pixel values, it becomes difficult to get accurate segments. | It is not suitable when there are too many edges in the image and if there is less contrast between objects. |
| Edge Detection Segmentation | It is good for images to have better contrast between objects. | a. Simple calculations
b. Fast operation speed c. When the object and background have high contrast, this method performs well | Works well on small datasets and generates excellent clusters. |
| Segmentation based on Clustering | Divides the pixels of the image into homogeneous clusters. | a. Simple, flexible, and general approach
b. It is also the current state-of-the-art for image segmentation | a. Computation time is too large and expensive.
b. k-means is a distance-based algorithm. It is not suitable for clustering non-convex clusters. |
| Mask R-CNN | Gives three outputs for each object in the image: its class, bounding box coordinates, and object mask | a. Simple, flexible and general approach
b. It is also the current state-of-the-art for image segmentation | High training time |
Image segmentation techniques are fundamental in image processing. They divide a digital image into meaningful parts, like partitioning it into different regions containing pixels with similar characteristics. This simplifies the image and allows for a more focused analysis of specific objects or areas of interest.
Here’s a breakdown of what image segmentation is and how it works:
Purpose:
How it Works:
This article is just beginning our journey to learn about image segmentation. In the next article of this series, we will explore the implementation of Mask R-CNN. So stay tuned!
The image segmentation algorithm is a useful function in my deep learning career. The level of granularity I get from these techniques is astounding. How much detail we can extract with a few lines of code always amazes me.
Hope you like the article and get a clear understanding of image segmentation techniques and types of image segmentation also image segmentation in image processing, image segmentation indeep learning, What is image segmentation . These you have cleared get everything about the image segmentation topic.
Part 2 of this series is also live now: Computer Vision Tutorial: Implementing Mask R-CNN for Image Segmentation (with Python Code)
If you’re new to deep learning and computer vision, I recommend the below resources to get an understanding of the key concepts:
A. There are mainly 4 types of image segmentation: region-based segmentation, edge detection segmentation, clustering-based segmentation, and mask R-CNN.
A. Clustering-based segmentation techniques such as k-means clustering are the most commonly used method for image segmentation.
A. Image segmentation employs various methods, including thresholding, region-based segmentation, edge detection, and clustering algorithms like K-means and Gaussian mixture models. Each method aims to partition an image into distinct regions or objects based on certain criteria such as color, intensity, texture, or spatial proximity.
A. Image segmentation offers several advantages, including object recognition, image understanding, feature extraction, and image compression. By dividing an image into meaningful segments, it facilitates more precise analysis and manipulation of specific areas of interest, leading to enhanced accuracy in tasks like object detection, classification, and tracking in computer vision applications.
Two main ways to segment images:
Classic methods: Analyze pixel features (color, location) for grouping. Examples: thresholding, region-based segmentation (clustering), edge detection.
Deep learning: Powerful neural networks learn from data to segment. Uses convolutional neural networks (CNNs) for complex tasks.
Lorem ipsum dolor sit amet, consectetur adipiscing elit,
Hello Pulkit, I am looking for an expert in machine learning like you , i have a project to detect missing bolts, screws, nuts, washers.... from an item or a device for example car dynamo. I would like your help in that. can you please contact me [email protected]
Hi Amr, This would be an interesting project. Here you would require a training dataset with labeled pixels. Once you have the dataset with you, you can train an image segmentation model such as Mask R-CNN and use that trained model to get predictions for new images.
iam doing my research on image segmentation for lidar point cloud, can you suggest me which model is better for segmentation
Excellent article!! Explained in very simple way.
Glad you liked it Vijit!
This is exactly what I need! Thank you! When will part II be live?
Hi Libo, Glad you liked the post! I am currently working on Part II and it will be published soon.
Excellent article. Well explained, well detailed with clear examples. Thank you
Glad you liked it Vincente!
segmentation knowledge is so simplified. I learnt a lot. thank you so much. I would like to segment dental x-ray images to extract regions of interest, label and train a system. How do i go about it please.
Hi, To approach this problem, you first need some labelled images. By labelled images, I mean that you must have few images for which you have the region of interest. Once you have the labelled images, you can train a segmentation model like Mask RCNN and once the model is trained, you can give new images to it and it will predict the region of interest for those new set of images.
Hello, Pulkit...Can you show me the codes for comparison of Kmeans and Simple linear iterative clustering( slic) in python using skimage. Also the implementation of region growing in python using Skimage. Thank you so much.
Hi, It's quite simple to understand..! Good job
Glad you liked it Mamatha!
Hello. I am learning Computer Vision and Deep Learning. Can you please advise me how to go about it? I want to learn image segmentation in depth as of now. I really found your article very useful.
Hi Srishti, You can go through the part 2 of this series: https://www.analyticsvidhya.com/blog/2019/07/computer-vision-implementing-mask-r-cnn-image-segmentation/
sir i want to find the count of red and gray cell from a box using opencv can u guide me little bit sir please
Hello Pulkit, Thanks for your article. Very interesting. I understood the article for image segmentation techniques with datasets that include only images. But at the end you add Mask R-CNN, which require labeled pixels or masks, if started the project from scratch (not based on ImageNet, Coco, etc. type of images). Please could you confirm my assumption that Mask R-CNN should have an extra entry in the comparison table with this requirement. Thanks!
Hi Peluso, Yes! If you want to train your own Mask R-CNN model, you have to have a training set which will require the labeled pixels of the masks. But you can also use the pre-trained models to perform image segmentation using Mask R-CNN and in this case you will not require labeled pixels. You can refer to the part 2 of this series here and understand how to use pre-trained weights to perform image segmentation using Mask RCNN.
Hi i am Himanshu, I am trying to create boundry of perticular farm of satelite data of whole gujarat ...i have training data (shp file) but i don't know how to do !
Hello Pulkit, I am new in image segmentation topic, but you tutorial helped me a lot. It is very well explained and easy to understand. Thank you and looking forward more from you.
Hi Mahnaz, Thank you for your response. You can access the Part 2 of this series here: https://www.analyticsvidhya.com/blog/2019/07/computer-vision-implementing-mask-r-cnn-image-segmentation/
Hi, I have used ICnet and Deeplab v3+ for image segmentation, I managed to run them on image sequences but problem is HOW TO SAVE SEGMENTED PNG file for those images? PLT is saving only the last segmented file. I am using plt.savefig. Can you please help?
Great stuff Pulkit! When can we expect the tutorial on implementation of Mask R-CNN?
Glad you liked it kumar. The part 2 of this series is now live. You can access it here: https://www.analyticsvidhya.com/blog/2019/07/computer-vision-implementing-mask-r-cnn-image-segmentation/
Hello, I’m trying for image segmentation using K-Means and I tried the above specified code but my clustered image is not not getting displayed(getting blank white image) .I checked the cluster centers and labels by printing them and there wasn’t any problem in the clustering.I’m not able to display the clustered image.I get this message before cluster_pic gets displayed "Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers)." which is blank white image.Can anyone help me out . Thank you
What if i wanna do semantic segmentation to find tic tac toe box in my image, can i use my traning dataset consisting of images which only contain tic tac toe images completely and a set of equal no. of garbage images as other class?
Hi Shivam, You can do that. You have to train your own model first using your data and then use that trained model to make predictions for new images.
Hi Pulkit, Thanks for this insightful article. I am working on a vehicle damage assessment problem using images. I am researching on ideal approach and sample size. Based on this article, one approach came in my mind, I can segment the image in all possible ROIs, but is there a way I can label selected regions out of all ROIs for my selected damage classes (may be 8-10) and then train the model? Please let me know your thoughts on this. Thanks, Ajay
Hi Ajay, As per my understanding, you will label the images as either 8, 9 or 10. This will turn the problem into an image classification problem where your aim will become to classify images into different damage categories. Then you have to use different approach to solve this problem. For image classification, you can refer this article: https://www.analyticsvidhya.com/blog/2019/01/build-image-classification-model-10-minutes/
Hi Pulkit, Its a great article on imge segmentation and easy to underdtand those who are beginers in python based image processing. I am working on the project liver segmentation on CT scan image using Unet architecture. In this regard I need your help to read CT dataset (CT images and Labels/masks) to provide input to my network. I am new to CT images and finding lot of difficulties in reading this the database and preprocessing it for network. Your help will really move my work ahead . Please provide some directions to me. I am Hoping for your postive reply. Please respond me on my email ID [email protected]
Hi, I would like to do image segmentation for quality checking of winding in transformer.in that I done threshold and histogram and get array value of image of winding of transformer mainly autotransformer consist of 200 turns.will u please give more idea.
i want source code for extract multiple text from complex image please help me ...
Superbly written! Very much informative and lot many good new things to learn. Thanks a lot.
Thank you Sandeep!
Great Article, the way you have articulated about pixel level details on R CNN was great. Can you also provide a comparison between RCNN's competitors? Also a step by step guide on training RCNN if possible. Great job! keep writing. Thanks, Arun
Hi Arun, Here is a step by step implementation of Faster RCNN: https://www.analyticsvidhya.com/blog/2018/11/implementation-faster-r-cnn-python-object-detection/ Also, if you wish to look at the step by step implementation of Mask RCNN, here is the guide: https://www.analyticsvidhya.com/blog/2019/07/computer-vision-implementing-mask-r-cnn-image-segmentation/
hi bro, i have a doubt can u please help me, i want to recognize the " bud" of a cotton plant and also want to recognize the "cotton plant", which algorithm is helpful or which method of segmentation is useful
Hy i have a problem of my research i have dataset which based on images and every images there is multi digits but i want to recognize some specific digits. help me plzz
Really useful article. Basic concepts of image segmentation are cleared very well.
Glad you liked it Suchita!
this was very help full thank you very much
Thank you Ahmed!
Hi there I need your help please.. I have to do my project in this feild in mask RCNN ...can you help me please?
Hi Yassmein, I would really be glad to help. Please elaborate your project and then I can guide you.
Hi Pulkit, It is a great article. I am new to this feild. I am doing PhD in machine vision. I would like to know that what should be configuration of laptop so that i can run program.
Hi Manoj, These are the basic techniques and does not require high computation power. You can even run this on CPU. But, if you wish to run Mask R-CNN, then you must have a GPU with more than 32 GB RAM.
Hi Pulkit, I'm doing tests to identify the various parts of a car (fender, bonnet, bumper ...). At the moment I am trying with the fender and I have about fifty training images but the results are not very good. I just want to ask you if you believe you can get good results to decide whether to continue with the tests or give up. Maybe you could give me some indication of the number of images needed to get good results. Fulvio
When doing your own labeling/annotating of a custom dataset, how precise do you have to be with your polygons? I'm looking at labeling images of bricks with cracks or chips in them. If I bound in some of the surrounding good brick (not a lot) is that a problem? Are there any good communities where this kind of stuff is discussed? Thanks, and great post!
gray_r = gray.reshape(gray.shape[0]*gray.shape[1]) for i in range(gray_r.shape[0]): if gray_r[i] > gray_r.mean(): gray_r[i] = 1 else: gray_r[i] = 0 gray = gray_r.reshape(gray.shape[0],gray.shape[1]) plt.imshow(gray, cmap='gray') very good article, congratulations, nevertheless I have a small contribution: I would suggest to calculate the mean of gray_r outside the loop, ie immediately after the reshape function, which will improve the complexity of the algorithm. if not you will observe a slowness of execution when you take a large image Congratulation
Hi, Thank you and that's really a valuable point to reduce the execution time.
Hi there, i would love to focus on segmentation techniques for research. What is being worked on currently.
great article 10x
Thank You!
very interesting and helpful. is there any technique which can merge two segmentation algorithm example region base and edge base and get a single output.
If the image contain text means how did I segment the text from the background image. Is it same like the object detection..
Thanks for sharing this content with us, your content is really helpful for me, you describe it very well about Image Segmentation Techniques and keep posting.
Very informative post! Image segmentation is a crucial step in many image processing applications, and it's interesting to learn about the different types of segmentation. Can't wait to see more posts like this!
Excellent material for understanding Segmentation of images, also the related materials CNN, Mask RCNN are composed brilliantly.
Good. I liked it. Short , simple and precise. Make some more like this please. Thanks