Image captioning is another exciting innovation in artificial intelligence and its contribution to computer vision. Salesforce’s new tool, BLIP, is a great leap. This image captioning AI model provides a great deal of interpretation through its working process. Bootstrapping Language-image Pretraining (BLIP) is a technology that generates captions from images with a high level of efficiency.
This article was published as a part of the Data Science Blogathon.
The BLIP image captioning model uses an exceptional deep learning technique to interpret an image into a descriptive caption. It also effortlessly generates image-to-text with high accuracy using natural language processing and computer vision.
You can explore this model with several key features. Using a few text prompts allows you to get the most descriptive part of an image. You can easily find these prompts when you upload an image to the Salesforce BLIP captioning tool on a hugging face. Their functionalities are also great and effective.
With this model, you can ask questions about the details of an uploaded picture’s colors or shape. They also use beam search and nucleus features to provide a descriptive image caption.
This model has great accuracy and precision in recognizing objects and exhibiting real-life processing when providing captions to images. There are several features to explore with this tool. However, three main features define the capability of the BLIP image captioning tool. We’ll briefly discuss them here;
The context of an image is the game-changing detail that helps in the interpretation and captioning. For example, a picture of a cat and a mouse would not have a clear context if no relationship existed between them. Salesforce BLIP can understand the relationship between objects and use spatial arrangements to generate captions. This key functionality can help create a human-like caption, not just a generic one.
So, your image gets a caption with a clear context, such as “a cat chasing a mouse under the table.” This generates a better context than a caption that reads “a cat and a mouse.”
Salesforce’s quest to cater to the global audience encouraged the implementation of multiple languages for this model. So, using this model as a marketing tool can benefit international brands and businesses.
The fact that BLIP allows for real-time processing of images makes it a great asset. Using BLIP image captioning as a marketing tool can benefit from this function. Live event coverage, chat support, social media engagement, and other marketing strategies can be implemented.
BLIP Image Captioning employs a Vision-Language Pre-training (VLP) framework, integrating understanding and generation tasks. It effectively leverages noisy web data through a bootstrapping mechanism, where a captioner generates synthetic captions filtered by a noise removal process.
This approach achieves state-of-the-art results in various vision-language tasks like image-text retrieval, image captioning, and Visual Question Answering (VQA). BLIP’s architecture enables flexible transferability between vision-language understanding and generation tasks.
Notably, it demonstrates strong generalization ability in zero-shot transfers to video-language tasks. The model is pre-trained on the COCO dataset, which contains over 120,000 images and captions. BLIP’s innovative design and utilization of web data set it apart as a pioneering solution in unified vision-language understanding and generation.
BLIP uses the Vision Transformer ViT. This mechanism encodes the image input by dividing it into patches, with an additional token representing the global image feature. This process uses less computational costs, making it an easier model.
This model uses a unique training/pretraining method to generate tasks and understand functionalities. BLIP adopts a multimodal mixture of Encoder and Decoder to transmit its main functionalities: Text Encoder, Image ground text encoder, and decoder.
Here is a graphical representation of how this works;
This model runs smoothly using several runtimes. Due to varying development environments, we run inferences on GPUs and CPUs to see how this model generates image captions.
Let’s look into running the Salesforce BLIP Image captioning on GPU (In full precision)
The first line allows HTTP requests in Python. Then, the PIL helps import the image module from the library, allowing the opening, changing, and saving of images in different formats.
The next step is loading the processor from Salesforce/Blip image captioning. This is where the processor’s initialization starts. It is carried out by loading the pre-trained processor configuration and tokenization associated with this model.
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")The variable ‘img_url’ indicates the image to be downloaded after using PIL’s image. In the open function, you can view the URL’s raw image after it has been downloaded.
img_url = 'https://www.shutterstock.com/image-photo/young-happy-schoolboy-using-computer-600nw-1075168769.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')When you enter a new code block and type ‘raw image,’ you will be able to get a view of the image as shown below:
This model captions images in two ways: conditional and unconditional image captioning. For the former, the input is your raw image, text (which sends a request for the image caption based on the text), and then the ‘generate’ function gives out processed input.
On the other hand, unconditional image captioning can provide captions without text input.
# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))Let’s look into running the BLIP Image captioning on GPU (In half-precision)
This step imports the necessary libraries and requests in Python. The other steps include the BLIP image generation model and a processor for loading pre-trained configuration and tokenization.
import torch
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large", torch_dtype=torch.float16).to("cuda")When you have the image URL, PIL can do the job from here, as opening the picture would be easy.
img_url = 'https://www.shutterstock.com/image-photo/young-happy-schoolboy-using-computer-600nw-1075168769.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')Here again, we talk about the conditional and unconditional image captioning methods and you can write something more than “a photography of” to get other information on the image. But for this case, we want just a caption;
# unconditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
#import csvLet’s look into running the BLIP Image captioning on CPU runtime.
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGenerationprocessor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")img_url = 'https://www.shutterstock.com/image-photo/young-happy-schoolboy-using-computer-600nw-1075168769.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))The BLIP Image captioning model’s ability to generate captions from images provides great value to many industries, especially digital marketing. Let’s explore a few real-life applications of the BLIP image captioning model.
Image captioning has become a valuable development in AI today. This model helps in many ways with this development. Leveraging advanced natural language processing techniques, this setup equips developers with powerful tools for generating accurate captions from images.
Here are some notable points from the BLIP Image captioning model;
Ans. BLIP image captioning model is not only accurate at detecting objects. Its understanding of spatial arrangement provides an edge contextually when giving the image caption.
Ans. This model satisfies a global audience as it supports multiple languages. BLIP Image captioning is also unique because it can process captions in real-time.
Ans. For conditional image captioning, BLIP provides captions to images using text prompts. On the other hand, this model can carry out unconditional captioning based on the image alone.
Ans. BLIP employs a Vision-Language Pre-training (VLP) framework, utilizing a bootstrapping mechanism to leverage noisy web data effectively. It achieves state-of-the-art results across various vision-language tasks.
The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.
Hey there! I'm David Maigari a dynamic professional with a passion for technical writing writing, Web Development, and the AI world. David is an also enthusiast of data science and AI innovations.
Lorem ipsum dolor sit amet, consectetur adipiscing elit,
