How to Build Your AI Chatbot with NLP in Python?

Introduction

In this article, we will create an AI chatbot using Natural Language Processing (NLP) in Python. Our goal is to help you build a smart chatbot. First, we’ll explain NLP, which helps computers understand human language. Then, we’ll show you how to use AI to make a chatbot to have real conversations with people. Finally, we’ll talk about the tools you need to create a chatbot like ALEXA or Siri. Also, We Will tell in this article how to create ai chatbot projects with that we give highlights for how to craft Python ai Chatbot.

This guide is hands-on. We’ll give you step-by-step instructions, and you can follow our examples or change them to fit your needs. So, let’s begin this journey into the world of NLP and AI chatbots!

This article was published as a part of the Data Science Blogathon.

Quiz Time

Get ready for the Ultimate Trivia Challenge! Put your knowledge to the test and see how many questions you can answer correctly.

Start Quiz

Question

Your answer:

Correct answer:

Next

You got {{SCORE_CORRECT}} out of {{SCORE_TOTAL}}

Your Answers

Introduction to AI Chatbot

Were you ever curious as to how to build a talking ChatBot with Python and also have a conversation with your own personal AI?

As the topic suggests we are here to help you have a conversation with your AI today. To have a conversation with your AI, you need a few pre-trained tools which can help you build an AI chatbot system. In this article, we will guide you to combine speech recognition processes with an artificial intelligence algorithm.

Natural Language Processing or NLP is a prerequisite for our project. NLP allows computers and algorithms to understand human interactions via various languages. In order to process a large amount of natural language data, an AI will definitely need NLP or Natural Language Processing. Currently, we have a number of NLP research ongoing in order to improve the AI chatbots and help them understand the complicated nuances and undertones of human conversations.

AI Chatbots are applications that businesses and other entities employ to facilitate automated conversations between AI and humans. These conversations can occur through text or speech. Chatbots must comprehend and imitate human conversation when engaging with people worldwide. Chatbots have made significant progress from ELIZA, the first-ever chatbot, to today’s Amazon ALEXA. This tutorial will cover all the fundamental concepts necessary for creating a basic chatbot that can comprehend and respond to human interactions. We will utilize speech recognition APIs and pre-trained Transformer models.

What is NLP?

NLP, or Natural Language Processing, stands for teaching machines to understand human speech and spoken words. NLP combines computational linguistics, which involves rule-based modeling of human language, with intelligent algorithms like statistical, machine, and deep learning algorithms. Together, these technologies create the smart voice assistants and chatbots we use daily.

In human speech, there are various errors, differences, and unique intonations. NLP technology, including AI chatbots, empowers machines to rapidly understand, process, and respond to large volumes of text in real-time. You’ve likely encountered NLP in voice-guided GPS apps, virtual assistants, speech-to-text note creation apps, and other chatbots that offer app support in your everyday life. In the business world, NLP, particularly in the context of AI chatbots, is instrumental in streamlining processes, monitoring employee productivity, and enhancing sales and after-sales efficiency.

Tasks in NLP

Interpreting and responding to human speech presents numerous challenges, as discussed in this article. Humans take years to conquer these challenges when learning a new language from scratch. Programmers, with the help of AI chatbot technology, have integrated various functions into NLP technology to tackle these hurdles and create practical tools for understanding human speech, processing it, and generating suitable responses.

NLP tasks involve breaking down human text and audio signals from voice data in ways that computers can analyze and convert into comprehensible data. Some of the tasks in NLP data ingestion include:

1. Speech Recognition: This process involves converting speech into text, a crucial step in speech analysis. Within speech recognition, there is a subprocess called speech tagging, which allows a computer to break down speech and add context, accents, or other speech attributes.
2. Word Sense Disambiguation: In human speech, a word can have multiple meanings. Word sense disambiguation is a semantic analysis that selects the most appropriate meaning for a word based on its context. For instance, it helps determine whether a word functions as a verb or a pronoun.
3. Named Entity Recognition (NER): NER identifies words and phrases as specific entities, such as recognizing “Dev” as a person’s name or “America” as the name of a country.
4. Sentiment Analysis: Human speech often contains sentiments and undertones. Extracting these nuances and hidden emotions, like attitude, sarcasm, fear, or joy, is one of the most challenging tasks undertaken by NLP processes.

Types of AI Chatbots

AI Chatbots are a relatively recent concept and despite having a huge number of programs and NLP tools, we basically have just two different categories of chatbots based on the NLP technology that they utilize. These two types of chatbots are as follows:

Scripted Chatbots

Scripted ai chatbots are chatbots that operate based on pre-determined scripts stored in their library. When a user inputs a query, or in the case of chatbots with speech-to-text conversion modules, speaks a query, the chatbot replies according to the predefined script within its library. One drawback of this type of chatbot is that users must structure their queries very precisely, using comma-separated commands or other regular expressions, to facilitate string analysis and understanding. This makes it challenging to integrate these chatbots with NLP-supported speech-to-text conversion modules, and they are rarely suitable for conversion into intelligent virtual assistants.

Artificially Intelligent Chatbots

Artificially intelligent ai chatbots, as the name suggests, are designed to mimic human-like traits and responses. NLP (Natural Language Processing) plays a significant role in enabling these chatbots to understand the nuances and subtleties of human conversation. When NLP is combined with artificial intelligence, it results in truly intelligent chatbots capable of responding to nuanced questions and learning from each interaction to provide improved responses in the future. AI chatbots find applications in various platforms, including automated chat support and virtual assistants designed to assist with tasks like recommending songs or restaurants.

How to create your own AI chatbot Projects ?

Creating your own AI chatbot project can be an exciting and rewarding experience. There are two main approaches to consider, depending on your coding experience:

1. No-code chatbot builders:

• Suitable for: Beginners with no coding experience.
• Pros: Easy to use, fast development, user-friendly interfaces.
• Cons: Limited customization, may not be suitable for complex chatbots.

If you’re new to chatbot development, no-code chatbot builders are a great place to start. These platforms allow you to design conversation flows using a visual interface, drag-and-drop functionality, and pre-built elements. Some popular options include:

• Chatfuel
• ManyChat
• Tidio

2. Coding your own chatbot:

• Suitable for: Developers with experience in programming languages like Python.
• Pros: Highly customizable, greater control over functionality.
• Cons: Steeper learning curve, requires coding knowledge.

For more advanced users, building a chatbot from scratch using code offers more flexibility and customization. Python is a popular choice due to its readability and libraries like ChatterBot that simplify chatbot development. Here’s a general roadmap to follow:

1. Define your purpose: What problem will your chatbot solve? Who is your target audience?
2. Choose your development method: No-code builder or coding with Python.
3. Design the conversation flow: Plan how users will interact with your chatbot.
4. Train your chatbot: Feed your chatbot with data (text, examples) to understand user queries.
5. Test and refine: Continuously test and improve your chatbot based on user interactions.

Challenges for your AI Chatbot

In the current world, computers are not just machines celebrated for their calculation powers. Today, the need of the hour is interactive and intelligent machines that can be used by all human beings alike. For this, computers need to be able to understand human speech and its differences.

NLP technologies have made it possible for machines to intelligently decipher human text and actually respond to it as well. However, communication amongst humans is not a simple affair. There are a lot of undertones dialects and complicated wording that makes it difficult to create a perfect chatbot or virtual assistant that can understand and respond to every human.

To overcome the problem of chaotic speech, developers have had to face a few key hurdles which need to be explored in order to keep improving these chatbots. To understand these hurdles or problems we need to under how NLP works to convert human speech into something an algorithm or AI understands. Here’s a list of snags that a chatbot hits whenever users are trying to interact with it:

1. Synonyms, homonyms, slang
2. Misspellings
3. Abbreviations
4. Complex punctuation rules
5. Accents, dialects and speech differences with the age and other issues of humans. (for eg. lisps, drawls, etc)

To a human brain, all of this seems really simple as we have grown and developed in the presence of all of these speech modulations and rules. However, the process of training an AI chatbot is similar to a human trying to learn an entirely new language from scratch. The different meanings tagged with intonation, context, voice modulation, etc are difficult for a machine or algorithm to process and then respond to. NLP technologies are constantly evolving to create the best tech to help machines understand these differences and nuances better.

Installing Packages required to Build AI Chatbot

We will begin by installing a few libraries which are as follows :

Code:

# To be able to convert text to Speech
! pip install SpeechRecognition  #(3.8.1)
#To convey the Speech to text and also speak it out
!pip install gTTS  #(2.2.3)
# To install our language model
!pip install transformers  #(4.11.3)
!pip install tensorflow #(2.6.0, or pytorch)

We will start by importing some basic functions:

import numpy as np

We will begin by creating an empty class which we will build step by step. To build the chatbot, we would need to execute the full script. The name of the bot will be “ Dev”

# Beginning of the AI
class ChatBot():
    def __init__(self, name):
        print("----- starting up", name, "-----")
        self.name = name
# Execute the AI
if __name__ == "__main__":
    ai = ChatBot(name="Dev")

Output :

What is Speech Recognition?

NLP or Natural Language Processing has a number of subfields as conversation and speech are tough for computers to interpret and respond to. One such subfield of NLP is Speech Recognition. Speech Recognition works with methods and technologies to enable recognition and translation of human spoken languages into something that the computer or AI chatbot can understand and respond to.

For computers, understanding numbers is easier than understanding words and speech. When the first few speech recognition systems were being created, IBM Shoebox was the first to get decent success with understanding and responding to a select few English words. Today, we have a number of successful examples which understand myriad languages and respond in the correct dialect and language as the human interacting with it. Most of this success is through the SpeechRecognition library.

Using Google APIs

To use popular Google APIs we will use the following code:

Code:

import speech_recognition as sr
def speech_to_text(self):
    recognizer = sr.Recognizer()
    with sr.Microphone() as mic:
         print("listening...")
         audio = recognizer.listen(mic)
    try:
         self.text = recognizer.recognize_google(audio)
         print("me --> ", self.text)
    except:
         print("me -->  ERROR")

Note: The first task that our chatbot must work for is the speech to text conversion. Basically, this involves converting the voice or audio signals into text data. In summary, the chatbot actually ‘listens’ to your speech and compiles a text file containing everything it could decipher from your speech. You can test the codes by running them and trying to say something aloud. It should optimally capture your audio signals and convert them into text.

Speech to Text Conversion

# Execute the AI
if __name__ == "__main__":
     ai = ChatBot(name="Dev")
     while True:
         ai.speech_to_text()

Output :

Note:  Here I am speaking and not typing

Processing Suitable Responses

Next, our AI needs to be able to respond to the audio signals that you gave to it. In simpler words, our ai chatbot has received the input. Now, it must process it and come up with suitable responses and be able to give output or response to the human speech interaction. To follow along, please add the following function as shown below. This method ensures that the chatbot will be activated by speaking its name. When you say “Hey Dev” or “Hello Dev” the bot will become active.

Code:

def wake_up(self, text):
    return True if self.name in text.lower() else False

As a cue, we give the chatbot the ability to recognize its name and use that as a marker to capture the following speech and respond to it accordingly. This is done to make sure that the chatbot doesn’t respond to everything that the humans are saying within its ‘hearing’ range. In simpler words, you wouldn’t want your chatbot to always listen in and partake in every single conversation. Hence, we create a function that allows the chatbot to recognize its name and respond to any speech that follows after its name is called.

Fine-tuning Bot Responses

After the ai chatbot hears its name, it will formulate a response accordingly and say something back. For this, the chatbot requires a text-to-speech module as well. Here, we will be using GTTS or Google Text to Speech library to save mp3 files on the file system which can be easily played back.

The following functionality needs to be added to our class so that the bot can respond back

Code:

from gtts import gTTS
import os
@staticmethod
def text_to_speech(text):
    print("AI --> ", text)
    speaker = gTTS(text=text, lang="en", slow=False)
    speaker.save("res.mp3")
    os.system("start res.mp3")  #if you have a macbook->afplay or for windows use->start
    os.remove("res.mp3")

Code :

#Those two functions can be used like this
# Execute the AI
if __name__ == "__main__":
     ai = ChatBot(name="Dev")
     while True:
         ai.speech_to_text()
         ## wake up
         if ai.wake_up(ai.text) is True:
             res = "Hello I am Dev the AI, what can I do for you?"
         ai.text_to_speech(res)

Output :

Next, we can consider upgrading our chatbot to do simple commands like some o the virtual assistants help you to do. An example of such a task would be to equip the chatbot to be able to answer correctly whenever the user asks for the current time. To add this function to the chatbot class, follow along with the code given below:

Code:

import datetime
@staticmethod
def action_time():
    return datetime.datetime.now().time().strftime('%H:%M')
#and run the script after adding the above function to the AI class

# Run the AI
if __name__ == "__main__":
ai = ChatBot(name="Dev")
while True:
         ai.speech_to_text()
         ## waking up
         if ai.wake_up(ai.text) is True:
             res = "Hello I am Dev the AI, what can I do for you?"
         ## do any action
         elif "time" in ai.text:
            res = ai.action_time()
         ## respond politely
         elif any(i in ai.text for i in ["thank","thanks"]):
            res = np.random.choice(
                  ["you're welcome!","anytime!",
                   "no problem!","cool!",
                   "I'm here if you need me!","peace out!"])
         ai.text_to_speech(res)

Output :

After all of the functions that we have added to our chatbot, it can now use speech recognition techniques to respond to speech cues and reply with predetermined responses. However, our chatbot is still not very intelligent in terms of responding to anything that is not predetermined or preset. It is now time to incorporate artificial intelligence into our chatbot to create intelligent responses to human speech interactions with the chatbot or the ML model trained using NLP or Natural Language Processing.

The Language Model for AI Chatbot

Here, we will use a Transformer Language Model for our AI chatbot. This model, presented by Google, replaced earlier traditional sequence-to-sequence models with attention mechanisms. The AI chatbot benefits from this language model as it dynamically understands speech and its undertones, allowing it to easily perform NLP tasks. Some of the most popularly used language models in the realm of AI chatbots are Google’s BERT and OpenAI’s GPT. These models, equipped with multidisciplinary functionalities and billions of parameters, contribute significantly to improving the chatbot and making it truly intelligent.

This is where the AI chatbot becomes intelligent and not just a scripted bot that will be ready to handle any test thrown at it. The main package we will be using in our code here is the Transformers package provided by HuggingFace, a widely acclaimed resource in AI chatbots. This tool is popular amongst developers, including those working on AI chatbot projects, as it allows for pre-trained models and tools ready to work with various NLP tasks. In the code below, we have specifically used the DialogGPT AI chatbot, trained and created by Microsoft based on millions of conversations and ongoing chats on the Reddit platform in a given time.

Code:

import transformers
nlp = transformers.pipeline("conversational", 
                            model="microsoft/DialoGPT-medium")
#Time to try it out
input_text = "hello!"
nlp(transformers.Conversation(input_text), pad_token_id=50256)

Reminder: Don’t forget to provide the pad_token_id as the current version of the library we are using in our code raises a warning when this is not specified. What you can do to avoid this warning is to add this as a parameter.

nlp(transformers.Conversation(input_text), pad_token_id=50256)

You will get a whole conversation as the pipeline output and hence you need to extract only the response of the chatbot here.

Code:

chat = nlp(transformers.Conversation(ai.text), pad_token_id=50256)
res = str(chat)
res = res[res.find("bot >> ")+6:].strip()

Finally, we’re ready to run the Chatbot and have a fun conversation with our AI. Here’s the full code:

Great! The bot can both perform some specific tasks like a virtual assistant (i.e. saying the time when asked) and have casual conversations. And if you think that Artificial Intelligence is here to stay, she agrees:

Final Code:

# for speech-to-text
import speech_recognition as sr
# for text-to-speech
from gtts import gTTS
# for language model
import transformers
import os
import time
# for data
import os
import datetime
import numpy as np
# Building the AI
class ChatBot():
    def __init__(self, name):
        print("----- Starting up", name, "-----")
        self.name = name
    def speech_to_text(self):
        recognizer = sr.Recognizer()
        with sr.Microphone() as mic:
            print("Listening...")
            audio = recognizer.listen(mic)
            self.text="ERROR"
        try:
            self.text = recognizer.recognize_google(audio)
            print("Me  --> ", self.text)
        except:
            print("Me  -->  ERROR")
    @staticmethod
    def text_to_speech(text):
        print("Dev --> ", text)
        speaker = gTTS(text=text, lang="en", slow=False)
        speaker.save("res.mp3")
        statbuf = os.stat("res.mp3")
        mbytes = statbuf.st_size / 1024
        duration = mbytes / 200
        os.system('start res.mp3')  #if you are using mac->afplay or else for windows->start
        # os.system("close res.mp3")
        time.sleep(int(50*duration))
        os.remove("res.mp3")
    def wake_up(self, text):
        return True if self.name in text.lower() else False
    @staticmethod
    def action_time():
        return datetime.datetime.now().time().strftime('%H:%M')
# Running the AI
if __name__ == "__main__":
    ai = ChatBot(name="dev")
    nlp = transformers.pipeline("conversational", model="microsoft/DialoGPT-medium")
    os.environ["TOKENIZERS_PARALLELISM"] = "true"
    ex=True
    while ex:
        ai.speech_to_text()
        ## wake up
        if ai.wake_up(ai.text) is True:
            res = "Hello I am Dave the AI, what can I do for you?"
        ## action time
        elif "time" in ai.text:
            res = ai.action_time()
        ## respond politely
        elif any(i in ai.text for i in ["thank","thanks"]):
            res = np.random.choice(["you're welcome!","anytime!","no problem!","cool!","I'm here if you need me!","mention not"])
        elif any(i in ai.text for i in ["exit","close"]):
            res = np.random.choice(["Tata","Have a good day","Bye","Goodbye","Hope to meet soon","peace out!"])
            ex=False
        ## conversation
        else:   
            if ai.text=="ERROR":
                res="Sorry, come again?"
            else:
                chat = nlp(transformers.Conversation(ai.text), pad_token_id=50256)
                res = str(chat)
                res = res[res.find("bot >> ")+6:].strip()
        ai.text_to_speech(res)
    print("----- Closing down Dev -----")

Output:

Note: I had later switched from google collab to my local machine due to some module issues which I faced during implementation and hence I am sharing my experience here so that if any of you also face the same issue can solve it. Obviously, Google is also there but the following lines will explain the issue. I used Python 3.9 as it had all the modules necessary and Python 3.6 and older versions will also work. Python 3.8 or the latest version might not have all the modules ported to match the version and hence I would suggest using Python 3.9 or older versions than 3.6.

To run a file and install the module, use the command “python3.9” and “pip3.9” respectively if you have more than one version of python for development purposes. “PyAudio” is another troublesome module and you need to manually google and find the correct “.whl” file for your version of Python and install it using pip.

The link to the full code can be found here.

Bonus tips: Feel free to drop a star if you liked this tutorial or bot and feel free to fork and create your own AI chatbot and call it whatever you want!

Building Your First Python AI Chatbot

Building your first Python AI chatbot can be an exciting project! Here’s a simplified guide to get you started:

1. Choosing your tools:

There are various Python libraries that simplify chatbot development. A popular option for beginners is ChatterBot. It provides a user-friendly interface for training your chatbot with conversation data.

2. Setting up the environment:

• Install Python if you haven’t already.
• Use pip (Python’s package manager) to install ChatterBot:

pip install chatterbot

3. Building the chatbot:

• Import the necessary libraries (ChatterBot and ChatterBot’s CorpusTrainer).
• Create a chatbot instance and give it a name.
• Train the chatbot using a predefined corpus (collection of text data) that ChatterBot provides. This corpus includes common conversations and helps your chatbot understand language patterns.

4. Testing and Refining:

• Run the chatbot and interact with it! See how it responds to your questions.
• You can further train the chatbot with your custom conversation data (text files, chat logs). This personalizes its responses and makes it more relevant to your needs.

Conclusion

In this guide, we’ve provided a step-by-step tutorial for creating a conversational AI chatbot. You can use this chatbot as a foundation for developing one that communicates like a human. The code samples we’ve shared are versatile and can serve as building blocks for similar AI chatbot projects.

Consider enrolling in our AI and ML Blackbelt Plus Program to take your skills further. It’s a great way to enhance your data science expertise and broaden your capabilities. With the help of speech recognition tools and NLP technology, we’ve covered the processes of converting text to speech and vice versa. We’ve also demonstrated using pre-trained Transformers language models to make your chatbot intelligent rather than scripted.

Frequently Asked Questions

The media shown in this article on AI Chatbot is not owned by Analytics Vidhya and are used at the Author’s discretion.

Arnab 06 Aug, 2024