How to Build a Conversational AI Chatbot with Stream Chat and React

Modern chat applications are increasingly incorporating voice input capabilities because they offer a more engaging and versatile user experience. This also improves accessibility, allowing users with different needs to interact more comfortably with such applications.

In this tutorial, I’ll guide you through the process of creating a conversational AI application that integrates real-time chat functionality with voice recognition. By leveraging Stream Chat for robust messaging and the Web Speech API for speech to text conversion, you’ll build a multi-faceted chat application that supports both voice and text interaction.

Table of Contents

• Prerequisites

• Sneak Peek

• Core Technologies

• Backend Implementation Guide

  • Project Setup

• Frontend Implementation Guide

  • Project Setup

  • Understanding the App Component

  • Adding AI to the Channel

  • Configuring the ChannelHeader

  • Adding an AI State Indicator

  • Building the Speech to Text Functionality

• Complete Process Flow

• Conclusion

Prerequisites

Before we begin, ensure you have the following:

• A Stream account with an API key and secret (Read on how to get them here)

• Access to an LLM API (like OpenAI, Anthropic).

• Node.js and npm/yarn installed.

• Basic knowledge of React and TypeScript.

• Modern browser with WebSpeech API support (like Chrome, Edge)

Sneak Peek

Let’s take a quick look at the app we’ll be building in this tutorial. This way, you get a feel for what it does before we jump into the details.

If you’re now excited, let’s get straight into it!

Core Technologies

This application is powered by three main players: Stream Chat, the Web Speech API, and a Node.js + Express backend.

Stream Chat is a platform that helps you easily build and integrate rich, real-time chat and messaging experiences into your applications. It offers a variety of SDKs (Software Development Kits) for different platforms (like Android, iOS, React) and pre-built UI components to streamline development. Its robustness and engaging chat functionality make it a great choice for this app – we don’t need to build anything from scratch.

Web Speech API is a browser standard that allows you to integrate voice input and output into your apps, enabling features like speech recognition (converting spoken speech to text) and speech synthesis (converting text to speech). We’ll use the speech recognition feature in this project.

The Node.js + Express backend manages correct agent instantiation and processes the conversational responses generated by our LLM API.

Backend Implementation Guide

Let’s begin with our backend, the engine room – where user input is routed to the appropriate AI model, and a processed response is returned. Our backend supports multiple AI models, specifically OpenAI and Anthropic.

Project Setup

1. Create a folder, call it ‘My-Chat-Application’.

2. Clone this Github repository

3. After cloning, rename the folder to ‘backend’

4. Open the .env.example file and provide the necessary keys (you’ll need to provide either the OpenAI or Anthropic key – the Open Weather key is optional).

5. Rename the env.examplefile to .env

6. Install dependencies by running this command:

    npm install
   

7. Run the project by entering this command:

    npm start
   

   Your backend should be running smoothly on localhost:3000.

Frontend Implementation Guide

This section explores two broad, interrelated components: the chat structure and speech recognition.

Project Setup

We will be creating and setting up our React project with the Stream Chat React SDK. We'll use Vite with the TypeScript template. To do that, navigate to your My-Chat-Application folder, open your terminal and enter this command:

npm create vite frontend -- --template react-ts
cd chat-example
npm i stream-chat stream-chat-react

With our frontend project set up, we can now run the app:

npm run dev

Understanding the App Component

The main focus here is to initialize a chat client, connect a user, create a channel, and render the chat interface. We’ll go through all these processes step by step to help you understand them better:

Define Constants

First, we need to provide some important credentials that we need for user creation and chat client setup. You can find these credentials on your Stream dashboard.

const apiKey = "xxxxxxxxxxxxx";
const userId = "111111111";
const userName = "John Doe";
const userToken = "xxxxxxxxxx.xxxxxxxxxxxx.xx_xxxxxxx-xxxxx_xxxxxxxx"; //your stream secret key

Note: These are dummy credentials. Make sure to use your own credentials.

Create a User

Next, we need to create a user object. We’ll create it using an ID, name and a generated avatar URL:

const user: User = {
  id: userId,
  name: userName,
  image: `https://getstream.io/random_png/?name=${userName}`,
};

Setup a Client

We need to track the state of the active chat channel using the useState hook to ensure seamless real-time messaging in this Stream Chat application. A custom hook called useCreateChatClient initializes the chat client with an API key, user token, and user data:

  const [channel, setChannel] = useState();
  const client = useCreateChatClient({
    apiKey,
    tokenOrProvider: userToken,
    userData: user,
  });

Initialize Channel

Now, we initialize a messaging channel to enable real-time communication in the Stream Chat application. When the chat client is ready, the useEffect hook triggers the creation of a messaging channel named my_channel, adding the user as a member. This channel is then stored in the channel state, ensuring that the app is primed for dynamic conversation rendering.

  useEffect(() => {
    if (!client) return;
    const channel = client.channel("messaging", "my_channel", {
      members: [userId],
    });

    setChannel(channel);
  }, [client]);

Render Chat Interface

With all the integral parts of our chat application all set up, we’ll return a JSX to define the chat interface's structure and components:

 if (!client) return <div>Setting up client & connection...div>;

  return (
    <Chat client={client}>
      <Channel channel={channel}>
        <Window>
          <MessageList />
          <MessageInput />
        Window>
        <Thread />
      Channel>
    Chat>
  );

In this JSX structure:

• If the client is not ready, it displays a "Setting up client & connection..." message.

• Once the client is ready, it renders the chat interface using:

  • : Wraps the Stream Chat context with the initialized client.

  • : Sets the active channel.

  • : Contains the main chat UI components:

    • : Displays the list of messages.

    • : Uses a custom CustomMessageInput for sending messages.

  • : Renders threaded replies.

With this, we've set up our chat interface and channel, and we have a client ready. Here's what our interface looks like so far:

Adding AI to the Channel

Remember, this chat application is designed to interact with an AI, so we need to be able to both add and remove the AI from the channel. On the UI, we’ll add a button in the channel header to enable users add and remove AI. But we still need to determine whether or not we already have it in the channel to know which option to display.

For that we’ll create a custom hook called useWatchers. It monitors the presence of the AI using a concept called watchers:

import { useCallback, useEffect, useState } from 'react';
import { Channel } from 'stream-chat';

export const useWatchers = ({ channel }: { channel: Channel }) => {
  const [watchers, setWatchers] = useState([]);
  const [error, setError] = useState<Error | null>(null);

  const queryWatchers = useCallback(async () => {
    setError(null);

    try {
      const result = await channel.query({ watchers: { limit: 5, offset: 0 } });
      setWatchers(result?.watchers?.map((watcher) => watcher.id).filter((id): id is string => id !== undefined) || [])
      return;
    } catch (err) {
      setError(err as Error);
    }
  }, [channel]);

  useEffect(() => {
    queryWatchers();
  }, [queryWatchers]);

  useEffect(() => {
    const watchingStartListener = channel.on('user.watching.start', (event) => {
      const userId = event?.user?.id;
      if (userId && userId.startsWith('ai-bot')) {
        setWatchers((prevWatchers) => [
          userId,
          ...(prevWatchers || []).filter((watcherId) => watcherId !== userId),
        ]);
      }
    });

    const watchingStopListener = channel.on('user.watching.stop', (event) => {
      const userId = event?.user?.id;
      if (userId && userId.startsWith('ai-bot')) {
        setWatchers((prevWatchers) =>
          (prevWatchers || []).filter((watcherId) => watcherId !== userId)
        );
      }
    });

    return () => {
      watchingStartListener.unsubscribe();
      watchingStopListener.unsubscribe();
    };
  }, [channel]);

  return { watchers, error };
};

Configuring the ChannelHeader

We can now build a new channel header component by utilizing the useChannelStateContext hook to access the channel and initialize the custom useWatchers hook. Using the watchers' data, we define an aiInChannel variable to display relevant text. Based on this variable, we invoke either the start-ai-agent or stop-ai-agent endpoint on the Node.js backend.

import { useChannelStateContext } from 'stream-chat-react';
import { useWatchers } from './useWatchers';

export default function ChannelHeader() {
  const { channel } = useChannelStateContext();
  const { watchers } = useWatchers({ channel });

  const aiInChannel =
    (watchers ?? []).filter((watcher) => watcher.includes('ai-bot')).length > 0;
  return (
    <div className='my-channel-header'>
      <h2>{(channel?.data as { name?: string })?.name ?? 'Voice-and-Text AI Chat'}h2>
      <button onClick={addOrRemoveAgent}>
        {aiInChannel ? 'Remove AI' : 'Add AI'}
      button>
    div>
  );

  async function addOrRemoveAgent() {
    if (!channel) return;
    const endpoint = aiInChannel ? 'stop-ai-agent' : 'start-ai-agent';
    await fetch(`http://127.0.0.1:3000/${endpoint}`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ channel_id: channel.id, platform: 'openai' }),
    });
  }
}