How to Build a Video Conferencing Application with LiveKit and Symbl.ai

This blog demonstrates how to build and use a multi-party video-conferencing application with LiveKit and Symbl.ai.

LiveKit is an open-source project that provides scalable, multi-user conferencing over WebRTC. It’s designed to provide everything you need to build real-time audio and video experiences in your applications.

A video conferencing application may not be accessible to all audiences and can be arduous to build. Modern applications need built-in intelligence to be more inclusive of all audiences.

Symbl.ai provides out-of-the-box conversation intelligence capabilities to deeply analyze the spoken conversations that occur over video applications. One of these capabilities is live transcription, or speech-to-text. This feature allows you to convert the conversations from a verbal form to a textual form and display it to users.

Combining these two platforms, we have built a multi-party video-conferencing application that demonstrates Symbl.ai’s real-time APIs in the form of Closed Captions.

This application was inspired by LiveKit’s video app and is built using LiveKit’s React SDK and Create React App.

Features

Before we get into the details on how to build Symbl.ai’s LiveKit RTC App, let’s talk about its available features. This video app integrated with Symbl.ai’s real-time APIs provides the following out-of-the-box conversation intelligence features:

• Live closed captioning: Live closed captioning is enabled by default and provides a real-time transcription of your audio content.
• Real-time transcription: Symbl.ai offers state-of-the-art speech-to-text capability (also called transcription). You can get audio and video conversations transcriptions in real-time.
  Real-time insights: Symbl.ai allows you to get the real-time insights (action items, questions and follow ups) of the conversations.
• Real-time topics with sentiments: Symbl.ai allows you to get the real-time topics of the conversations along with their sentiments.
• Real-time trackers detection: Symbl.ai allows real-time detection of the trackers in the conversation.
• Real-time sentiment analysis of the conversations: Symbl.ai allows you to get real-time sentiment analysis (sentence level) and aggregated/overall sentiment categorisation of the conversations.
• Real-time meeting metrics: Symbl.ai allows you to get real-time analytics of total talktime, total silence and total overlapping time of the meeting.
• Real-time member metrics: Demonstrates the talktime or contribution of a member in the conversations.
• Video conferencing with real-time video and audio: This allows you to use this for real-time use cases where both the video, audio, and its results from Symbl.ai’s back-end need to be available in real-time. It can be integrated directly via the browser or server.
• Enable/disable camera: After connecting your camera you can enable or disable the camera when you want.
• Mute/unmute mic: After you connect to your device’s microphone you can mute or unmute when you want.
• Screen sharing: This can be used to capture the screen directly from the web app.

Without further ado, let’s get started! 

Prerequisites

Some essential items for the checklist:

• Node JS (version 14+)
• Symbl.ai Account

Setting up Symbl.ai 

Create a free Symbl.ai account here and generate an access token.

Structure of Components

Installation

Setup

1. Clone the project.

git clone https://github.com/SymblDev/symbl-livekit-rtc-app.git

2. Go to the project directory.

cd symbl-livekit-rtc-app

3. Install dependencies.

npm install

Run Locally

1. Navigate to the src/symbl/config/ directory and open the config.js file.
2. Add your Symbl.ai App Id and App Secret values in the respective fields below:
   

appId: ' ',
appSecret: ' '

3.  Modify the rest of the configuration such as Trackers, Sentiment Analytics as per requirements. [Optional]

realtimeSentimentAnalysis: true,
realtimeAnalyticsMetric: true,
trackers: [
  {
    name: "Promotion Mention",
    vocabulary: [
     "We have a special promotion going on if you book this before",
     "I can offer you a discount of 10 20 percent you being a new customer for us",
     "We have our month special this month",
     "We have a sale right now on"
    ]
  }
]

4. Start the application.

npm start

The application will be started at http://localhost:3000

Usage

1. Navigate to http://localhost:3000
2. Enter the LiveKit URL—utilize the LiveKit’s playground server URL: wss://demo.livekit.cloud.
3. Enter the Token generated using the LiveKit’s Server. If the LiveKit server is not running at the localhost, we can utilize the LiveKit’s playground server URL to generate token.

curl --location --request POST 'https://livekit.io/api/playground/tokens' \
--header 'authority: livekit.io' \
--header 'Content-Type: text/plain;charset=UTF-8' \
   	--data-raw '{
       "roomName": "tower",
       "userName": "Stark"
   	}'

4. Click on Connect to enter the Room.

5. Closed Captioning will appear on the screen as soon as a person starts speaking.

6. To view the Symbl’s Conversation Intelligence (Transcripts, Insights, Topics, Trackers), click on the down arrow that appears in the bottom right corner as soon as Symbl’s intelligence is generated.

Adding Multiple Participants in a Room

If you want to see how the application behaves with multiple participants, you can simply open localhost:3000 in multiple tabs in your browser and connect to the same room using different user names.

Additionally, if you would like to invite other participants to a room, each participant would need to have their own installation of this application and use the same room name and Account SID (the API Key and Secret can be different).

Screenshots

• Real-time Transcripts, Insights (Action Items, Questions and Follow Ups), Topics, Live Closed captioning
• Real-time Topics with sentiments
• Sentiment and Meeting Analytics, overall sentiment

Real Life Use Cases

There are many use cases and benefits of using a video conferencing app with conversation intelligence capabilities. You can extend your application and build wonderful projects based on the following things. 

Recruiting and HR Intelligence

• Empower hiring managers with empathy-driven insights that improve behaviors and emotional intelligence during interviews.
• Detect and measure candidates and interview empathy in real-time and post-interaction.
• Coach hiring managers by creating a playbook of the most insightful questions and interview tactics which creates a consistent and impartial interview process.
  

Remote Collaboration 

• Increase productivity for every meeting attendee with advanced conversation intelligence.
  • Aggregate meeting data. Get real-time transcription and topics for quicker and more seamless collaboration
  • Integrate and share anywhere, anytime. Create integrations with to-do’s directly with your favorite calendar, log follow-up items to your CRM, send meeting summaries to Slack or any other meeting application.

Revenue and Sales Intelligence

• Empower sales teams with AI-driven meeting notes, follow-ups, and action items.
  • Real-Time call performance – Understand key conversation drivers as they happen. Track script adherence, specific playbook utterances, and caller intent. Detect when specific risk factors are present like low talk-time, silence, negative sentiments, etc.
  • Sales Coaching – Analyze call behavior for your top performers and improve your entire team’s skills to increase deal performance. Uncover topic duration, question rate, and talk ratios to create personalized coaching recommendations for each sales rep in real-time and asynchronously.

Trend Analysis – Analyze thousands of sales calls and uncover hidden trends in your customer conversations.

Contributing

We welcome your contributions to make the Symbl LiveKit RTC App better! This guide is actively developed, and we love to hear from you.

Please feel free to create an issue or open a pull request with your questions, comments, suggestions, and feedback. If you liked our integration guide, please star our repo! 

Community

If you have any further questions, feel free to reach out to us through our Community Slack or submit PRs.

Connect with us 

• Slack: https://bit.ly/symbldotai-slack
• Twitter: @symbldotai 
• Github: https://github.com/symblai 

Feel free to reach out to me, Akanksha Bhasin, your Symbl.ai Community advocate, if you are interested in learning more about our community, becoming a Symbl.ai ambassador, or have general questions about Symbl.ai.

• Linkedin: https://www.linkedin.com/in/akankshabhasin

Next Steps

In this tutorial, you learned how to build a multi-party video-conferencing application using LiveKit and integrate conversation intelligence using Symbl.ai. If you have any questions, join Symbl.ai’s Community Slack or reach out to [email protected]. This entire code for this project is available in this GitHub repository. I want to hear if you found it helpful or have any feedback so say hi on Twitter or email!