This article demonstrates how to build and use the multi-party video-conferencing application using Twillio and Symbl.ai’s Real-time APIs.
Twilio is a developer-centric Customer Engagement Platform. Twilio Video allows you to embed customized video into your applications at scale. However, a video application may not be accessible to all audiences and can be arduous to build. Modern applications need built-in intelligence to be more inclusive of all audiences.
Symbl.ai provides out-of-the-box Conversational Intelligence capabilities to deeply analyze the spoken conversations that happen over video applications. One of these capabilities is Live Transcription, or Speech-to-Text. This allows you to convert the conversations from a verbal form to textual form and display it to users.
Combining these two platforms, we have built a multi-party video-conferencing application that demonstrates Symbl.ai’s Real-time APIs in the form of Closed Captions.
This application was inspired by Twilio’s video app and is built using Twilio-video.js, Create React App, and Symbl.ai’s Streaming API.
Features
Before we get into the details on how to build the Twilio video conference app, let’s talk about available features. This video app integrated with Symbl.ai’s Real-time APIs provides the following out-of-the-box conversational intelligence features:
- Live Closed Captioning: Live closed captioning is enabled by default and provides a real-time transcription of your audio content.
- Real-time Transcription: Symbl.ai offers state-of-the-art Speech-to-Text capability (also called transcription). You can get audio and video conversations transcriptions in real-time.
- Video conferencing with real-time video and audio: This allows for real-time use cases where both the video, audio (and its results from Symbl.ai’s back-end) need to be available in real-time. It can be integrated directly via the browser or server.
- Enable/Disable camera: After connecting your camera, you can enable or disable the camera when you want.
- Mute/unmute mic: After you connect to your device’s microphone you can mute or unmute when you want.
- Screen sharing: This can be used to capture the screen directly from the web app.
- Dominant Speaker indicator: The Dominant Speaker refers to the Participant having the highest audio activity at a given time.
- Network Quality Indicator: It highlights the call panel which displays the current network conditions of the user and if it is Strong, Weak, or Poor.
Supported Platforms
This application is currently supported only on Google Chrome.
How to Build the Video Conference App
This section describes how to run this Symbl.ai-powered Twilio Video Conference App. The code samples are written in cURL.
Prerequisites
Before running the Symbl-Twilio Conference App, ensure that the following prerequisites are met:
- JS ES6+: Make sure to install the latest version.
- Node.js v10+: Make sure to install the current latest version, minimum v10.
- NPM v6+: Node versions v6 or latest is required.
- Twilio account: You will require a Twilio account, you’ll also need your account SID, available on your Twilio dashboard, and an API key and secret, which you can generate in your console.
- Symbl Account: Sign up to the Symbl Platform and gather your Symbl credentials, i.e. your App Id and your App Secret.
Integration Steps
This section walks you through the steps necessary to install the Symbl for Twilio Video Conference App and run it on your local machine.
Step 1: Clone App and Install dependencies
On your local machine, clone the repo by running the following command on your terminal/console:
Then navigate to the folder where the code is:
cd symbl-twilio-video-react |
From the master folder, install dependencies using the following command:
npm install |
* Note: The full list of dependencies can be found in the package.json file.
Step 2: Store your Symbl credentials in App
This application offers two options for authorizing your Symbl account, in the application, or via the included token server. The local token server is managed by server.js.
The default behavior is for your Symbl account to authorize in-app. A dialog box will be shown automatically if you’re opening the app for the first time. In the config.js file, you will find enableInAppCredentials would have been set to true. For this option, you are not> required to update the .env file with Symbl credentials.
If you are planning to use the included token server for generating your Symbl token you may disable the in-app App ID/App Secret configuration. You can disable it by setting enableInAppCredentials to false in the config.js. In this case, store your Symbl credentials in the >.env file as follows:
SYMBL_APP_ID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SYMBL_APP_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Step 3: Store your Twilio credentials in the .env file:
Your Twilio account will be authorized via the token server, which is managed by server.js. In your Twilio console click on ‘Settings’ and take note of your Account SID. Navigate to Settings/API Keys to generate a new Key SID and Secret. Store it in the .env file as follows:
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_API_KEY_SID=SKxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_API_KEY_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
This completes the first time set-up of the application.
Step 4: Run the app locally
You can now run the app using:
npm start |
This will start the local token server and run the app in development mode.
Open http://localhost:3000 to see the application in the browser. The page will reload if you make changes to the source code in src. You will also see any linting errors in the console.
When you run the app, you will be prompted to enter your name and a room name. Once you join a room, it should look like this:
You can change the audio/video device settings as follows:
(Optional) Step 5: Run only the local token server
By default, when you run the npm start script, the token server is also started. However, if you want to run only the token server locally, you can do so with:
npm run server |
The token server runs on port 8081 and exposes two GET endpoints: one to generate an access token for Symbl, and one for generating an access token for Twilio.
Symbl token endpoint expects to GET request at /symbl-token route with no parameters. You can test it with the following curl command:
curl 'localhost:8081/symbl-token'The response will be a JSON response with accessToken and expiresIn values with Symbl access token and expiry of the token.
Twilio token endpoint expects to >GET> request at /twilio-token route with the following query parameters:
identity: string, // the user’s identity
roomName: string // the room name
The response will be a token that can be used to connect to a room. Test it out with this sample >curl> command:
curl 'localhost:8081/twilio-token?identity=TestName&roomName=TestRoom'Adding Multiple Participants in a Room
If you want to see how the application behaves with multiple participants, you can simply open localhost:3000 in multiple tabs in your browser and connect to the same room name using different user names.
Each participant must have a unique Twilio token. If the app is deployed using the same set of Twilio credentials for all participants, then each participant needs a unique identity string. Participants may also have their own local installations of the app, but they require the same Twilio Account SID (the Twilio API Key and Secret can be different).
API Reference
Find comprehensive information about our REST APIs in the API Reference section.
Next Steps
To learn more about different Integrations offered at Symbl.ai, go to our Integrations Directory.
Community
This guide is actively developed, and we love to hear from you! If you liked our integration guide, please star our repo!
Please feel free to create an issue or open a pull request with your questions, comments, suggestions, and feedback, or reach out to us at [email protected],through our Community Slack or our forum.
