Conversational Intelligence for Streaming Applications: Twilio Media Streams

Capturing audio and deriving real-time insights seems complicated, but it’s not as hard as you think. Twilio Media Streams provide real-time raw audio and gives developers the flexibility to integrate this audio in the voice stack of your choice. Couple that with the power of Symbl, and you can do even more to surface actionable insights with customer interactions through the Symbl WebSocket API.

What you get: You’re receive a post-conversation email with email with topics generated, number of action items, and a link to go view the full summary output

This blog post will guide you step-by-step through integrating the Symbl WebSocket API into Twilio Media Streams.

Requirements

Before we can get started, you’ll need to make sure to have:

  • A Symbl Account. Sign up here.
  • A Twilio Account

Setting up the Local Server

Twilio Media Streams use the WebSocket API to live stream the audio from the phone call to your application. Let’s get started by setting up a server that can handle WebSocket connections.

Open your terminal and create a new project folder and create an index.js file.

To handle HTTP requests we will use node’s built-in http module and Express. For WebSocket connections we will be using ws, a lightweight WebSocket client for node.

In the terminal run these commands to install ws , websocket and Express :

Open your index.js file and add the following code to set up your server.

Save and run index.js with

Open your browser and navigate to

Your browser should show Hello World

Setting up Symbl WebSocket API

Let’s set up our Twilio number to connect to our WebSocket server.

First, we need to modify our server to handle the WebSocket messages that will be sent from Twilio when our phone call starts streaming. There are four main message events we want to listen for: connected , start , media and stop .

Modify your index.js file to log messages when each of these messages arrive at our server.

Now we need to set up a Twilio number to start streaming audio to our server. We can control what happens when we call our Twilio number using TwiML. We’ll create a HTTP route that will return TwiML instructing Twilio to stream audio from the call to our server.

Add the following POST route to your index.js file.

For Twilio to connect to your local server we need to expose the port to the internet. We need to use ngrok to create a tunnel to our localhost port and expose it to the internet. In a new terminal window run the following command:

You should get an output with a forwarding address like this. Copy the URL onto the clipboard. Make sure you record the HTTPS URL.

Head over to a new terminal window and run your index.js file.

Setting up your Twilio Studio

Now that our WebSocket server is ready, the remaining configuration needed to have Symbl join your customer and agent conversations, will be done through your Twilio Studio Dashboard.

Navigate to Studio and create a new flow.

Twilio offers three different triggers that you can use to build out this integration. Depending on your use case, you can choose to begin the flow from either the message, call, or REST API triggers.

In our example, we want Symbl to join a voice conversation when a customer calls our agent, so we will be using the incoming call trigger to build out our flow.

First, use the Fork Stream widget and connect it to the Incoming Call trigger. In the configuration, the URL should match your ngrok domain.

NOTE: Use the WebSocket protocol wss instead of http for the ngrok url.
startws

Next connect this widget to the `Flex Agent` widget which will connect the call to the Flex Agent:
flexagent

Finally, we need to end the stream once the call is completed. To do so, use the same `Fork Stream` widget but the configuration for `stream action` should be `Stop`.
flexagent

Test out the integration

To test out the integration, navigate to the Flex tab and click on Launch Flex:

On your Flex dashboard, locate your Twilio phone number and call that number from your cellular device.

When the agent accepts the call, the audio will be streamed through the WebSocket API and at the end of the call, you will get an email with the transcript and insights generated from the conversation.

Wrapping up

What can you do with the data next? Fetch the data out of the conversation and With this output, you can push the data to several downstream channels Trello, Slack, Jira, and others platforms.

Congratulations! You can now harness the power of Symbl and Media Streams to extend your application capabilities.

Sign up to start building!

Need additional help? You can refer to our API Docs for more information and view our sample projects on Github.

Helping you transform conversations into real time actionable outcomes for your business.

© 2020 Symbl.ai All rights reserved.