Every day millions of users around the world communicate with each other through email, social networks, and other online communities including chat platforms and message boards. Many times, users find a community interesting and want to participate in the chat conversation, but the primary language of the online community might be different than the language that user understands.
One of those communities is Twitch, the world’s leading social video service for gamers. Each day, millions of community members gather to watch, talk, and chat about shared interests. Chat is built into every stream, so instead of passively watching the stream, you can be part of the show. You will find many streamers on Twitch who don’t necessarily speak your language, but you might still enjoy their streams and would want to participate in the chat.
To make this communication possible, AWS provides Amazon Translate, a neural machine translation service that delivers fast, high-quality, and affordable language translation. Amazon Translate can provide on-demand translation to enable cross-lingual communication between users. By adding real-time translation to chat, email, helpdesk, and ticketing applications, an English-speaking agent or employee can communicate with customers across multiple languages.
This blog post covers how you can use Amazon Translate to enable cross-lingual chat for a Twitch channel. You will also learn how to use Amazon Polly, a Text-to-Speech service, to help voice incoming real-time messages. This additional functionality provides further access and offer users the option to listen to the messages while concentrating on other tasks at hand.
Solution overview
In this post, you will create an HTML page with CSS and JavaScript. You will use a JavaScript library to connect to a Twitch channel and start receiving messages. As those real-time messages come in, you will use the AWS SDK to call Amazon Translate and get those messages translated and displayed in the UI. If you choose to use the user-enabled voice option, you’ll use AWS SDK to call Amazon Polly and get synthesized speech for those messages and play them back to the users.
Create and Configure a Twitch account
- Go to twitch.tv to sign up for a Twitch account.
- After successfully signing up for your Twitch account, go to https://twitchapps.com/tmi to create a Twitch OAuth token.
- Click on the button “Connect with Twitch” and note the OAuth token.
Create an IAM user
AWS Identity and Access Management (IAM) enables you to manage access to AWS services and resources securely. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources. To get started, create an IAM user with the minimum required permissions to run this example.
- Go to https://console.aws.amazon.com/iam/home?region=us-east-1#/users.
- Choose Add user.
- Enter the user name, check Programmatic access, and choose Next Permissions.
- Choose Attach existing policies directly, and search for translate.
- Select the check box next to TranslateReadOnly and choose Next Review.
- Choose Create user and note the access key ID and secret access key.
Create and run the application
- Create an HTML page and copy the code from the following code block.
<!doctype html>
<html lang="en">
<head>
<title>Amazon Translate</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<!-- Latest compiled and minified CSS for Bootstrap -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
<!-- Custom CSS -->
<style>
.topHeader
{
background-color: #6441a4;
padding: 10px;
border-bottom: solid 1px #cacaca;
color: white
}
.panelHeading
{
background-color: #6441a4 !important;
}
.panelBody
{
min-height: 450px; max-height: 450px;overflow-y: scroll;
}
body{
margin-left: 0px;
margin-right: 0px;
height: 100%;
}
</style>
</head>
<body>
<div class="container-fluid">
<!--Top Header-->
<div class="row topHeader">
<div class="col-md-12">
<h4>Amazon Translate - Artificial Intelligence on AWS - Powerful machine learning for all Developers and Data Scientists</h4>
</div>
</div>
<!--Status Label-->
<div class="row">
<div class="col-md-12">
<p class="bg-info">
<div id="connecting-div"></div>
</p>
</div>
</div>
<div class="row" style="padding: 10px;">
<div class="col-md-6">
<div class="form-inline">
<div class="form-group">
<input type="text" id="channel" class="form-control" value="" placeholder="Channel"/>
</div>
<div class="form-group">
<select id="sourceLanguage" class="form-control">
<option value="en">en</option>
<option value="ar">ar</option>
<option value="de" selected="selected">de</option>
<option value="es">es</option>
<option value="fr">fr</option>
<option value="pt">pt</option>
<option value="zh">zh</option>
</select>
</div>
<div class="form-group">
<select id="targetLanguage" class="form-control">
<option value="en" selected="selected">en</option>
<option value="ar">ar</option>
<option value="de">de</option>
<option value="es">es</option>
<option value="fr">fr</option>
<option value="pt">pt</option>
<option value="zh">zh</option>
</select>
</div>
<div class="form-group">
<button type="button" class="form-control" id="btn-go" onclick="connect()">Go</button>
<button type="button" class="form-control" id="btn-stop" onclick="location.href='index.html';">Stop</button>
<span id="status"></span>
</div>
</div>
</div>
<div class="col-md-6">
<div class="form-inline">
<div class="form-group">
<input type="checkbox" id="cbSpeak" value="Speak"> Speak Live Translation
<input type="text" id="follow" class="form-control" value="" placeholder="follow"/>
</div>
</div>
</div>
</div>
<!--Chat Boxes-->
<div class="row">
<!--Live Chat-->
<div class="col-md-6">
<div class="panel panel-primary">
<div class="panel-heading panelHeading">Live Chat</div>
<div id="livechatc" class="panel-body panelBody">
<div class="subscribe" id="livechat"></div>
</div>
</div>
</div>
<!--Live Chat-->
<!--Translated Chat-->
<div class="col-md-6">
<div class="panel panel-primary">
<div class="panel-heading panelHeading">Live Translation</div>
<div id="livetranslationc" class="panel-body panelBody">
<div class="imageDetected" id="livetranslation"></div>
</div>
</div>
</div>
<!--Translated Chat-->
</div>
<!--Send Message-->
<div class="row">
<div class="col-md-11">
<input type="text" id="message" class="form-control"/>
</div>
<div class=" col-md-1">
<button type="button" class="form-control btn btn-default" id="btn-send" onclick="sendMessage()">Send</button>
</div>
</div>
</div>
<!-- Latest compiled and minified JavaScript -->
<!-- jQuery first, then Bootstrap JS -->
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
<script src="aws-js-sdk/dist/aws-sdk-all.js"></script>
<script src="http://cdn.tmijs.org/js/1.2.1/tmi.min.js" integrity="sha384-eE0n7sm1W7DOUI2Xh5I4qSpZTe6hupAO0ovLfqEy0yVJtGRBNfssdmjbJhEYm6Bw" crossorigin="anonymous"></script>
<script>
cred = {
twitchUsername: "Twitch user name",
twitchOAuthToken: "Twitch OAuth token",
awsAccessKeyId: "access key",
awsSecretAccessKey: "secret key"
};
AWS.config.region = 'region';
ep = new AWS.Endpoint('endpoint');
AWS.config.credentials = new AWS.Credentials(cred.awsAccessKeyId, cred.awsSecretAccessKey);
window.translator = new AWS.Translate({endpoint: ep, region: AWS.config.region});
/**************************Init and Connect to Chat****************************/
function connect(){
init();
//Twitch Client
var options = {
options: {
debug: false
},
connection: {
cluster: "aws",
reconnect: true
},
identity: {
username: cred.twitchUsername,
password: cred.twitchOAuthToken
},
channels: [con.channel]
};
window.client = tmi.client(options);
window.client.connect();
//Attached Handlers
window.client.on("chat", onChat);
window.client.on("connecting", onConnecting);
window.client.on("connected", onConnected);
//Disable UI Elements
document.getElementById("sourceLanguage").disabled = true;
document.getElementById("targetLanguage").disabled = true;
document.getElementById("channel").disabled = true;
document.getElementById("btn-go").disabled = true;
}
function init(){
//Get UI Controls
var lc = document.getElementById("livechat");
var lt = document.getElementById("livetranslation")
var lcc = document.getElementById("livechatc");
var ltc = document.getElementById("livetranslationc")
var cbspeak = document.getElementById("cbSpeak")
var follow = document.getElementById("follow");
var sendMessage = document.getElementById("message");
//Cache values
con = {
channel: document.getElementById("channel").value,
sourceLanguage: document.getElementById("sourceLanguage").value,
targetLanguage: document.getElementById("targetLanguage").value,
liveChatUI: lc,
liveTranslationUI: lt,
liveChatUIContainer: lcc,
liveTranslationUIContainer: ltc,
cbSpeak: cbspeak,
follow: follow,
sendMessage: sendMessage
}
lc.innerHTML = '';
lt.innerHTML = '';
//Speaker
var voiceId = "Joanna";
if(con.targetLanguage == "en")
voiceId = "Joanna";
else if(con.targetLanguage == "de")
voiceId = "Marlene";
else if(con.targetLanguage == "es")
voiceId = "Conchita";
else if(con.targetLanguage == "fr")
voiceId = "Celine";
else if(con.targetLanguage == "pt")
voiceId = "Ines";
else
voiceId = "Joanna";
window.audioPlayer = AudioPlayer(voiceId);
}
/**************************Init and Connect to Chat****************************/
/**************************Receive and Translate Chat****************************/
function onChat (channel, userstate, message, self) {
// Don't listen to my own messages..
if (self) return;
//Translate
if (message) {
var username = userstate['username'];
var params = {
Text: message,
SourceLanguageCode: con.sourceLanguage,
TargetLanguageCode: con.targetLanguage
};
window.translator.translateText(params, function onIncomingMessageTranslate(err, data) {
if (err) {
console.log("Error calling Translate. " + err.message + err.stack);
}
if (data) {
console.log("M: " + message);
console.log("T: " + data.TranslatedText);
//Print original message in chat UI
con.liveChatUI.innerHTML += '<strong>' + username + '</strong>: ' + message + '<br>';
//Print translation in translation UI
con.liveTranslationUI.innerHTML += '<strong>' + username + '</strong>: ' + data.TranslatedText + '<br>';
//If speak translation in enabled, speak translated message
if(con.cbSpeak.checked){
if(con.follow.value == "" || username == con.follow.value)
audioPlayer.Speak(username + " says " + data.TranslatedText);
}
//Scroll chat and translated UI to bottom to keep focus on latest messages
con.liveChatUIContainer.scrollTop = con.liveChatUIContainer.scrollHeight;
con.liveTranslationUIContainer.scrollTop = con.liveTranslationUIContainer.scrollHeight;
}
});
}
}
/**************************Receive and Translate Chat****************************/
/**************************Client Connecting****************************/
function onConnecting (address, port) {
document.getElementById("status").innerHTML = " [ Connecting...]"
}
function onConnected (address, port) {
document.getElementById("status").innerHTML = " [ Connected ]"
window.audioPlayer.Speak("Connected to channel " + con.channel + ". You should now be getting live chat messages.");
}
/**************************Client Connecting****************************/
/**************************Send Message****************************/
function sendMessage(){
if(con.sendMessage.value){
message = con.sendMessage.value;
var params = {
Text: con.sendMessage.value,
SourceLanguageCode: con.targetLanguage,
TargetLanguageCode: con.sourceLanguage
};
window.translator.translateText(params, function onSendMessageTranslate(err, data) {
if (err) {
console.log("Error calling Translate. " + err.message + err.stack);
}
if (data) {
console.log("M: " + message);
console.log("T: " + data.TranslatedText);
//Send message to chat
window.client.action(con.channel, data.TranslatedText);
//Clear send message UI
con.sendMessage.value = "";
//Print original message in Translated UI
con.liveTranslationUI.innerHTML += '<strong> ME: </strong>: ' + message + '<br>';
//Print translated message in Chat UI
con.liveChatUI.innerHTML += '<strong> ME: </strong>: ' + data.TranslatedText + '<br>';
//Scroll chat and translated UI to bottom to keep focus on latest messages
con.liveChatUIContainer.scrollTop = con.liveChatUIContainer.scrollHeight;
con.liveTranslationUIContainer.scrollTop = con.liveTranslationUIContainer.scrollHeight;
}
});
}
}
/**************************Send Message****************************/
/**************************Audio player****************************/
function AudioPlayer(voiceId) {
var audioPlayer = document.createElement('audio');
audioPlayer.setAttribute("id", "audioPlayer");
document.body.appendChild(audioPlayer);
var isSpeaking = false;
var speaker = {
self: this,
playlist:[],
Speak: function (text) {
//If currently speaking a message, add new message to the playlist
if (isSpeaking) {
this.playlist.push(text);
} else {
speakTextMessage(text).then(speakNextTextMessage)
}
}
}
// Speak text message
function speakTextMessage(text) {
return new Promise(function (resolve, reject) {
isSpeaking = true;
getAudioStream(text).then(playAudioStream).then(resolve);
});
}
// Speak next message in the list
function speakNextTextMessage() {
var pl = speaker.playlist;
if (pl.length > 0) {
var txt = pl[0];
pl.splice(0, 1);
speakTextMessage(txt).then(speakNextTextMessage);
}
}
// Get synthesized speech from Amazon polly
function getAudioStream(textMessage) {
return new Promise(function (resolve, reject) {
var polly = new AWS.Polly();
var params = {
OutputFormat: 'mp3',
Text: textMessage,
VoiceId: voiceId
}
polly.synthesizeSpeech(params, function (err, data) {
if (err)
reject(err);
else
resolve(data.AudioStream);
});
});
}
// Play audio stream
function playAudioStream(audioStream) {
return new Promise(function (resolve, reject) {
var uInt8Array = new Uint8Array(audioStream);
var arrayBuffer = uInt8Array.buffer;
var blob = new Blob([arrayBuffer]);
var url = URL.createObjectURL(blob);
audioPlayer.src = url;
audioPlayer.addEventListener("ended", function () {
isSpeaking = false;
resolve();
});
audioPlayer.play();
});
}
return speaker;
}
/**************************Audio player****************************/
</script>
</body>
</html>
- Update the following code snippet with path to the AWS JavaScript SDK.
<script src="aws-js-sdk/dist/aws-sdk-all.js"></script>
- Update the following code snippet with the AWS region and endpoint.
AWS.config.region = 'region';
ep = new AWS.Endpoint('endpoint');
- Update the following code snippet with your Twitch username, OAuth token, AWS access key ID and AWS secret access key.
cred = {
twitchUsername: "Twitch user name",
twitchOAuthToken: "Twitch OAuth token",
awsAccessKeyId: "access key",
awsSecretAccessKey: "secret key"
};
- Open the web page in a browser.
- Enter the name of a Twitch channel.
- Select source and destination language and click Go.
You should now see real-time messages along with their translation on the screen. If you compare Twitch chat with this example application, you will see messages appear in the chat simultaneously as they appear in the Twitch chat showing the performance and very low latency from Amazon Translate.
You can also view the video demonstration showing the real-time translation of Twitch chat at re:Invent 2017.
Receive and Translate Messages
The following code block shows how you receive real-time incoming messages and then use AWS SDK to call Amazon Translate to show the translated message in the UI. As you see most of the code that follows is about input and UI handling, while the code to translate the messages is only few lines.
/**************************Receive and Translate Chat****************************/
function onChat (channel, userstate, message, self) {
// Don't listen to my own messages..
if (self) return;
//Translate
if (message) {
var username = userstate['username'];
var params = {
Text: message,
SourceLanguageCode: con.sourceLanguage,
TargetLanguageCode: con.targetLanguage
};
window.translator.translateText(params, function onIncomingMessageTranslate(err, data) {
if (err) {
console.log("Error calling Translate. " + err.message + err.stack);
}
if (data) {
console.log("M: " + message);
console.log("T: " + data.TranslatedText);
//Print original message in chat UI
con.liveChatUI.innerHTML += '<strong>' + username + '</strong>: ' + message + '<br>';
//Print translation in translation UI
con.liveTranslationUI.innerHTML += '<strong>' + username + '</strong>: ' + data.TranslatedText + '<br>';
//If speak translation in enabled, speak translated message
if(con.cbSpeak.checked){
if(con.follow.value == "" || username == con.follow.value)
audioPlayer.Speak(username + " says " + data.TranslatedText);
}
//Scroll chat and translated UI to bottom to keep focus on latest messages
con.liveChatUIContainer.scrollTop = con.liveChatUIContainer.scrollHeight;
con.liveTranslationUIContainer.scrollTop = con.liveTranslationUIContainer.scrollHeight;
}
});
}
}
/**************************Receive and Translate Chat****************************/
Send translated messages
The following code block shows how you translate an outgoing message and then send it the Twitch channel.
/**************************Send Message****************************/
function sendMessage(){
if(con.sendMessage.value){
message = con.sendMessage.value;
var params = {
Text: con.sendMessage.value,
SourceLanguageCode: con.targetLanguage,
TargetLanguageCode: con.sourceLanguage
};
window.translator.translateText(params, function onSendMessageTranslate(err, data) {
if (err) {
console.log("Error calling Translate. " + err.message + err.stack);
}
if (data) {
console.log("M: " + message);
console.log("T: " + data.TranslatedText);
//Send message to chat
window.client.action(con.channel, data.TranslatedText);
//Clear send message UI
con.sendMessage.value = "";
//Print original message in Translated UI
con.liveTranslationUI.innerHTML += '<strong> ME: </strong>: ' + message + '<br>';
//Print translated message in Chat UI
con.liveChatUI.innerHTML += '<strong> ME: </strong>: ' + data.TranslatedText + '<br>';
//Scroll chat and translated UI to bottom to keep focus on latest messages
con.liveChatUIContainer.scrollTop = con.liveChatUIContainer.scrollHeight;
con.liveTranslationUIContainer.scrollTop = con.liveTranslationUIContainer.scrollHeight;
}
});
}
}
/**************************Send Message****************************/
Summary
In this post you learned how you can use Amazon Translate for real-time translation of chat messages. This blog used a Twitch channel as an example, but you can use it as a starting point for other real-time streaming text like other chat platforms, customer service interactions, message boards and more.
About the Authors
Kashif Imran is a Solutions Architect at Amazon Web Services. He works with some of the largest strategic AWS customers to provide technical guidance and design advice. His expertise spans application architecture, serverless, containers, NoSQL and machine learning.