Welcome!

Eclipse Authors: Liz McMillan, Elizabeth White, XebiaLabs Blog, Ken Fogel, Sematext Blog

Blog Feed Post

Extreme HTML5 Video Interactivity: Sending WebSocket Messages with Popcorn.js

One of our most popular demos at Kaazing is using a Web browser on a smartphone to control a physical toy truck from continents away. The truck has a Raspberry Pi attached to it, connecting to a WebSocket server, and listening to control command messages: drive forward or backward, turn left or right, or turn headlight on and off. You can learn more about the project in our Remote Controlling a Car over the Web. Ingredients: Smartphone, WebSocket, and Raspberry Pi blog post.

The most interesting way to demonstrate the truck is by having a remote person control the truck, and join in over a video conference. Here’s the recording of us doing just this. Fast forward to 4:08 for the truck demo.

Now, there are certain circumstances, when running Skype, or other live video chat apps is not an option. You may be off the grid, or simply not have anybody handy controlling your truck remotely.

To address this challenge, we wanted to create a self-contained environment where the same dialog and experience can be presented, but without all the above mentioned dependencies. To achieve this, we decided to record a video of someone operating the remote control that the presenter could use as the “Skyped-in” portion of the presentation. There are a number of ways we could hack this, for example by pretending to control the car by emulating the controls in the room. How cool would it be to instead have the recorded video actually trigger the remote controls using WebSocket messages? Instead of a real person controlling the car in real time, we could have the video control the car in real time.

First, we recorded the video. In the video, David Witherspoon operates the remote control. (Aside: David is a software engineer at Kaazing who, along with colleague Prashant Khanal, was instrumental in dreaming up and building the truck). David followed the script of the dialog very precisely. Knowing the script was not sufficient, we had to do it with exact timing, as it was specified to be run during the actual demo.

After processing the video, I embedded it in a web page, and overlaid the video with a live video feed of the presenter’s laptop camera. This is an important step to make the experience more realistic; after all, every video conference does this.

Here’s the HTML code:

	<head>
		<script src="http://popcornjs.org/code/dist/popcorn-complete.min.js"></script>
		<script src="http://demo.kaazing.com/lib/client/javascript/StompJms.js"></script>
		<script>document.addEventListener( "DOMContentLoaded", function() {doConnect();}, false );</script>
		<link rel="stylesheet" href="css/truck.css"</style>
	</head>
	<body>
		<video id="selfVideo" autoplay width="256"></video>
		<video id="truckVideo" width="1024">
			<!-- <source src="videos/PeterTruck.mp4" type="video/mp4"> -->
			<source src="http://localhost/videos/DavidTruck.mp4" type="video/mp4">
		</video>
	</body>
	<script src="js/truck.js"></script>	

And here is the JavaScript code:

var errorCallback = function(e) {
console.log('Reeeejected!', e);
};

// Not showing vendor prefixes.
navigator.getUserMedia  = navigator.getUserMedia ||
              navigator.webkitGetUserMedia ||
              navigator.mozGetUserMedia ||
              navigator.msGetUserMedia;

var video = document.getElementById ("selfVideo");

if (navigator.getUserMedia) {
  navigator.getUserMedia({audio: false, video: true}, function(stream) {
    video.src = window.URL.createObjectURL(stream);
  }, errorCallback);
} else {
  video.src = 'somevideo.webm'; // fallback.
}

The simple CSS snippet ensures that the presenter’s “self” video overlays the remote person’s (recorded) video.

#selfVideo
	{position:fixed;
	top:30px;
	left:30px;}

Also, browsers are required to prompt for permission before Web apps can start using the built-in camera. First, you have to select Allow, for the “little video” in the top left corner to appear. Here’s what the permission request bar looks like in Chrome:

usecamera

And here’s the end result:

popcornvideo

Whenever my actor friend in the main video uses his remote control, we must trigger a corresponding WebSocket message. The messages are sent by the Web app hosting the video at the exact time when the control is touched in the video. I used popcorn.js, an open source media library, to get the timing right:

Popcorn.js is an HTML5 media framework written in JavaScript for filmmakers, web developers, and anyone who wants to create time-based interactive media on the web. Popcorn.js is part of Mozilla’s Popcorn project.

I created an array with the timing and the messages that needed to be sent. The timing is measured in seconds.

var davidTruckMsgs = [
[33,"frontlight;on"],
[35,"frontlight;off"],
[36,"frontlight;on"],
[38,"frontlight;off"],
[42,"steering;right : thrust;off"],
[43,"steering;left : thrust;off"],
[44,"steering;right: thrust;off"],
[45,"steering;left : thrust;off"],
[48,"steering;off : thrust;forward"],
[50,"steering;off : thrust;backward"],
[51,"steering;off : thrust;forward"],
[52,"steering;off : thrust;backward"],
[86,"steering;left : thrust;forward"]
];

Then, we have to schedule the WebSocket messages, as defined in the array specified above. Note: The above array is called davidTruckMsgs, and down below we iterate over the truckMsgs array. As you can see in the completed source code, I have multiple arrays for various videos/actors. Whichever is the one used at the moment is referenced as truckMsgs later on.

for (var truckMsg in truckMsgs) {
  var obj = truckMsgs[truckMsg];
  pop.cue( obj[0], makeCallback( obj ) );
}
pop.play();

The makeCallback function invokes the actual logic sending the WebSocket message. If you’re wondering why this is all needed in the first place, check out this question on Stack Overflow.

function makeCallback(obj) {
    return function() {
    	doSend(session.createTextMessage(obj[1]));
    };
}

For usability, I added pause/continue functionality whenever the main video is clicked. This gives the presenter more control, allowing him/her to preload the page with the main video paused on it.

vid = document.getElementById ("truckVideo");
vid.addEventListener ('click', function() {
     vid.paused ? vid.play() : vid.pause();
});

For the WebSocket communication we used the JMS edition of the Kaazing WebSocket Gateway, allowing us to leverage simple pub/sub messaging concepts. With the help of popcorn.js, from the HTML5 web app we publish WebSocket messages to a so called topic, and whoever is interested in (read: subscribed to) it will receive it.

This way the video is driving the truck, simply by having the WebSocket messages sent out properly timed to the pre-recorded video.

Here’s the end result. Isn’t it awesome?

You can see the entire source code on github.


Read the original blog entry...

More Stories By Kaazing Blog

Kaazing is helping define the future of the event-driven enterprise by accelerating the Web for the Internet of Things.

@ThingsExpo Stories
WebRTC is bringing significant change to the communications landscape that will bridge the worlds of web and telephony, making the Internet the new standard for communications. Cloud9 took the road less traveled and used WebRTC to create a downloadable enterprise-grade communications platform that is changing the communication dynamic in the financial sector. In his session at @ThingsExpo, Leo Papadopoulos, CTO of Cloud9, discussed the importance of WebRTC and how it enables companies to focus...
"My role is working with customers, helping them go through this digital transformation. I spend a lot of time talking to banks, big industries, manufacturers working through how they are integrating and transforming their IT platforms and moving them forward," explained William Morrish, General Manager Product Sales at Interoute, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
SYS-CON Events announced today that 910Telecom will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Housed in the classic Denver Gas & Electric Building, 910 15th St., 910Telecom is a carrier-neutral telecom hotel located in the heart of Denver. Adjacent to CenturyLink, AT&T, and Denver Main, 910Telecom offers connectivity to all major carriers, Internet service providers, Internet backbones and ...
SYS-CON Events announced today that LeaseWeb USA, a cloud Infrastructure-as-a-Service (IaaS) provider, will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. LeaseWeb is one of the world's largest hosting brands. The company helps customers define, develop and deploy IT infrastructure tailored to their exact business needs, by combining various kinds cloud solutions.
For basic one-to-one voice or video calling solutions, WebRTC has proven to be a very powerful technology. Although WebRTC’s core functionality is to provide secure, real-time p2p media streaming, leveraging native platform features and server-side components brings up new communication capabilities for web and native mobile applications, allowing for advanced multi-user use cases such as video broadcasting, conferencing, and media recording.
SYS-CON Events announced today that Venafi, the Immune System for the Internet™ and the leading provider of Next Generation Trust Protection, will exhibit at @DevOpsSummit at 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Venafi is the Immune System for the Internet™ that protects the foundation of all cybersecurity – cryptographic keys and digital certificates – so they can’t be misused by bad guys in attacks...
ReadyTalk has expanded the capabilities of the FoxDen collaboration platform announced late last year to include FoxDen Connect, an in-room video collaboration experience that launches with a single touch. With FoxDen Connect, users can now not only engage in HD video conferencing between iOS and Android mobile devices or Chrome browsers, but also set up in-person meeting rooms for video interactions. A host’s mobile device automatically recognizes the presence of a meeting room via beacon tech...
The cloud market growth today is largely in public clouds. While there is a lot of spend in IT departments in virtualization, these aren’t yet translating into a true “cloud” experience within the enterprise. What is stopping the growth of the “private cloud” market? In his general session at 18th Cloud Expo, Nara Rajagopalan, CEO of Accelerite, explored the challenges in deploying, managing, and getting adoption for a private cloud within an enterprise. What are the key differences between wh...
It’s 2016: buildings are smart, connected and the IoT is fundamentally altering how control and operating systems work and speak to each other. Platforms across the enterprise are networked via inexpensive sensors to collect massive amounts of data for analytics, information management, and insights that can be used to continuously improve operations. In his session at @ThingsExpo, Brian Chemel, Co-Founder and CTO of Digital Lumens, will explore: The benefits sensor-networked systems bring to ...
On Dice.com, the number of job postings asking for skill in Amazon Web Services increased 76 percent between June 2015 and June 2016. Salesforce.com saw its own skill mentions increase 37 percent, while DevOps and Cloud rose 35 percent and 28 percent, respectively. Even as they expand their presence in the cloud, companies are also looking for tech professionals who can manage projects, crunch data, and figure out how to make systems run more autonomously. Mentions of ‘data science’ as a skill ...
Manufacturers are embracing the Industrial Internet the same way consumers are leveraging Fitbits – to improve overall health and wellness. Both can provide consistent measurement, visibility, and suggest performance improvements customized to help reach goals. Fitbit users can view real-time data and make adjustments to increase their activity. In his session at @ThingsExpo, Mark Bernardo Professional Services Leader, Americas, at GE Digital, discussed how leveraging the Industrial Internet a...
Amazon has gradually rolled out parts of its IoT offerings in the last year, but these are just the tip of the iceberg. In addition to optimizing their back-end AWS offerings, Amazon is laying the ground work to be a major force in IoT – especially in the connected home and office. Amazon is extending its reach by building on its dominant Cloud IoT platform, its Dash Button strategy, recently announced Replenishment Services, the Echo/Alexa voice recognition control platform, the 6-7 strategic...
In addition to all the benefits, IoT is also bringing new kind of customer experience challenges - cars that unlock themselves, thermostats turning houses into saunas and baby video monitors broadcasting over the internet. This list can only increase because while IoT services should be intuitive and simple to use, the delivery ecosystem is a myriad of potential problems as IoT explodes complexity. So finding a performance issue is like finding the proverbial needle in the haystack.
There will be new vendors providing applications, middleware, and connected devices to support the thriving IoT ecosystem. This essentially means that electronic device manufacturers will also be in the software business. Many will be new to building embedded software or robust software. This creates an increased importance on software quality, particularly within the Industrial Internet of Things where business-critical applications are becoming dependent on products controlled by software. Qua...
Big Data, cloud, analytics, contextual information, wearable tech, sensors, mobility, and WebRTC: together, these advances have created a perfect storm of technologies that are disrupting and transforming classic communications models and ecosystems. In his session at @ThingsExpo, Erik Perotti, Senior Manager of New Ventures on Plantronics’ Innovation team, provided an overview of this technological shift, including associated business and consumer communications impacts, and opportunities it ...
The 19th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportuni...
IoT is rapidly changing the way enterprises are using data to improve business decision-making. In order to derive business value, organizations must unlock insights from the data gathered and then act on these. In their session at @ThingsExpo, Eric Hoffman, Vice President at EastBanc Technologies, and Peter Shashkin, Head of Development Department at EastBanc Technologies, discussed how one organization leveraged IoT, cloud technology and data analysis to improve customer experiences and effi...
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, demonstrated how to move beyond today's coding paradigm and shared the must-have mindsets for removing complexity from the develo...
SYS-CON Events announced today that MangoApps will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. MangoApps provides modern company intranets and team collaboration software, allowing workers to stay connected and productive from anywhere in the world and from any device.
The IETF draft standard for M2M certificates is a security solution specifically designed for the demanding needs of IoT/M2M applications. In his session at @ThingsExpo, Brian Romansky, VP of Strategic Technology at TrustPoint Innovation, explained how M2M certificates can efficiently enable confidentiality, integrity, and authenticity on highly constrained devices.