Exploring Esplora – III Pushing Data

The Esplora hardware and software library have made things quite easy in terms of generating a stream of sensor data.  I discussed creating the data stream in the prior post in this series (http://blog.volksdata.net/2013/01/08/exploring-esplora-ii-creating-a-data-stream/6 ).  This stream could be used by an application running on the local host computer, but I am more interested in moving the data off to the Internet and then onwards to other devices and applications which need to receive the data.  Perhaps the remote device is another computer running a multiplayer game in a web browser, or maybe it is a controller to position a telescope, or a system to trigger some action such as the taking of a picture, or the firing of retro-rockets.  The possibilities are bounded merely by imagination.

Traditionally, the data stream could be moved via an HTTP request to a central server and then interested client devices could poll the central server, via HTTP, at regular intervals to see if any new data has arrived.  For some applications, this approach is entirely adequate, especially if the system receiving the data does not need to respond in a particularly timely manne.  But in the case of a real-time game or triggering a camera (or firing a retro-rocket!), it would be a requirement that the response to data in the stream be as rapid as possible.  Polling could be tweaked to be reasonably fast, but the overhead can be enormous.  Imagine the scenario of a remote camera which can have its position controlled by a remote user.  Perhaps the user wishes to take photos 6 times during the day, but a random times.  To handle this, the camera system would have to poll continuously all day long to know when to trigger a photo – and if it’s polling once per second there would be 86,400 polling requests every day – a ratio of 14,400 polling requests for each actual event.  Not necessarily overwhelming for the server if there is just that one device, but what if there are 1,000 connected cameras and users?  Certainly a better approach is for the remote device to simply wait for instructions to arrive without constantly having to ask “any instructions yet?”.  Instead of asking 86,400 times each day, the remote system could just receive the handful of commands that are being sent.  The problem of course becomes, “how are such notifications sent?” and “how are they received?”.

A quick Google search hints that web sockets could be the perfect solution to this problem.  Using node.js, I could write an app to sit on a central server as well as some client side code for each device which could open a web socket back to the server.  Such a project sounds like a lot of fun, but it also sounds like a lot of work.  Perhaps this would be a good time to avoid the temptation to “roll my own” and instead use a pre-existing solution.  Back to Google and sure enough, there are some really interesting looking online APIs which will provide exactly this functionality.  A particularly intriguing solution is offered by a service called “Pusher” (http://pusher.com).  Pusher describes itself as “A hosted API for quickly, easily and securely adding scaleable, real time functionality to web and mobile apps”.  The phrase “real time functionality” stands out with regards my potential use cases.

What then, is involved with using Pusher?  For the most part, it’s really rather simple.  The first step is to create an account at the Pusher web site.  Easy enough; they only need an e-mail and a password to create a free “sandbox” account.  The sandbox account is limited compared to the premium accounts, but even this basic level allows serious testing of the Pusher service. Included in the sandbox account are 100,000 messages per day.  If I need more messages, they have a ramped pricing scheme to handle different levels of activity.  For now though, 100,000 is more than enough.  The next step is to add an “app” to my Pusher dashboard.  Again, simple enough – just enter a name for the “app” and click the “Create App” button.  With an “app” in hand there are two pieces of information I need from Pusher:  the App Id and the app token, which consists of a “key” and a “secret”.  Both of these are presented in the “API Access” page on my dashboard for a specified app.

And now, I’m ready to give things a try.  The Pusher site conveniently provides a snippet of HTML/javascript which can be copied into a test web page.  The sample is not even 20 lines long with the key lines being one which loads the javascript library from pusher.com:

<script src="<a href="http://js.pusher.com/1.12/pusher.min.js">http://js.pusher.com/1.12/pusher.min.js</a>" type="text/javascript"></script>

and then several lines which subscribe to my test app and provide an alert popup when there is activity:

var pusher = new Pusher('key_as_provided_by_pusher.com_dashboard');

var channel = pusher.subscribe('test_channel');

channel.bind('my_event', function(data) {



That’s about as simple as things could be.  I was able to spin up a test web page in just a couple of minutes.  To test things, there is a handy “event creator” panel on my Pusher dashboard.  From that panel, I can enter a simple text message to send to my app on the ‘test_channel’.  I try the obvious “Hello World!” message and boom! on my web page, an alert appears with my message.  Very cool.  Very simple.

Unfortunately, things got a little more complex when I wanted to move past the basic example.  My use case is to push events (i.e. sensor data) from my system to the internet.  Pushing events (vs merely subscribing to event channels) requires the use of “private” channels.  A private channel is indicated by simply prepending “private-” to a channel name.  So far, easy enough.  However, for client apps to subscribe to a private channel an added authorization step is required.  This is where things get a touch more complicated.  The specifics of the process are described  on the Pusher site:  http://pusher.com/docs/authenticating_users  – It looks intimidating, but it really isn’t all too bad.

The bottom line is that to authorize use of a private channel, I needed to add some code to my web server which can reply to the authorization request made by a client application.  The server needs to create a hash based message authorization code (or HMAC) based on the channel name, and the “socket id” which is returned when connecting to pusher.com, and signed using my private Pusher key.  It all sounds more complicated than it really is.  I will discuss the details in my next post when I explore some Python code for pushing my Esplora data to Pusher.

So, where do I go from here?  Pusher seems a great solution for pushing data to web pages.  But, I still need to be able to move sensor data from the Esplora to Pusher from where it can be broadcast to connected browsers.  Fortunately, this too is all rather straightforward.  On the Pusher site, there are a number of examples provided in a variety of programming languages (Python, Ruby, Java).  I have a Python environment installed on the desktop machine to which my Esplora is connected, so I will be using Python.  The basic idea will be to use my Esplora streaming code (see prior posts) to move data to my desktop machine.  On the desktop system, a Python script will grab the incoming data and then forward it to Pusher.  This is a task I will leave until my next post.  Until then, back to the code mines.