Leap Motion

Leap Motion is a slick little infrared sensor unit you can buy for $80 online, or at your local Best Buy. A quick install later and you can now wave your hands in space above the unit and interact with your computer in three dimensions.

I had the pleasure of working with Leap for a partner proof of concept and thought I’d give you some of my early thoughts and observations.

Some of these resources are linked from an impress.js slideshow I presented at NebraskaJS. The open-source presentation and relevant source code are available on github.

The first decision you’ll want to make is what computer language you’d like to work with the Leap from. developer.leapmotion.com walks you through your options: The SDK supports C++, Java, Python, C#, Unity and Javascript. We had the deepest bench of resources available in Javascript, and I very much liked the idea of deployment via web servers without requiring installation on the client side, so decided to pursue Javascript as our platform.

The Leap SDK is available for OS X, Linux, and Windows. (Sorry, AIX fans.)

Since we chose Javascript, js.leapmotion.com became our starting point for much of our research and development. Plan on spending a couple hours exploring that resource. There’s a lot of ground to familiarize yourself with.

There are two broad categories of Leap software I’d like to touch on. The first is WebGL powered by three.js, and the second is a 3D-ready cursor management library called leap-cursor-library.js.

WebGL (three.js)

If you haven’t perused the three.js examples lately you owe it to yourself to spend an hour clicking through them. Some are amazing, and the variety is impressive. We can interact with these in 3D, hands-free, with the Leap! Perhaps we want to control camera movement, flying through our scenes? Or perhaps we want to manipulate (or create) 3D objects via hand movements? 3D pinch, spread, and swipe gestures can be useful additions to a traditional command set. Or simply augment mouse controls with positional cues, or by adding a few Leap gestures to your existing user interface design.

The possibilities are vast, and I predict you’ll find yourself limited not by the broad constraints of the Leap API, but by the gap between what seems like a good idea at the time versus what real-world users find intuitive. I’ve been repeatedly educated by plopping a Leap in front of various users cold and watching them attempt to interact with applications. Often interactions I predict will be obvious fail badly while other design decisions seem to work well for untrained users. The only way to know is to try it out on actual users. I predict the results will surprise you.

Cursors (leap-cursor-library.js)

If you don’t want to jump into the deep end of 3D environments yet, perhaps you’d like to try your hand at mouse cursor augmentation with the Leap?

You may not be aware (I certainly wasn’t) how complex “simple” cursor manipulation is if you try to handle everything yourself in Javascript from scratch. Modern Javascript libraries make it easy (terse syntax) to add lots of mousemove event handlers quickly. But how they did that under the hood ain’t pretty, unless you’re amongst the javascript elite who debug jquery.js for fun on the weekends.

Luckily for all of us, Ross Gerbasi already wrote a heap of underpinnings, and put his work on github. Using his library, adding basic Leap support to any website becomes trivial:

<script type="text/javascript" src="../build/js/leap-manager.min.js"></script>
<script type="text/javascript" src="../lib/js/leap.min.js"></script>
<script type="text/javascript">
  LeapManager.init({
    maxCursors:5
  });
</script>

Now suddenly when you put your hand over your Leap your fingertips all magically appear on top of your website. And your hover and click events will fire based on Leap position. That was easy!

From there, explore each of the examples to see the various additional features you get for free. If you’re like me you won’t use most of them, but it’s nice to know they’re there if you should ever decide to use them.

This library works quite well out of the box. A huge time saver. You should check it out.

Gestures

Whatever platform and libraries you decide to use, you’ll want to be aware of gestures early on. The JSON API cannot be accused of dumping insufficient data at you. You’ll pile up hundreds of kilobytes of JSON in no time.

By default each frame contains everything you could possibly want to know about a point in time. The x, y, and z positions of each of your fingertips (and palms), and each of their vectors and velocities relative to the previous frame. (And lots of other goodies I won’t get into here.)

So when I make a swipe motion over my Leap, I can access and process ~60 frames per second, each containing all of the data mentioned above. But what if I really don’t care about all of that, I just want to do something when a swipe happens?

Gestures are one possible answer to coping with the data flood. In addition to all the normal granular data above, the Leap also throws gesture event information into frames whenever it detects a gesture. My swipe over my Leap throw a swipeGesture into the JSON across every frame during which the swipe started, was in progress, or ended. Mercifully we can decide to only pay attention to the single frame where state = 'stop'. In this way, we can ignore 99% of data and only examine a single frame wherein the gesture stopped:

Leap.loop(function(frame) {
  frame.gestures.forEach(function(gesture,i) {
    if (gesture.state == "stop") {
      // Debug info into the DOM (via jQuery)
      var id = "gesture_" + gesture.id;
      $("#gestures").append("<span id='" + id + "'>" + gesture.type + " </span>");
      $("#" + id).fadeOut(400, function() {
        this.remove();
      });

This can be very useful, but be sure to allocate lots of time to practice with Leap gestures. I, and others, have found that you need to move you hand very slowly when triggering gestures. But not too slowly. And when you swipe (slowly!) to the right you have to move your hand even slower back to the left to then swipe to the right again. Practice, practice, practice.

Good luck!

I hope you found these tips useful. Feel free to contact me for more pointers if you get stuck. And stop by irc.freenode.net #leapmotion and say “hi.” :)

Enjoy!

Tags: technology javascript leap-motion graphics