Researchers use Apple Watch to pilot drone, control HomeKit Hue lamps via hand gestures

in Apple Watch
Engineering students at Taiwan's National Chung Hsing University demonstrated a clever use of the motion sensors in Apple Watch to interpret hand gestures, enabling them to remotely control real world devices akin to the science fiction fantasy depicted in Star Wars.

Mark Ven of PVD+ controls a drone using Apple Watch, via hand gestures Source: Reuters

The Force Awakens: use the Dong

A group of five researchers, including civil engineering PhD student Mark Ven and University professor Yang Ming-der have been working at PVD+ since 2013, developing software they call Dong coding to interpret hand gestures, notes a report by Reuters.

Simply wearing an Apple Watch provides enough motion controls--thanks to the device's gyroscope and accelerometers--to allow the researchers to pilot a Parrot AR Drone 3.0 using hand movements, or alternatively turn on Philips Hue HomeKit lamps using a clap, then activate a given color by tracing the outline of a character (such as drawing out a "R" to turn the lamp red).

Ven demonstrated using PVD+ software to fly a drone in Taichung City (above), where he was interviewed by Reuters. Ven also demonstrated using Apple Watch to remotely control a Sphero robotic toy and control HomeKit-capable devices.

"Previously we've needed complicated controls to fly drones, but now we can use a wearable device, and through human behavior and gestures directly interact with them - using a hand to control and fly drones directly," he said.

Google's YouTube appends a dreadfully annoying 45 second ad on the one minute, 47 second video clip published by Reuters, but you can also watch the video ad-free, albeit using Adobe Flash (above).

PVD+ is seeking to patent and commercialize the technology, which appears to be an interesting new application of wearables, an emerging market Apple entered over the past year with Apple Watch, which it rapidly turned into a $7 billion business across its first 9 months on the market.

Swiss Watch industry annual sales ~$25 billion. First three quarters of Apple Watch sales ~7 billion..

— Ben Bajarin (@BenBajarin)

Apple in motion

Apple has been working on enabling technologies related to hardware motion sensitivity since the iPhone first appeared in 2007 with a proximity sensor, a 3-axis accelerometer (for tilt, motion and bump/shake detection) and WiFi location features.

The company subsequently gave iPhone 3G full Global Positioning Satellite support, added a digital compass to iPhone 3GS, and then updated the motion-sensing accelerometer to a 6-axis gyroscope in iPhone 4, capable of determining pitch, yaw and roll (twisting movements).

In addition to making motion sensors available to developers, Apple also began using introducing novel motion gesture applications of its own, including shake-to-shuffle playback on iPod nano and shake to undo in iOS, as well as more sophisticated health and sports related motion tracking managed by HealthKit.

Apple has also incorporated novel support for proximity and micro-location sensing with Bluetooth and WiFi related geofencing features, including Continuity HandOff and retail-related iBeacons.

Apple has also incorporated sophisticated low-power chip logic for managing motion-related data in its M-series components used in iPhones and iPads, as well as the software frameworks to make background tracking of motion sensor data easier for developers to access and use, once the user gives their apps permissions.

In addition to iPods, iOS devices and Apple Watch, Apple has also added basic motion controls to the Siri Remote used by Apple TV 4.


  • Reply 1 of 10
    This plus Apple's increasing interest in Virtual Reality / Augmented Reality are exciting raw material for what could be a major new form of human/computer interaction. Imagine a VR game on Apple TV -- possibly while wearing lightweight VR glasses -- and an Apple Watch on each wrist to control your interactions with the virtual world.
  • Reply 2 of 10
    SoliSoli Posts: 10,035member
    It appears to mirror Myo's abilities, but in a considerably smaller package, and without requiring it to be a stand-alone device.

    edit: I'm not 100% certain, but I do think that's the company from around 2011(?) that had what appeared to be a vaporware-only, CGI-based demo video. I can't seem to find the specific video in question to make sure it's from the same people, but Myo does sound familiar. If anyone knows the video to which I'm referring, I would appreciate if you posted a link.

    edited January 2016
  • Reply 3 of 10
    rezwitsrezwits Posts: 879member
    Apple Watch is fail, throw away now, (Psyche). We are not going to wait for people to write apps for the watch throw it away! (Psyche), It's funny because this is pretty awesome, there are so many applications for just this "one" application.
  • Reply 4 of 10
    lkrupplkrupp Posts: 10,557member
    “Any technology sufficiently advanced is indistinguishable from magic.” Arthur C. Clark
  • Reply 5 of 10
    lkrupplkrupp Posts: 10,557member
    Hey all you anti-immigrant types. You still wanna keep these guys out? If I were Tim Cook I’d be sending them gold plated H1 Bs.
  • Reply 6 of 10
    cnocbuicnocbui Posts: 3,613member
    A drone being piloted by an engineering student using an Watch has crashed, narrowly missing...  when a hornet...
  • Reply 7 of 10
    Wow! Is the Globe and Mail for real? Still using Flash for video? Talk about a dinosaur of a company.
  • Reply 8 of 10
    The control lags horribly making the idea worthless.
  • Reply 9 of 10
    nolamacguynolamacguy Posts: 4,758member
    Telstar said:
    The control lags horribly making the idea worthless.
    if only we were able to make computing devices faster. damn.
  • Reply 10 of 10
    We did something similar a year ago. KITA-MATA (means Up-Down in Congolese language) : A Drone for agriculture. We used a Google Glass app to pilot a Drone using hands gesture and voice command in African language (local dialect). So a farm owner could fly it down on the farm and assess data collected from the sensors via Google Glass. 
Sign In or Register to comment.