Apple's patent for a remote collaboration system uses digital pens and 'large digital surfaces'

in General Discussion edited January 2014
The U.S. Patent and Trademark Office on Tuesday granted Apple a patent that improves on large displays, like Microsoft's original Surface project, by adding digital pens and the internet, to give users a powerful, cost-effective collaboration tool that can be used to hold interactive meetings almost anywhere.

Source: USPTO

Apple's U.S. Patent No. 8,402,391, simply titled "Collaboration system," describes an enhanced user interface that leverages "large digital surfaces" and digital pens to create an environment in which a number of users can collaborate on various projects.

The invention largely centers on digital pens, mentioning consumer products like IOGear's Mobile Digital Scribe and tools from Anoto by name, as a means of inputting data into the collaboration system. These specialized pens have built-in sensors, either visual or motion-based, that capture user movements in real time, thus allowing for the digital reproduction of whatever is written on paper.

From the patent summary:
In one embodiment, users can add content to a digital system by using a pen that streams coordinates so that input to the digital system may be based on conventional pen and paper handwriting.
Apple gives the example of a user streaming pen coordinates to a collaboration server. From there, the system can share the information with other collaborators who are connected to the session, with the data showing up on an electronic whiteboard or other large interactive display. Here the invention notes that a user's mobile device, such as an iPhone, can be used as a type of bridge between the pen and a wireless network. Collaborators can be in the same room, or situated remotely, and the writer can choose to stream their notes or upload them at chosen intervals.

By utilizing digital pens instead of dedicated table PCs, the invention looks to cut down on costs while at the same time enhancing the collaboration process. Since the design doesn't call for multiple large displays, any number of users can effectively participate around a central collaboration station.

Moving on to these "large digital surfaces," the patent recognizes that some systems suffer from occlusions, either in screen blockage or shadows cast by a user's hand. The occlusions may pose as a hinderance to interacting with the on-screen menus, or otherwise negatively affect performance.

In one embodiment of Apple's invention, the menus are drawn to avoid the occlusion areas, adapting to a user depending on their "handedness," orientation and position on the tabletop or in front of the electronic whiteboard.

Pie Menu
Illustration of an on-screen "pie menu" with digital pen.

Instead of traditional drop down menus, the patent proposes an interface that can register clicks, drags and other UI actions oriented to a user's hands. The motions or gestures can also be recorded in the occluded area despite not having any visual feedback.

When a user wants to invoke a menu, they can use a special gesture with the digital pen, or press a dedicated button, which then triggers a specialized menu to appear around the pen's tip. The "pie menu" can contain a point and click area and a gesture area, each of which holds a plurality of icons or other visual assets that help the user navigate their options.

Various implementations of pie menu as dictated by user orientation.

Further embodiments include gesture recognition, selective omission of errant touches by elbows and the like, and various descriptions of pie menu functionality.

Apple's '391 patent was first filed for in 2009 and credits Bernard Doray, Paul To, Michael Haller, Peter Brandl and Thomas J. Seifried as its inventors.


  • Reply 1 of 3
    rogifanrogifan Posts: 10,669member

    Inventors: Doray; Bernard (Nepean, CA), To; Paul (Menlo Park, CA), Haller; Michael (Linz, AT), Brandl; Peter (Linz, AT), Seifried; Thomas J. (Linz, AT)


    So who are the guys from Austria? Apple employees?

  • Reply 2 of 3
    gatorguygatorguy Posts: 20,615member


    Originally Posted by Rogifan View Post

    Inventors: Doray; Bernard (Nepean, CA), To; Paul (Menlo Park, CA), Haller; Michael (Linz, AT), Brandl; Peter (Linz, AT), Seifried; Thomas J. (Linz, AT)


    So who are the guys from Austria? Apple employees?

    Apparently not. Here's Haller's bio:


    and Brandl:


    If I were to guess the Austrian researchers developed the idea and Apple is figuring out how to put it to use and monetize it. A joint effort between academics and private business which isn't unusual.

  • Reply 3 of 3
    jasenj1jasenj1 Posts: 916member

    Hmm... Not sure what the real innovation is from the article. Digital whiteboards have been around for a while - but they are expensive. There are also projects around that use the Wii controllers (Wiimote) to create low cost interactive whiteboards.



    I've read of teachers using Wiimote whiteboards in classes.


    At work I hooked up two of the above systems using VNC. Ta-da! Instant multi-site collaborative whiteboard. I believe there were some people working on the LEDs blinking very quickly in different patterns to be able to detect different pens.


    The Wiimote solutions have the problem that the camera always has to have line-of-sight to the LED which can be annoying. If other digital pens can provide more accurate tracking without that issue, that'd be cool.


    There's also the issue of UI elements & conventions to support such interactive surfaces. MS and others have done some cool work with the Surface, but I'm sure Apple has some good contributions to make in the space, too.


    I don't see a mass, consumer market application, though. But for schools and business meetings, a cheap, well-done interactive whiteboard would be very nice.


    - Jasen.


    P.S. FWIW, we have a couple digital whiteboards at my office. No one ever uses them. They have drivers that need to be installed (everyone just wants to hook up their laptop to the projector and go) and the software is a bit clunky. I've put in the effort to use them a couple times, and they do add value, but people in my office don't seem to care enough.

Sign In or Register to comment.