Apple aims to patent ergonomic, full-hand multi-touch input

[AppleInsider]

Three patent filings set into motion by Apple just a month after the iPhone's US debut open the door to curved multi-touch surfaces that can recognize more than just fingertips.

Pieced together through US-based patents for a sensor layout as well as those for mobile sensors and compliant conductors, the collective technology uses improved touch input nodes that are accurate enough to create a sensor image of different parts of the hand while not being bound to any particular size, shape, or resolution.

This will let a given multi-touch device not just recognize more complicated gestures, such as grabbing or swiping motions with one or more fingers, but also selectively disable input depending on the immediate context. Typists could leave their palms on a touch-sensitive device without activating controls while gaining the palm rest area back for other functions when necessary, or cease moving a cursor when a finger comes to a complete stop.

A version of the technology with pressure sensitivity could also exploit this ability to recognize rolling, tilting, or twisting motions for manipulating content in 3D, Apple explains in the patents.

But because the touch controls would not have to be flat, the combination of these advancements could lead to particularly unique designs. One concept explained in the filings would have a curved surface designed to be ergonomic over long periods, such as with a keyboard. It could also detect when a user is drawing with a stylus through both the inner fingers and the movement of the palm.



The patents are not connected to any existing product design from the Cupertino, California-based firm, whose iPhone and iPod touch, or any future handhelds, are unlikely to benefit from the advancement. Instead, the notion of a hand-sensitive, curved controller is intended primarily as a "computer input device" that may eventually replace both a keyboard and mouse.

[ View this article at AppleInsider.com ]

[coolfactor]

Once this is a real product, traditional keyboards and mice will look archaic, and Windows PCs will still use them. The Mac will, once again, lead the pack for how human-computer interaction should be.

Ah, the dreams of Star Trek continue to make their way into real life.

[g3pro]

We have here the patent for the Mac Touch, an 8.5" x 11" input device/screen/computer, essentially the iPod Touch/iPhone for work. The true tablet computer without a rotating screen or clanky keyboard. Fully functional computer with stylus input included for handwriting. Debuting in Macworld January 2008.

Thread over.

[Kolchak]

Nothing more than an upgraded version of the Fingerworks Touchstream LP, still going for up to $1000 on eBay.

The "dreams of Star Trek"? More rubbish. When did Trek ever use MultiTouch? All they ever had were fake single-point touchscreens, no gestural or pressure input. At most, they used three fingers to simulate the original transporter sliders. Take Trek off that pedestal where it doesn't belong and come back to the real world.

Originally Posted by g3pro

We have here the patent for the Mac Touch, an 8.5" x 11" input device/screen/computer, essentially the iPod Touch/iPhone for work. The true tablet computer without a rotating screen or clanky keyboard. Fully functional computer with stylus input included for handwriting. Debuting in Macworld January 2008.

Thread over.

I think not. Where does the article say anything about a tablet computer or even a touchscreen? This is a replacement for keyboard and mouse, as the last sentence said.

[Kevinneal]

Strange, but last week I heard a rumour that Apple were working on iPhone2 and it used a flexible design to allow it to bend when you sit down - I dismissed it as rubbish, but you never know...

[aplnub]

Originally Posted by Kolchak

Nothing more than an upgraded version of the Fingerworks Touchstream LP, still going for up to $1000 on eBay.

The "dreams of Star Trek"? More rubbish. When did Trek ever use MultiTouch? All they ever had were fake single-point touchscreens, no gestural or pressure input.

I just want to state that they could totally reconfigure their stations to better suite their needs. Also, you have to define multi-touch because I am pretty sure they had some of this technology on a few shows and a movie.

But I think the point was not that it was a rip-off of Trek, but rather Futuristic like Trek.

[earthy]

I have read the whole application through and what it is mainly about (apart from the arrangement of sensors plus conducting surface and the software algorithms for distinguishing different handshapes and gestures) is a single input device that combines keyboard and multi-touch input functions on a single surface. The device they talk about has an arched, curved surface for a more ergonomic typing position and also to avoid unintended crossing of the two hands (and to make it clear to the system that this is intentional hand crossing by the user). This input device is supposed to have a slight grooved surface for home row fingers and the thumbs plus small conductive raised dots for the other fingers to facilitate touch typing. The key characters would be preprinted on the top surface. The single groove and small dots are intended to keep the overall surface as smooth as possible for advanced multi-touch input to replace the need for a primitive pointer system such as a mouse (and the need for things such as scroll bars).

The application also mentions in passing other possible devices, such as a curvable surface that could be placed on the forearm or an armrest, and a larger surface including a display, but without tactile feedback, where the sensor system would display typing key positions appropriate to the size of the hand detected on the surface. The first might be a possible future wearable version of an iPod or iPhone; the second might point to an all-in-one display plus input device. They seem to be making sure to say that there are various ways the technology can be realised in future devices but the main emphasis is clearly on an ergonomic, soft surface touchpad combining keyboard and multi-touch interfaces in the same unit, with output to a separate display screen.

If anyone is interested, I can post the relevant parts of the application that refer to the actual physical device(s) (as opposed to the algorithms and technical details of the sensors).

[Bageljoey]

Originally Posted by Kolchak

The "dreams of Star Trek"? More rubbish. When did Trek ever use MultiTouch? All they ever had were fake single-point touchscreens, no gestural or pressure input. At most, they used three fingers to simulate the original transporter sliders. Take Trek off that pedestal where it doesn't belong and come back to the real world.

Major Buzzkill, dude.
Sure they didn't need MultiTouch tech in Star Trek because, by then, Steve Jobs the 13th had developed a streamlined, simple, clean OneFinger interface. But we are a long way from there now with our primitive, pre-warp drive technology...

.. STAR TREK
. ----------
. ---------
... \\\|||///
.... \\|||//
..... |||||||
..... |||||||
..... |||||||
.... //|||\\
.. ======

[Banalltv]

Let's hope they make it so.

[hypoluxa]

I can almost bet (guarantee) we will see this come to fruition in the next 5 years... bye bye old keyboard!

[Kolchak]

Originally Posted by aplnub

I just want to state that they could totally reconfigure their stations to better suite their needs. Also, you have to define multi-touch because I am pretty sure they had some of this technology on a few shows and a movie.

But I think the point was not that it was a rip-off of Trek, but rather Futuristic like Trek.

Nope. They never had it. All they ever had were virtual buttons that they would tap with single fingertips, never any rolling, twisting, grabbing, tilting motions or anything involving more than one finger (with the exception of the aforementioned transporter sliders). Besides, nothing in the article says anything about reconfigurability. These aren't touchscreens but rather a more advanced version of FingerWorks products.

I'm just sick and tired of Trek being brought up as something prophetic every time a vaguely similar design appears, like Leonard Nimoy claiming that clamshell cell phones were an offshoot of the original communicator. Sure, Leonard, maybe if you held the old communicator against your ear! Would you like to claim Uhura's earpiece as the genesis for Bluetooth headsets, too? It was fantasy, just some production designer saying, "Hey, it'd look neat if it did this" for no real reason. Not even hard science fiction (now real science) like Arthur C. Clarke's geosyncronous satellite, which actually needed some knowledge of science to be conceived.

[aplnub]

Very bold to say it never happened. I'll take your word for it until I prove otherwise. Of course, I may be thinking of the latest Star Wars movies where they used holograms in 3d to do computer stuff. Not that it is a big deal. I don't know why it can't be as simple as Superman's computer system. Just shove a crystal in the thing!!

[SpamSandwich]

Originally Posted by earthy

I have read the whole application through and what it is mainly about (apart from the arrangement of sensors plus conducting surface and the software algorithms for distinguishing different handshapes and gestures) is a single input device that combines keyboard and multi-touch input functions on a single surface. The device they talk about has an arched, curved surface for a more ergonomic typing position and also to avoid unintended crossing of the two hands (and to make it clear to the system that this is intentional hand crossing by the user). This input device is supposed to have a slight grooved surface for home row fingers and the thumbs plus small conductive raised dots for the other fingers to facilitate touch typing. The key characters would be preprinted on the top surface. The single groove and small dots are intended to keep the overall surface as smooth as possible for advanced multi-touch input to replace the need for a primitive pointer system such as a mouse (and the need for things such as scroll bars).

The application also mentions in passing other possible devices, such as a curvable surface that could be placed on the forearm or an armrest, and a larger surface including a display, but without tactile feedback, where the sensor system would display typing key positions appropriate to the size of the hand detected on the surface. The first might be a possible future wearable version of an iPod or iPhone; the second might point to an all-in-one display plus input device. They seem to be making sure to say that there are various ways the technology can be realised in future devices but the main emphasis is clearly on an ergonomic, soft surface touchpad combining keyboard and multi-touch interfaces in the same unit, with output to a separate display screen.

If anyone is interested, I can post the relevant parts of the application that refer to the actual physical device(s) (as opposed to the algorithms and technical details of the sensors).

Sounds like future Mac users will all dress like this...

[mdriftmeyer]

Originally Posted by SpamSandwich

Sounds like future Mac users will all dress like this...

Those looks were hard to find back when the world's majority wasn't obese.

Best of luck in finding a ship full of Linda Carters.

[nvidia2008]

Originally Posted by AppleInsider

...curved multi-touch surfaces that can recognize more than just fingertips....

Umm... Am I the only one that thought of pr0n applications straight away when I read that????

[JeffDM]

Originally Posted by Kolchak

Nope. They never had it. All they ever had were virtual buttons that they would tap with single fingertips, never any rolling, twisting, grabbing, tilting motions or anything involving more than one finger (with the exception of the aforementioned transporter sliders). Besides, nothing in the article says anything about reconfigurability. These aren't touchscreens but rather a more advanced version of FingerWorks products.

I could have sworn that the "map room" in Generations had some stuff like that.

Has Apple implemented rolling, twisting or tilting yet?

I'm just sick and tired of Trek being brought up as something prophetic every time a vaguely similar design appears, like Leonard Nimoy claiming that clamshell cell phones were an offshoot of the original communicator. Sure, Leonard, maybe if you held the old communicator against your ear!

You're leaving out the two-way mode. Not that I like that feature.

[digitalclips]

Originally Posted by Kolchak

I'm just sick and tired of Trek being brought up

They did this just to annoy you I think

"Enterprise, the first Space Shuttle Orbiter, was originally to be named Constitution (in honor of the U.S. Constitution's Bicentennial). However, viewers of the popular TV Science Fiction show Star Trek started a write-in campaign urging the White House to select the name Enterprise."

[digitalclips]

Originally Posted by JeffDM

Has Apple implemented rolling, twisting or tilting yet?

Tilt and games over

[onlooker]

Originally Posted by Kolchak

Nothing more than an upgraded version of the Fingerworks Touchstream LP, still going for up to $1000 on eBay.

..........................

Immediately what I thought also. This will probably NOT be a on screen typing system, but an improvement to the fingerworks iGesture pad (wacom-ish) who's patents Apple owns through an acquisition. Something of an onscreen menu will probably let you choose how you wish to navigate. It was a fantastic idea before Apple got a hold of it. I would think of it in terms of the PARC-GUI vs. (1984) what Apple did with it.

[wheelhot]

I wonder when we will see Apple releasing a product using this patent

[PhuzzyToes]

Originally Posted by Kolchak

Nope. They never had it. All they ever had were virtual buttons that they would tap with single fingertips, never any rolling, twisting, grabbing, tilting motions or anything involving more than one finger (with the exception of the aforementioned transporter sliders). Besides, nothing in the article says anything about reconfigurability. These aren't touchscreens but rather a more advanced version of FingerWorks products.

Agreed.

But I may respectfully point out that the multi-touch screen idea has precedents out there. One example has been shown by Jeff Han (see the February 2006 TED talk). Notice how he handles pictures on his virtual platform. This is no Star Trek, but a enabled application that shows both utility and novelty. Curiously this presentation was given almost a year prior to Steve's iPhone announcement at MacWorld 2007. Hopefully we're not gonna experience a Xerox-Lisa-Mac syndrome all over again.

[onlooker]

Originally Posted by PhuzzyToes

Agreed.

But I may respectfully point out that the multi-touch screen idea has precedents out there. One example has been shown by Jeff Han (see the February 2006 TED talk). Notice how he handles pictures on his virtual platform. This is no Star Trek, but a enabled application that shows both utility and novelty. Curiously this presentation was given almost a year prior to Steve's iPhone announcement at MacWorld 2007. Hopefully we're not gonna experience a Xerox-Lisa-Mac syndrome all over again.

One example. Where have you been? Jeff Han pioneered all of it. Even Minority Report was based on his stuff. Apple has some sort of an arrangement with Jeff Han involving similar patents. I believe Microsoft should have some arrangements with him too, but I wouldn't put it past them not to. They'll probably try to rip him off like they do XEROX, and Apple.

[PhuzzyToes]

Originally Posted by onlooker

One example. Where have you been?

Living under a rock. Not much is going on down there.

Originally Posted by onlooker

Jeff Han pioneered all of it. Even Minority Report was based on his stuff.

I stand corrected.

Originally Posted by onlooker

Apple has some sort of an arrangement with Jeff Han involving similar patents. I believe Microsoft should have some arrangements with him too, but I wouldn't put it past them not to. They'll probably try to rip him off like they do XEROX, and Apple.

Apparently it's FingerWorks that was acquired by Apple. Not sure how Jeff fits in this transaction. I was hoping to find some clarification, but all I found is a March 2007 NY Times article instead. I'm hoping what's not said is Jeff receiving generous compensation.

[onlooker]

Originally Posted by PhuzzyToes

Apparently it's FingerWorks that was acquired by Apple. Not sure how Jeff fits in this transaction. I was hoping to find some clarification, but all I found is a March 2007 NY Times article instead. I'm hoping what's not said is Jeff receiving generous compensation.

The Fingerworks acquisition, and the "thing" with Jeff Han have nothing to do with each other.

[chris v]

Originally Posted by g3pro

Debuting in Macworld January 2008.

Dream on.

[pixelcruncher]

I would love for my Wacom tablet to have a few multi-touch features to manipulate images and objects (rotating an object along an axis with a finger while painting with the stylus, for example), but I don't see how that surface can double as my keyboard. My tablet is skewed to favor my right-handedness, and the tablet and keyboard are at completely different angles to my desk (for comfort). Not to mention that I frequently am giving one set of inputs with one hand on the keyboard while I am moving my cursor/providing input with my stylus (most people do the same thing with a mouse). If your surface doubles as your keyboard and stylus/mouse, not sure how you can keep both going at once. It may be able to switch to mouse mode automatically when I drag my finger across the surface, but what if I'm just tapping on a paint swatch or trying to select a file on my desktop? Seems like there has to be at least two independent surfaces.

[onlooker]

pixelcruncher, you'll just have to see what Apple does to find out.

[vinea]

Originally Posted by onlooker

One example. Where have you been? Jeff Han pioneered all of it. Even Minority Report was based on his stuff. Apple has some sort of an arrangement with Jeff Han involving similar patents. I believe Microsoft should have some arrangements with him too, but I wouldn't put it past them not to. They'll probably try to rip him off like they do XEROX, and Apple.

You must be being sarcastic. Jeff Han did not pioneer multitouch and as far as I know holds no patents on FTIR for which there should be numerous prior art. He did get a nice little contract from DARPA to kick off his company.

He also did not influence Minority Report. Looking at the concept art for MR and these cannot be implemented with any of the stuff Han was working on at NYU...primarily FTIR

MS likely has no current arrangements with Han and there shouldn't be a real need to since they have their own excellent team and has hired some of the folks that DID pioneer multitouch...like Bill Buxton.

I'm waiting on the MS MultiTouch SDK. The current examples look pretty exciting and while their timeline keeps getting pushed to the right it still seems to be further along that things like OpenTouch. Which is actually kinda interesting since there is an osx/xcode version...

Multitouch has been around since 1982. In 1992 Tog created a video that showed a multi-touch workstation while he was at Sun. Given that Han was an undergrad at Cornell in 1998 the field of multi-touch predates him by nearly two decades before he started working on it.

[SpamSandwich]

Originally Posted by vinea

You must be being sarcastic. Jeff Han did not pioneer multitouch and as far as I know holds no patents on FTIR for which there should be numerous prior art. He did get a nice little contract from DARPA to kick off his company.

He also did not influence Minority Report. Looking at the concept art for MR and these cannot be implemented with any of the stuff Han was working on at NYU...primarily FTIR

I'm not so sure about Han not being the influencer of Minority Report. Spielberg held a little conference with futurists and visionaries prior to making the movie to get a feel for where things could be in the future. No doubt someone familiar with Jeff Han's work contributed to this discussion, if not Han himself.

[blingem]

Originally Posted by SpamSandwich

I'm not so sure about Han not being the influencer of Minority Report. Spielberg held a little conference with futurists and visionaries prior to making the movie to get a feel for where things could be in the future. No doubt someone familiar with Jeff Han's work contributed to this discussion, if not Han himself.

Speculation. When did pre-production of Minority Report take place? The movie came out in 2002, so that futuristic meeting took place probably in 2000 or before. Jeff Han was at Cornell in 1998, and he quit to start up a video conferencing company (thanks to Wikipedia), so I doubt he had much hand in the 2000 meeting, let alone an advanced and public body of work on the topic.

Jeff Han didn't invent multitouch anyways, his ideas regard "interface-free" designing. He sure can show it off though. And he's making fat stacks of cash either way, as most people associated with Cornell and NYU evidently do or could if they wanted to...save the janitors.

[vinea]

Originally Posted by SpamSandwich

I'm not so sure about Han not being the influencer of Minority Report. Spielberg held a little conference with futurists and visionaries prior to making the movie to get a feel for where things could be in the future. No doubt someone familiar with Jeff Han's work contributed to this discussion, if not Han himself.

No. According to Wikipedia that conference occured in 1999. At that point Han was doing CU See Me.

Why is it so hard to believe that Han did not invent something that is almost as old as he is? He's a slick presenter and bright guy but even he acknowledges that he isn't even the first to use FTIR as the basis for multi-touch.

More than likely, Minority Report is what influenced Han to do more MT research. Timeframe is about right...2002 movie to 2005 SIGGRAPH presentation. Trust me, give a bright person 3 years and a couple grad students and you can create some amazing stuff.

[aplnub]

Originally Posted by Kolchak

Nothing more than an upgraded version of the Fingerworks Touchstream LP, still going for up to $1000 on eBay.

The "dreams of Star Trek"? More rubbish. When did Trek ever use MultiTouch? All they ever had were fake single-point touchscreens, no gestural or pressure input. At most, they used three fingers to simulate the original transporter sliders. Take Trek off that pedestal where it doesn't belong and come back to the real world.


I think not. Where does the article say anything about a tablet computer or even a touchscreen? This is a replacement for keyboard and mouse, as the last sentence said.

The flip phone on Star Trek: The Original Series

Karate Chop extra!

[onlooker]

Originally Posted by SpamSandwich

Sounds like future Mac users will all dress like this...

That woman is freakishly good looking for her age. I mean now not then. She was hot then too, but I was like 4, or 5.

[Olternaut]

Originally Posted by onlooker

That woman is freakishly good looking for her age. I mean now not then. She was hot then too, but I was like 4, or 5.

I think she was only like 20 in that picture. I saw her the other day as the principal in "Sky High" and she is still smokin! I wish she was my principal in high school.
I would have done mischief to get sent to the principal's office every day!

[Olternaut]

Wait. Back on topic. So what about this patent then? When will they debut products based on this stuff? If not macworld when? The February sdk announcement as in a mactouch? Or later in the year?

[SpamSandwich]

Perhaps soon we'll see the iOrb from Apple. I envision a grapefruit sized ball that responds to squeezes, taps and multiple finger input. That would be hilarious.