Apple considered VR controllers & rings for Vision Pro, but didn't like the ideas

AppleInsider

Early in the Apple Vision Pro development cycle, the company toyed with bespoke VR controllers for the headset, but ultimately decided that eye and finger tracking with cameras was the way to go.

Apple Vision Pro

At some point in early development, Apple reportedly company considered a device like a smart ring to control the Apple Vision Pro. According to the latest newsletter from Mark Gurman, the company also reportedly tested third-party VR controllers from HTC at least.

And, it apparently has no intention of developing a VR controller for itself. Apple's distaste for this kind of controller also extends to lack of support for third party controllers -- at least for now.

But, it works with game controllers from Microsoft and Sony that are intended for consoles. Apple's macOS and iOS support a wide array of controllers as it stands now, and it's not clear if this range will work on Vision Pro.

Apple has an in-air keyboard for the Vision Pro, of course. The headset will also support a Bluetooth one, or one connected to a Mac.

Apple's work on a "smart ring" may have been a predecessor to Apple Vision Pro

The company has been working on technology surrounding a smart ring for some time. Apple has been researching smart rings plus accessories for it.

Most recently, a newly-granted patent, "Skin-To-Skin Contact Detection," covers multiple ways of detecting "contact or movement gestures between a first body part and a second body part" like snapping fingers, or gestures that were covered in the WWDC keynote that revealed the Apple Vision Pro. That includes options that seem more relevant to Apple Watch bands, or even just a person's hands interlocking, but it comes down to both skin and gesture detection.

One of the example illustrations shows how a person's hand position changes when they press their finger and thumb together, for instance. A second illustration shows when "the index finger is now making contact with the thumb."

Apple's proposed system for that smart ring, and likely the Apple Vision Pro too, will recognize the touch, and also generate "a sense output signal when the index finger and thumb make and break contact."

Other examples of Apple's research on gesture detection show a user pressing a finger of one hand into the palm of the other, which in theory doesn't necessitate a physical controller.

Read on AppleInsider

xed

Just like with the Apple Pencil and the iPad, I don't expect an auxiliary device will be the default use for Vision Pro, but I do expect it to be exist. There are simply too many use cases where even the most advanced finger tracking can't do everything one might need in a VR or AR environment.

retrogusto

The eye+gesture control appears to be one of the best things about Vision Pro, but I can see something like wireless haptic gloves with small haptic clickers built into the fingertips coming out eventually, either from Apple or a third party.

tht

retrogusto said:

The eye+gesture control appears to be one of the best things about Vision Pro, but I can see something like wireless haptic gloves with small haptic clickers built into the fingertips coming out eventually, either from Apple or a third party.

The hand tracking pushes against the idea of having hand controllers, especially for Apple. The current implementation is not fast enough for fast twitch games, but if Apple continues to improve hand tracking, I think the will continue to view it as the best option. Say, the latency on hand tracking gets down to 5 ms (200 Hz). If it gets that fast, you won't need controllers to play a fast twitch game.

Camera and IR frame rates would have to go up to 400 to 500 Hz. So, probably a doubling to tripling from where the VP is at now. It isn't going to be cheap computationally or power efficient, but I would definitely prefer them to push hand and eye tracking to the point below human perception. It would benefit all applications, not just games.

mariner8

I’ll take another run at this, expanding a bit on a previous post. 

Think about the thumb-finger tap as a secondary ‘single’ switch, locking in or activating a primary selection that’s made by eye-gaze, timing (e.g. capturing the moment when a target comes into view or moves into the crosshairs, or when a rolling highlight hits the desired selection in a drop-down or matrix menu).  

Consider that *any* single-switch has exactly the same function and can accomplish the same purpose as a thumb-finger tap.  Under what circumstances would this be useful?  Well, if your fingers don’t work but your toes still do, that would be useful (try it now).  If your fingers and toes don’t work but your tongue still does, that would be useful (try making a little ‘tetch’ sound with your tongue now).  

There are scores of single-switches that have been developed and used by people with disabilities for over 40 years.  Think about Steven Hawking writing A Brief History of Time with a single switch (Walt and Ginger Waltoz built that solution to run on an off-the-shelf laptop instead of a $7,000 AAC device).  Run a Google or Chat GPT search on the phrase [disability single-switch interface]. 

Speech-to-text and text-to-speech were developed in the disability field and then adopted for use by ‘the rest of us’.  The first full-fledged Dragon Dictate system cost many thousands of dollars, and its successors are now a free component of every major operating system, including devices in the $50-$100 range. I just filled in a YouTube search on my TV set by speaking into an Apple TV controller. 

Think about sound-activated switches, sip-and-puff switches, proximity and capacitance switches (you may have used one the last time you got on an elevator), voice-powered switches, and coming up, EEG switches. The interesting thing here is that *any* single switch can be linked to any device through a universal port (there’s at least one on the device that you’re using to read this right now). Apple in particular has added dozens of accessibility and switching features to its standard iOS and OS systems.  Adding that capacity to the Vision Pro should be a cinch. 

David Wetherow 
The Star Raft Project 

edited July 2023

beowulfschmidt

tht said:

retrogusto said:

The eye+gesture control appears to be one of the best things about Vision Pro, but I can see something like wireless haptic gloves with small haptic clickers built into the fingertips coming out eventually, either from Apple or a third party.

The hand tracking pushes against the idea of having hand controllers, especially for Apple. The current implementation is not fast enough for fast twitch games, but if Apple continues to improve hand tracking, I think the will continue to view it as the best option. Say, the latency on hand tracking gets down to 5 ms (200 Hz). If it gets that fast, you won't need controllers to play a fast twitch game.

Camera and IR frame rates would have to go up to 400 to 500 Hz. So, probably a doubling to tripling from where the VP is at now. It isn't going to be cheap computationally or power efficient, but I would definitely prefer them to push hand and eye tracking to the point below human perception. It would benefit all applications, not just games.

I would expect the vast majority of current fast twitch gamers are going to insist on either a controller or keyboard/mouse.  The advantage of such is that there is no uncertainty at all as to which control is being pressed, even if one's hand might be in a slightly different position.  I'm not sure hand tracking will have that kind of precision and lack of uncertainty anytime soon.  I do think it's a solvable problem, but the current generation of gamers isn't going to want to throw away all the work they've put into developing muscle memory, especially since Apple is still showing only lukewarm interest in any kind of high end, fast twitch games.  Of course, based on some numbers I saw recently, the mobile game market is larger than all the console markets put together, so I'm not sure they really need "high end" games.

eriamjh

I don’t think the finger touch is meant to replace controllers.   Doesn’t apple show someone using a controller in their own promo shots?   

This is about stand-alone use.   Apple can easily add BT click-mouse device-like support, voice-select, tongue-select, and many other methods.   

It’s the dawn of a new age.   Let some light shine before running off into the dimness of it.  

jahblade

I am truly excited to see what third-party hardware accessories developers come up with. Very exciting times to come!

xed

chutzpah said:

Xed said:

ATLMacFan said:

Ya’ll deleted my comment about your typo, but left the typo.

”Apple reportedly company considered.”

Again, I ask, where are your editors?

I've never understood why someone would make an account and post just to point out a fucking typo. Did you understand what was being stated? Do you really get a feeling of superiority from noting the typo? Do you go to sleep thinking "I could make a better rumors website than AppleInsider because I wouldn't ever publish typos"?

Let's all tolerate slopping editing in online journalism.

Then send a private message to the editor if the error is that egregious to you—I’ve done that when something was woefully incorrect that it altered how the piece reads—but to create an account and then come find the typo in the article is fucking pathetic. Now the thread is permanently jacked because some you had to note that the foolish author wrote ‘teh ’ instead of ‘the’, for example. And for what? So you can prove how you are superior to the author? At least the author is contributing something..

edited July 2023