Apple Vision Pro research and early work was hidden in plain sight
From patents for 'smart rings' to ones for projecting a Mac's desktop out onto the wall, Apple has been hiding Apple Vision Pro in plain sight for years.
Why didn't we see what was right in front of our faces?
Over the last decade, there must have been hundreds of Apple patents and patent applications that were immediately and indisputably concerned with headsets. They even include the word "headset."
But there were also so many other patents that did not say "headset," and yet we now know that's what they were really for.
Such as one from 2015, eight years ago now, in which it appeared that Apple wanted to create a Mac keyboard out of thin air. Some kind of camera or sensor would be on the Mac's screen, and the user could make a typing gesture in front of it.
Data from the cameras would provide a "best guess of the user's intended input," said the original filing reads, which seemingly would allow Mac users to type more quickly.
Flash forward to Vision Pro, and it comes with exactly that kind of virtual keyboard. Rather than a camera facing out from a Mac to its user, it uses cameras looking from the Vision Pro wearer's perspective, out toward their fingers.
We thought this was about a future MacBook Pro, but it's key to the gestures Vision Pro supports
Or in 2020, it appeared that some future iMac would be able to have its display spill out onto the walls behind the device. Assuming you had a plain wall or a good splashback, the desktop could appear to extend beyond the boundaries of the Mac screen.
That would involve projectors built into the iMac, but it would also rely on the device being able to allow for screen displays to be wider than on the physical Mac. Now Vision Pro lets you take your Mac's screen and resize it, reposition it, in just the same way.
Handling a desktop that is much wider than a physical screen is surely tied to how Vision Pro can display Mac screens at any size or position
Gestures in thin air
One of the most impressive and Apple-like features of Vision Pro is that it does not come with any kind of separate controller. Instead, you just shrug, or you press two finger together -- and you can do that pretty much wherever your hands are.
You couldn't pinch your fingers together behind your back, but you can do it while casually sitting with your hands and arm to your side.
It's just that we should have guessed that no later than 2020. That's when a patent application called "Electronic Device with Sensing Strip" told us.
We just thought it was to do with a future MacBook Pro, since Apple worked quite hard to make us think so. Despite drawings that implied a laptop, some of the descriptions are now unmistakeable.
In 2020, this looked like a smart ring control. Now you see it and wonder about the pinching motion for Vision Pro
"An electronic device may have an elongated sensing strip," said the patent application. "Control circuitry may use the sensing strip to gather air gesture input from the fingers or other body part of a user."
It's not only iPhones that have cameras
Once you suspect that Apple has been hiding headset details in plain sight, you do start seeing conspiracy theories everywhere. Not every patent can be a secret headset patent, but so very many concern technologies that we can now see must inform Vision Pro as well as any other Apple device.
In 2021, there was a patent application called "System For Producing A Continuous Image From Separate Image Sources." At the time, it looked like it was quite a cumbersome idea of having one person being able to take a photograph using a whole group's iPhones, all at once.
Perhaps it is that, but while much of the patent talked about mounting multiple devices, some of it was about the stitching together of images from all of these cameras.
Using multiple cameras to create one image
"A photographer, videographer, or other person may desire to capture images using several image-capture devices," said Apple, "and combine the images captured by each of the individual devices into one continuous image."
"The combined, continuous image may have a greater field-of-view and include more image data than the individual images captured by the image-capture devices," continued the patent application.
We won't know just how Vision Pro's recording features work until it is released, but it sounds like this was paving the way.
Bezels and thin displays
There was reportedly a long-standing difference of opinion within Apple over whether the headset should come in two pieces or not. Ultimately, it appears a compromise was reached with just the battery being a separate segment, rather than any of the controls.
But whether it was to be one device, or the user would have to carry a connected iPhone as well, there must inevitably have been a pressure to make everything thin.
Such as the bezels around the screens. Apple Vision Pro still resembles goggles, but it's thin enough to be closer to ski glasses than most of its rivals.
Detail from the patent showing how light sensors might be embedded between the pixels of a display
To make thin bezels around the screen, and also to hide sensors behind the glass, Apple had another patent in 2021. "Sensing System for Detection of Light Incident to a Light Emitting Layer of an Electronic Device Display," cut the bezels, and embedded sensors.
""[This] protective outer cover defines an exterior surface of a housing of the electronic device," said Apple's patent application. "An electronic device can also include a light sensing system, such as an ambient light sensor."
"Typically, a light sensing system is positioned below the protective outer cover, adjacent to, and separated from, a light emitting region of the display stack," it continued.
Detecting the user
It was obvious that a headset would be a more personal device than a phone, and that it would have be adjustable to suit different levels of eyesight. What was less obvious was that Apple would seemingly limit the device to a user and one guest.
Then Apple didn't exactly spend a lot of time at WWDC talking about biometric identification. We now know that Vision Pro uses a new Optic ID system to verify the user, much like all the smart glasses in "Mission: Impossible."
Maybe we couldn't have guessed the term Optic ID, but we should have guessed at least Face ID would be involved. Ultimately there would be patents and patent applications that were more blatantly about headset biometrics, but in 2020 there was one about verifying identity that was ostensibly for iPhone users.
So that's why we got Spatial Audio
Spatial audio was also ostensibly for iPhone users, but now we know -- and, for once, we did guess. In 2021, one of several patent applications regarding spatial audio (SR) actually talked about its use in AR.
"For example, a SR system may detect an individual walking a few paces forward and, responsive thereto, adjust graphics and audio presented to the individual in a manner similar to how such scenery and sounds would change in a physical setting," said the patent application.
"For example, an individual may interact with and/or sense aural objects that create a multi-dimensional (e.g., three dimensional) or spatial aural setting, and/or enable aural transparency," it continued
Forget smart rings
It's debatable which rumor was more keenly followed, the "Apple Glass" headset or the Apple Car. But behind them both was the equally long-lasting rumor that Apple was working on a smart ring.
Some patents involved extending a smart ring to cover almost all of a finger, and others concerned accessories for such a device.
It's either a Smart Ring, or a Bandaid.
But there were two patent applications in 2020 that in retrospect now seem so obviously intended to be about headsets. A patent application for a "wearable ring device" was refined in 2020 to be about a "wireless finger-worn device," that was intended to control "an external electronic device."
Also in 2020, though, there was a patent application for what was called an "expandable ring device."
Notice that there's no smokescreen here. It specifically says that the wearable device is part of a system that also includes a "head-mounted device comprising a display viewable by the user."
The headset would feature "control circuitry configured to adjust the displayed images based on the received data." And the system as a whole would support "bidirectional communication between the ring device and the head-mounted device."
Given these, it's easy to see now that the one where a ring was extended into covering a whole finger could be to give a kind of headset trackpad. Then the granted application plainly concerns smart ring accessories actually talks about virtual reality, too.
"As the user is viewing virtual content... the finger devices may be used in gathering finger input," it says.
As we now know, Vision Pro does not come with any external control devices. But also as we now know, finger movement is central to the operation of the headset.
It's just that instead of some wireless connection between a ring and the headset, Apple Vision Pro cameras watch for the user pinching their fingers together.
So while this can't ever be definite, it does look likely that Apple researched smart rings specifically for use with its headset. And now that Vision Pro has a better system, the ring research may never appear in a shipping device.
Patent pending
All patent applications, by any firm, are written so that they are close to preposterously wide-ranging. The aim is lock out rivals from using the idea, or even doing anything similar.
So it's conceivable that any one example of a patent appearing to really be about the headset is a chance result of just how wide-ranging it was written to be.
But the sheer volume of research that has been hiding in plain sight mean it can't all be coincidence.
Perhaps Apple really will put thin-air gesture sensors in a MacBook Pro, and certainly all work on smaller bezels must benefit every one of its devices.
Or maybe Apple will decide it can't waste all that effort and chooses to release smart wedding rings that detect infidelity.
But we've been listening to Apple AR headset rumors for so long, and it now looks like we should've seen it coming in far more detail than we did.
Read on AppleInsider
Comments
You’re on record, at least a hundred times here alone, as being bearish on Vision Pro. So why do you feel required to chime in on every single article about the subject with the same recycled arguments?
Ok, so that was actually two questions.
It’s just a rear guard action by Apple, various judges Apple has run up against recently in our legal system, seem to interpret things as, if you don’t file a motion (case) on the front end, i.e. a direct patent challenge, then whatever you say afterwards, doesn’t matter, Apple just lost a case to Cal Tech, because on the front end, they didn’t challenge the patents upfront, in short preemptively sue them before they file their case, how do you do that upfront when you think that you are in the clear, are you supposed to sue multiple companies/schools beforehand like a patent troll? Has Apple ever done that?
JP234 said:
There should be, somewhere out there, a market analysis to determine the point of diminishing returns comparing markup/volume ratios.
IMO, there already has been plenty of market analysis by Apple and others to determine “diminishing returns” and what the market will pay, willingly pay, and “take my money!” pay for a product or service.
Some people have a problem with what THEY would pay, partly because of limited vision (pun intended) of the possibilities and because THEY believe the price MUST be lower to garner volume. They want “minimized risk” so that if the price is low enough, they won’t feel they’ve lost much if it turns out less than amazing. That’s a “you” problem.
Others know they can and will pay because they see the value and possibilities of said technology. They wish to try it for themselves, get the experience, and expand their ideas of what is possible. And they are willing, most all times capable, and able to afford the technology or experience. Of course, when you have sufficient discretionary income or assets, much more of the world’s options become available and doable.
Apple introduced the Mac at elevated prices to the cookie cutter PC industry. It’s survived and in some respects flourished. Apple introduced the costly iPod yet overtook the entire mobile MP3 player market for 10 years.
Apple introduced the $500 iPhone, roundly criticized for cost and design, and it now has dominated the upper end smartphone market since 2015’s iPhone 6 and 6S.
In 2017, Apple introduced the first $1000 iPhone X, breaching a price threshold some thought impossible for a mobile smartphone. Everyone decried its price - everyone who felt they couldn’t or wouldn’t afford it. It was a you/them problem. Yet somehow, in some manner and ways, hundreds of millions of people worldwide, that year and since, believe these products worthy enough to pay these prices, all at once or over time. Apple’s superior resale or trade-in value (even to competitors) many times makes this easier too.
Rinse and repeat for the Apple Watch, Apple Watch Bands, Apple AirPods and AirPods Pro (remember how people said THOSE looked goofy, no one would want to be seen wearing them, where’s my beloved headphone Jack and tangled wire cheap or expensive earbuds and headphones!!), iMac, Mac Studio, Mac Pro, iPad Air, iPad Pro, heck even iPhone SE and base iPad?
There’s always someone complaining about Apple’s prices, claiming Apple should price lower so THEY can afford it or they can get almost the same for a lot cheaper and in the same breath claim that Apple would benefit from selling more volume at cheaper prices. And it’s always been so. Yet Apple has backed their “prices” or rather customer value with high technology, superior and widely copied design, generally superior support, highly refined USER EXPERIENCE at all levels, from marketing, packaging, buying experience, user experience, after sales OS support, resale value, and integrated ecosystem management. For all of this, Apple asks a premium cost, and in exchange, generally delivers a premium value which apparently, over 1.4 billions active users and over 2 billion active device install base attests to. Apparently Apple is doing something right.
Contrast this to competitors like Android makers, especially Samsung, Chinese makers, etc. who have diminishing and shrinking high end premium sales even as they try like crazy to penetrate that level and and sell “upper end” models like Foldables, Ultras, etc. They sell some, make some bucks, but ALWAYS, ALWAYS, their revenues and profits severely lag behind Apple’s, both individually and collectively. Why? Because they dilute their brands with mid level and inexpensive (read cheap) products, undermine and undercut ANY pricing power they have by starting to discount even their flagship models within 2 months of introduction, sometimes earlier because of flagging sales, and all those bundles, BOGOs, promotional discounts eat directly into margins, revenues, and profits. If they built their premium products to a price point with a reasonably healthy margin, why do they undermine themselves? Because the dirty little secret is Apple’s competitors’ buying demographic is just not able or interested in spending more outside of first adopters. “Wait for the sale”, “I’ll switch to a different maker”, and “oh, it’s no better than what I have” dominates their thinking and mantras. And in tougher macroeconomic times like the last 3 years, buyers pulled back and it’s reflected in the overall smartphone and electronics marketplace shrinkage that’s hit Samsung and all of Apple’s competitors hard. Apple, by contrast, has fallen much less than everyone else, and in a few cases and quarters, has done better, sometimes far better YOY and compared to their competitors.
And so we have the Vision Pro and an extraordinary price ask, presumably set to preserve Apple’s 35-40% hardware gross margin, in this case likely 27-32% gross margin on a first generation product like this. That will make it exclusive, definitely lower volume, and very high powered and loaded with a cutting edge amount of well integrated technology, to be highly desired by some. Some, not all. Even maybe a select few (hundred thousand). I’m sure Apple gauged the interest, ran the numbers, and decided a new technology, concept, and execution done in an (ultra) premium manner, with premium solutions and support, needed to have an (ultra) premium price. Decide for yourself if Apple will be right over the short and long run.
But IMO, I wouldn’t be betting against Apple. People have been telling Apple for a long, long, long time about how Apple “should” be running their business, how Apple is pricing themselves out of sales, and how Apple could do things better if they just listened to all of the criticisms and business model conventions. And those people will continue to be wrong, continue to scratch their heads, and wonder how could a company doing so many things wrong be making annually almost $400 billion revenue, sell 225+ million iPhones, capture 50-60% of ALL smartphone revenues, 85% of all smartphone profits (without any Foldable models), have $100B in free cash flow, and again, have a $3 TRILLION market cap?
if Apple is so “wrong”, I don’t want to be the complainer’s “right”.
It is at the core a stereoscopic view of that same Mac desktop scene then improved significantly to make that something valuable to interact with.