How virtual and augmented reality could play a role in Apple's future
Apple CEO Tim Cook suggests that the company is investigating the possibilities of virtual reality and augmented reality. Here's an explanation of what the technology is, how Apple could adopt it, and how it could be practically implemented in a user's daily life.
Video gaming and education has depended for many decades now on computer simulations of the real world. Physics engines for engineers visualizing and testing products are both billion-dollar industries. Similar technologies are utilized in the gaming market, which is making more money that the entire movie industry and has for several years.
Virtual reality promises users a more immersive experience, by sinking a user in the environment with video and audio cues. Instead of a large screen presenting a singular vision of a scene, a user of VR can look around at the scene, and interact with the environment, to the limits of the programmed scenario.
Apple CEO Tim Cook has made remarks, most recently on Sunday, heralding the technology as one of the cornerstones of the company's future.
While similar, any discussion of the future of Virtual reality (VR) or augmented reality (AR) needs to have the differences between the two spelled out. Both utilize similar core technologies, but have different implementations.
Virtual reality
For about three decades, the term "virtual reality" has been used as a catch-all for any sort of real-world simulation or modification. Initial implementations were used by the entertainment industry for relatively compact rides in amusement parks, with additional user by the military for flight simulator training.
Both early uses employed conventional displays in conjunction with a series of hydraulics and other mechanical haptics to provide full feedback of the environment to the user. The most famous VR entertaiment venue was the BattleTech Center which launched in 1990, and have yet to fully close.
Military simulators often lagged behind the technology of "civilian" ones. However, one military flight simulator in use in the nineties would keep a pilot trainee upside-down in the simulator while the plane being flown was in inverted flight.
Later, VR headsets were introduced, and were initially a massive edifice of plastic and silicon strapped to a wearer's head, with the user tethered to a bank of computers supplying all audio and visual input to the wearer.
Augmented reality
Since the advent of VR, the immense hardware requirements over time have become apparent. What is less difficult to manipulate and generate are overlays over the existing environment. Augmented reality can be used to guide a user on a street, or highlight businesses as a user travels through a town. Another possible implementation is capture of virtual creatures with the flick of a finger, after spotting them in the park on a daily walk.
Google's currently shelved Glass project is an example of augmented reality, with the user getting an information overlay in one eye. Several copycats of Google Glass have arisen, but none have met with any commercial success.
Google Cardboard is a simple frame for insertion of a smartphone to use as a rudimentary AR or VR system. The technology is being used in schools all over the world, and the company offers virtual field trips to remote and culturally significant areas to expand the horizons of students that are geographically or financially bound.
The Microsoft HoloLens is a full-face AR system, tethered to the Windows ecosystem. Immensely popular mobile title Pokemon Go is also considered augmented reality, with the user aiming a phone to "spot" a virtual creature in the real world.
Enter the Oculus Rift
Oculus, the parent company behind the most public VR effort yet, was formed in 2012. Two months later, crowdfunding campaign launched which would garner $2.5 million dollars. The company promised a practical, light-weight, and easy to use solution for the consumer to experience full virtual reality in the home.
Ultimately, two developer models were released, with the development process hampered somewhat by the shift in available components. Facebook purchased Oculus in 2014, and it was thought that a commercial version would soon follow.
In the beginning of 2016, the first consumer model of the Oculus Rift shipped. It has two OLED displays running at 1080x1200, one for each eye. Positional input is provided through an infrared LED, with user controls executed by an Xbox One controller.
The Oculus Rift isn't a standalone device. It relies on a fairly powerful gaming PC, with a GPU that has more heft than anything currently shipping from Apple. Support was originally promised for macOS, but it has not been updated in well over a year, and has been "paused."
"That is up to Apple and if they ever release a good computer we will do it," Oculus founder Palmer Luckey said in January. "It just boils down to the fact that Apple doesn't prioritize high-end GPUs. You can buy a $6,000 Mac Pro with the top of the line AMD FirePro D700, and it still doesn't match our recommended spec."
SteamVR
A competitor to the Oculus Rift is the HTC Vive. The Vive was revealed in 2015 and released in April 2016, and has been developed by HTC and game developer Valve. Players uses hand held controllers to map a player's motion into the virtual world, with the headset and associated tether allowing the user to navigate freely in a hopefully clear play area.
The HTC Vive's required computer specifications are similar to that of the Oculus Rift. While the processor requirement is easily met, the GPU is an issue for OS X users -- at least for now.
The main difference between the HTC Vive and the Oculus Rift is the SDK. The Oculus Rift uses a closed SDK, while the HTC Vive's key motion sensing technologies were made open source in the beginning of August.
Tim Cook, AR, and VR
In 2016, Apple CEO Tim Cook has made some remarks about the company and potential VR and AR implementations. During Apple's January's quarterly report question and answer session, Cook said that VR was "cool" and noted that he didn't think it was a niche, but at the time he had little to say about its future at Apple.
During July's question and answer session, Cook had a bit more to say. "AR can be really great," Cook said. "We have been and continue to invest a lot in this. We are high on AR for the long run, we think there's great things for customers and a great commercial opportunity. So we're investing."
On Sunday, Cook said to the Washington Post that "I think AR is extremely interesting and sort of a core technology," adding that "it's something we're doing a lot of things on behind that curtain that we talked about."
VR and AR are open fields
There is no clear leader, or de facto standard behind AR or VR yet. The Oculus Rift led the field, but the HTC Vive is likely to have more third party support with the opening of hardware and software for the device, plus tight integration with the popular Steam digital distribution platform.
Apple's iOS devices can be used at the core of rudimentary AR and VR now. The Mattel ViewMaster VR is an outgrowth of Google Cardboard, and uses software to resurrect the 50-year old toy, that once used stereographic slides. With insertion of a four-inch smartphone or larger, an app now gives the same functionality as the aged ViewMaster in a more effective way for a modern audience.
Modern VR and AR technologies aren't perfect. The major source of user physical illness in the technology is illness induced by input lag.
When the displayed images don't respond immediately to user input, or a head rotation, the human body's vestibular system is confused by the input lag, causing the nausea and light-headedness associated with motion sickness.
Input lag affects people differently. If one can read in a moving car without ill-effect, then in all likelihood a VR headset won't pose a problem. However, people prone to seasickness or motion sickness nearly always suffer from bad VR implementations.
On Google Play's ViewMaster VR page, it lists hardware that is compatible with the toy. Motorola and Samsung phones are listed, as well as every iPhone newer than the iPhone 5 and 5c. It also notes that the ViewMaster VR "may also work with newer model Android smartphones not listed above" but no guarantees are made, because of the wide disparity of device capabilities out of the control of Google.
Apple's tight integration of software and hardware down to the iPhone's casing size can do a great deal to eliminate problems inherent with both AR and VR technology. Input lag can be minimized by leveraging Apple's strict control over the sensors used in a device, as well as managing the communication between the sensors and development SDKs -- much like Xcode does now for iOS.
Much of the work that Apple needs to do is simply refinement of existing technologies. If Apple should utilize the open source nature of the HTC Vive for positional tracking in a future full-VR implementation, both the Apple VR and Windows-based VR ecosystem can flourish.
While Apple was the first to market with a PC, it didn't set the standard -- IBM did that in 1981. Apple wasn't the first to release a MP3 player, but it did it better, and won the market in the end. Samsung released its smart watch a year before the Apple Watch came out, and in every regard, the Apple Watch is the superior product, with Samsung floundering with multiple models and operating systems.
VR and AR stand now where wearables did just a few years ago, and only have advanced lately after decades of work on computer and imaging technology making it practical and reasonably affordable to have a setup in the home. Apple can seize the day like it did with the iPod and iPad.
Based on remarks from Tim Cook, Apple appears to be looking at the possibilities.
Video gaming and education has depended for many decades now on computer simulations of the real world. Physics engines for engineers visualizing and testing products are both billion-dollar industries. Similar technologies are utilized in the gaming market, which is making more money that the entire movie industry and has for several years.
Virtual reality promises users a more immersive experience, by sinking a user in the environment with video and audio cues. Instead of a large screen presenting a singular vision of a scene, a user of VR can look around at the scene, and interact with the environment, to the limits of the programmed scenario.
Apple CEO Tim Cook has made remarks, most recently on Sunday, heralding the technology as one of the cornerstones of the company's future.
While similar, any discussion of the future of Virtual reality (VR) or augmented reality (AR) needs to have the differences between the two spelled out. Both utilize similar core technologies, but have different implementations.
Virtual reality
For about three decades, the term "virtual reality" has been used as a catch-all for any sort of real-world simulation or modification. Initial implementations were used by the entertainment industry for relatively compact rides in amusement parks, with additional user by the military for flight simulator training.
Both early uses employed conventional displays in conjunction with a series of hydraulics and other mechanical haptics to provide full feedback of the environment to the user. The most famous VR entertaiment venue was the BattleTech Center which launched in 1990, and have yet to fully close.
Military simulators often lagged behind the technology of "civilian" ones. However, one military flight simulator in use in the nineties would keep a pilot trainee upside-down in the simulator while the plane being flown was in inverted flight.
Later, VR headsets were introduced, and were initially a massive edifice of plastic and silicon strapped to a wearer's head, with the user tethered to a bank of computers supplying all audio and visual input to the wearer.
Augmented reality
Since the advent of VR, the immense hardware requirements over time have become apparent. What is less difficult to manipulate and generate are overlays over the existing environment. Augmented reality can be used to guide a user on a street, or highlight businesses as a user travels through a town. Another possible implementation is capture of virtual creatures with the flick of a finger, after spotting them in the park on a daily walk.
Google's currently shelved Glass project is an example of augmented reality, with the user getting an information overlay in one eye. Several copycats of Google Glass have arisen, but none have met with any commercial success.
Google Cardboard is a simple frame for insertion of a smartphone to use as a rudimentary AR or VR system. The technology is being used in schools all over the world, and the company offers virtual field trips to remote and culturally significant areas to expand the horizons of students that are geographically or financially bound.
The Microsoft HoloLens is a full-face AR system, tethered to the Windows ecosystem. Immensely popular mobile title Pokemon Go is also considered augmented reality, with the user aiming a phone to "spot" a virtual creature in the real world.
Enter the Oculus Rift
Oculus, the parent company behind the most public VR effort yet, was formed in 2012. Two months later, crowdfunding campaign launched which would garner $2.5 million dollars. The company promised a practical, light-weight, and easy to use solution for the consumer to experience full virtual reality in the home.
Ultimately, two developer models were released, with the development process hampered somewhat by the shift in available components. Facebook purchased Oculus in 2014, and it was thought that a commercial version would soon follow.
In the beginning of 2016, the first consumer model of the Oculus Rift shipped. It has two OLED displays running at 1080x1200, one for each eye. Positional input is provided through an infrared LED, with user controls executed by an Xbox One controller.
The Oculus Rift isn't a standalone device. It relies on a fairly powerful gaming PC, with a GPU that has more heft than anything currently shipping from Apple. Support was originally promised for macOS, but it has not been updated in well over a year, and has been "paused."
"That is up to Apple and if they ever release a good computer we will do it," Oculus founder Palmer Luckey said in January. "It just boils down to the fact that Apple doesn't prioritize high-end GPUs. You can buy a $6,000 Mac Pro with the top of the line AMD FirePro D700, and it still doesn't match our recommended spec."
SteamVR
A competitor to the Oculus Rift is the HTC Vive. The Vive was revealed in 2015 and released in April 2016, and has been developed by HTC and game developer Valve. Players uses hand held controllers to map a player's motion into the virtual world, with the headset and associated tether allowing the user to navigate freely in a hopefully clear play area.
The HTC Vive's required computer specifications are similar to that of the Oculus Rift. While the processor requirement is easily met, the GPU is an issue for OS X users -- at least for now.
The main difference between the HTC Vive and the Oculus Rift is the SDK. The Oculus Rift uses a closed SDK, while the HTC Vive's key motion sensing technologies were made open source in the beginning of August.
Tim Cook, AR, and VR
In 2016, Apple CEO Tim Cook has made some remarks about the company and potential VR and AR implementations. During Apple's January's quarterly report question and answer session, Cook said that VR was "cool" and noted that he didn't think it was a niche, but at the time he had little to say about its future at Apple.
During July's question and answer session, Cook had a bit more to say. "AR can be really great," Cook said. "We have been and continue to invest a lot in this. We are high on AR for the long run, we think there's great things for customers and a great commercial opportunity. So we're investing."
On Sunday, Cook said to the Washington Post that "I think AR is extremely interesting and sort of a core technology," adding that "it's something we're doing a lot of things on behind that curtain that we talked about."
VR and AR are open fields
There is no clear leader, or de facto standard behind AR or VR yet. The Oculus Rift led the field, but the HTC Vive is likely to have more third party support with the opening of hardware and software for the device, plus tight integration with the popular Steam digital distribution platform.
Apple's iOS devices can be used at the core of rudimentary AR and VR now. The Mattel ViewMaster VR is an outgrowth of Google Cardboard, and uses software to resurrect the 50-year old toy, that once used stereographic slides. With insertion of a four-inch smartphone or larger, an app now gives the same functionality as the aged ViewMaster in a more effective way for a modern audience.
Modern VR and AR technologies aren't perfect. The major source of user physical illness in the technology is illness induced by input lag.
When the displayed images don't respond immediately to user input, or a head rotation, the human body's vestibular system is confused by the input lag, causing the nausea and light-headedness associated with motion sickness.
Input lag affects people differently. If one can read in a moving car without ill-effect, then in all likelihood a VR headset won't pose a problem. However, people prone to seasickness or motion sickness nearly always suffer from bad VR implementations.
On Google Play's ViewMaster VR page, it lists hardware that is compatible with the toy. Motorola and Samsung phones are listed, as well as every iPhone newer than the iPhone 5 and 5c. It also notes that the ViewMaster VR "may also work with newer model Android smartphones not listed above" but no guarantees are made, because of the wide disparity of device capabilities out of the control of Google.
Apple's tight integration of software and hardware down to the iPhone's casing size can do a great deal to eliminate problems inherent with both AR and VR technology. Input lag can be minimized by leveraging Apple's strict control over the sensors used in a device, as well as managing the communication between the sensors and development SDKs -- much like Xcode does now for iOS.
Much of the work that Apple needs to do is simply refinement of existing technologies. If Apple should utilize the open source nature of the HTC Vive for positional tracking in a future full-VR implementation, both the Apple VR and Windows-based VR ecosystem can flourish.
While Apple was the first to market with a PC, it didn't set the standard -- IBM did that in 1981. Apple wasn't the first to release a MP3 player, but it did it better, and won the market in the end. Samsung released its smart watch a year before the Apple Watch came out, and in every regard, the Apple Watch is the superior product, with Samsung floundering with multiple models and operating systems.
VR and AR stand now where wearables did just a few years ago, and only have advanced lately after decades of work on computer and imaging technology making it practical and reasonably affordable to have a setup in the home. Apple can seize the day like it did with the iPod and iPad.
Based on remarks from Tim Cook, Apple appears to be looking at the possibilities.
Comments
Back in the nineties there was an Amiga 3000 powered system called Virtuality and it could run a car simulator, a flight simulator (both required you to sit in a chair), or a virtual world simulator that you wore gloves and walked on a treadmill. In other words Virtuality did MORE than Oculus Rift with less specs.
Programmers today are so lazy.
That said, Google Glass got a lot of people on the defensive simply because of the camera.
Remove the camera and a device like the glass can be your personal HUD but you need to be able to turn it off (and on) with a simple movement of the eye.
distraction is going to be a problem with some people until they can train themselves to see through the display yet be able to comprehend what the display is telling them.
However and from my POV (I have written code for both Civil and military Flight Simulators in the past) this sort of wearable tech is nowhere near ready for mainstream use yet. In 5-10 years maybe.
I am sure some less forward thinking states will already be looking at banning the use of this sort of thing in Vehicles until it can be proven to be safe. I'd rather see Cars built with HUD's projecting data onto the windscreen especially speed.
It has to be built-in to core aspects of life for it to take off. Car windshields would be a start. There is no limit to quantity and quality of the AR experience that could be had from inside a car. All sorts of signs, billboards, etc. Customizable. Opt in or Out. Endless.
There are ways currently to imbed prisms into glasses to display a small field of view (FOV) display into glasses. They are way expensive and would still require something to generate the graphics so a long chord or heavy battery for wireless. Both not the best solution.
I'd like to clarify a few points of yours because I do agree with the article that Apple only needs to refine existing tech. It's all about what Apple intends to do with it.
1) AR/VR isn't too computationally expensive anymore. Sure, to run a PC game it would be, but the iPhone itself is already running 1st person shooters and only the addition of a tracking system would be necessary and that could be handled by a dedicated processor.
2) I can't understand what you meant by your point #1. Can you clarify?
3) Your point #2 is a clarification of the article and that's a good point to bring up. Much of the literature on simulator sickness has concluded that it stems from perception/vestibular miss-match. Things such as high visual flow and tracking system lag severely generate sickness. For most people and applications, these can be minimized through dedicated hardware and a static focal length (cars have people staring at the road at a fairly far distance). The eye focal length issue is just something that can't really be perfectly fixed with any approach that I know of, just hacked (which is kinda what VR/AR is anyway).
4) I'm not sure about your point #3. You are correct about field of view and resolution, but I'm not sure what your point is, in discussing them at the same time. For imbedded imagery, like a car windshield, neither shouldn't be a problem.
Like all things Apple does. It will target something specific, with just the right tech. If it doesn't introduce something that does everything you can imagine, it's probably because the technology does not yet exist. It should be fun to see what they create as I would really like to see our world with embedded imagery and interactive data. The good and useful type, not advertisements for apps...
I've owned and used 2 sets of early glasses-mounted displays from Vuzix. While the display was good, the weight became awkward over time. If Apple were able to release a form factor that felt like wearing normal sunglasses -- but actually had full displays in each lense -- that could be compelling.
we are agree more than you think ;-). a bit old paper... need update.
http://www.owlnet.rice.edu/~kevinaboos/docs/flashback_mobisys2016.pdf
It's not actually shelved. It's now being further developed and marketed for business/occupational use. I doubt there will be a consumer version for a long time, if ever, but yes Google Glass Enterprise Edition is active.
This is not intended to distract from the other very informative parts of the AI article.
Most games from the 90s will run around 1000FPS or more. This can even be seen in mobile:
A game that ran at 30FPS on the first iPhone could run at >2500FPS on the iPhone 6. Developers just keep boosting the graphics quality when there's more power available. VR years ago looked like this:
but now it can look like this:
The requirements for VR entirely depend on what you want to see. With it being called Virtual Reality, it implies the goal is to reproduce reality but reality is by no means the minimum requirement and there isn't really a minimum requirement. 360 degree videos are VR and they'd play on almost anything.