Editorial: Apple's next hardware play could be in game controllers
Apple is thought to be working on Augmented Reality glasses that are perhaps two years out. But in moving the AR experience from the handheld screens of iOS devices to a fixed panel you look through, users will need a new way to navigate AR worlds. Little attention has been given to the role of game controllers in AR navigation, but Apple's patent filings and its recent announcements suggest that its next new play in hardware could be an advanced gaming controller, perhaps even a wearable one.
Oculus developed wireless controllers for Rift VR, but Apple could sell advanced controllers even before bringing AR glasses to market
Apple has an installed base of 1.5 billion devices globally, most of them iPhones that are always wirelessly connected to a mobile data network, and they all support Bluetooth connectivity. Apple has previously floated plans for Made For iOS (MFI) game controllers, but the types of games that are popular on iOS devices and Macs have somewhat limited the demand for third-party game controllers.
However, a few things are changing. First, Apple launched gaming on its Apple TV box two generations ago. And it has increasingly made games a bigger part of its Mac and iOS App Stores. And this year, it unveiled Apple Arcade as new gaming subscription to monetize the development of better games across its platforms-- games that are not trying to support themselves with annoying ads or In-App Purchases.
Apple Arcade not only hints at a new Apple TV, but also a new controller optimized for cross platform play
On the hardware side, back in 2013, Apple acquired PrimeSense, the motion-based gaming camera maker that powered Microsoft's original Xbox Kinect. Apple turned the technology into TrueDepth for iPhone X, launching new vocabulary into the world including Animoji and Memoji, along with Portraits and Portrait Lighting. It also developed all kinds of AR features including face-based graphics augmentation that was put into use by Snapchat and other third-party developers to turn participants into graphically enhanced avatars. Yet the original intent of PrimeSense involved motion-based gaming, something that Apple hasn't yet turned into a product.
Additionally, the advanced custom silicon powering Apple's mobile and home devices has made incredible leaps over just the past four years, accelerated by its own development of the new custom Apple GPU, as well as the maturing of its Metal framework for optimizing the performance and blurring the differences between its hardware platforms.
Finally, this summer Apple announced that it would be adding support in its Game Controller framework for Sony's Playstation 4 Dual Shock and Microsoft's Xbox One S and Elite 2 controllers. That required work to abstract the differences between the two standards and Apple's own MFi design so that existing games would work seamlessly with any of these third-party controllers.
Apple announced support for other game controllers iOS 13 and tvOS, but it can also make its own
What's stopping Apple from developing a game controller? In the past, the company has delegated this task to third parties-- perhaps because it thought that the market was too small to be concerned with. When it shipped the first tvOS-based Apple TV in 2015, it paired it with the Jony Ive-designed Siri Remote, effectively a simplified TV remote designed to act as a basic game controller. While the Siri Remote does okay as a playback controller, it is a terrible game controller.
But the elegantly bespoke thinness of Ive's Siri Remote isn't the only work Apple has put into input controllers. In a recent patent filing the company detailed the concepts behind a wearable interaction tool in the form of a "finger-mounted device with sensors and haptics."
An Apple patent application image showing potential mounting points for finger sensor units
Add up all of the above changes, advancements, and research and it creates the picture that Apple appears likely to enter the game controller business. In part, to sell new hardware to its Apple TV and iOS buyers, but also because it now has advanced technology capable of significantly improving the game controller as a product: the custom silicon driving proprietary advances of AirPods and Beats wireless headphones.
Imagine getting a set of "AirPads" which allow you to play games on Apple TV, your Mac, and your iOS devices, but which can also interact with AirPods to play audio-based AR games. You could virtually fight your friends or other players "Mortal Combat" style, with nothing more than your handheld or wearable AirPad controllers. Or you could challenge other players to a "DDR" dance-off, or something related to "Guitar Hero," "Tetris," "Jenga," "Angry Birds," "Pokemon," or any number of other arcade titles designed to make use of wireless mobile game controllers.
The next step in the future could be adding AR glasses for visuals that make your wireless gaming even more engaging. But Apple is still working on the building blocks of ARKit and the tools to make the development of AR worlds easy, such as its new Reality Composer tool. Game controllers already have immediate use and market, and the content for using them already exists in the App Store.
Apple is still building the foundation of AR tech and content, but gaming controllers already have a use and apps to use them with
These silicon and software advances that Apple made to the existing category of wireless headphones turned AirPods into a huge hit, one that is still raging years later. Competitors have been unable to successfully copy Apple's success, despite nearly everyone trying to do so. Many analysts kept telling us AirPods weren't going to "move the needle" until it was obvious that Wearables were indeed becoming a huge business segment for Apple, one that is growing rapidly.
With a wireless gaming controller taking advantage of the same or similar custom Apple silicon, the company could create another hit with an advanced controller that acts as a unique "anywhere" gaming system on its own, paired with a Watch or iPhone and linked up with a pair of AirPods. Apple could gamify exercise and fitness; create server-based multiplayer gaming for physical challenges such as virtual sword fights, capoeira, laser tag, or other exercises; and enable new ways to interact at meetings, presentations, and conferences.
It's not hard to imagine that the company that introduced and popularized the mouse, the trackpad, the touchwheel, and multitouch gestures could introduce a versatile new wireless gaming controller input device that would work everywhere from legacy game consoles to modern iOS devices to future AR systems. And now that Ive is no longer calling all the shots at Apple, such a game controller doesn't have to look like a super-thin rectangle with rounded corners.
Apple's pioneering support for FireWire and USB and its acquisition of what would become iTunes foreshadowed its venture into iPods, something that nobody in the Apple universe anticipated, even though the market for MP3 players was not brand new and Apple had been making the "rip, mix and burn" of music an increasingly important element on its Macs.
Years later, while the media imagined that the future would be played out by mobile phones playing MP3s-- largely because this was the biggest threat they could imagine for Apple's iPod business--Apple instead aimed higher to deliver desktop-class computing on its new mobile iPhone, tethered to a data plan. Apple had developed a series of things to make this possible, ranging from mobile hardware savvy with iPods to building its own Safari web browser and sophisticated support for device data sync and digital software app sales. Yet months before iPhone was released, many still thought Apple would just license Symbian from Nokia or repeat its effort to add iTunes to a Motorola handset.
Back in 2006 I actually imagined Apple could build a full screen, touch iPod Pro in two models, one oriented to business users with messaging, email, a browser, device-synced contacts and calendar apps, and another designed for consumers that would let them take photos, videos, and voice clips and blog their experiences or listen to music and show off their photo albums. As it turned out, Apple made both devices in one, opening up the world to Instagram, iMessage, Safari, Keynote and Angry Birds on a handheld device.
Imagine an iPod playing media and running apps. Apple delivered it.
Around 2008, at a time when many thought the future of PCs were represented by cheap netbooks, Apple introduced iPad, a new tablet based on multitouch and aiming at simplicity, not just cheapness and miniaturization of yesterday's PCs. Apple had already done the work to deliver such a device, but had to wait until the market for mobile tablet devices was opened up by sales of iPhones subsidized by mobile data plans. There had been many stabs at tablet PCs by Microsoft, Samsung, and others, but the iPad arrived and took over the entire market because Apple spent time thinking about how to launch it correctly.
Conversely, while HTC Vive, Samsung Gear VR, Facebook Oculus, Google Glass and Cardboard, and Microsoft Hololens all attempted to deliver the far-off future of VR headsets, Apple instead aimed at what was really deliverable, focusing its efforts on AirPods to deliver always-there audio AR instead. This far more practical approach has turned into a multi-billion dollar business for Apple while the rest of the industry diddled way billions to deliver something nobody could use or afford, and for which there wasn't any compelling content.
Across 2016, pundits demanded to know where Apple "VR" product was, after earlier demanding a Mac netbook and a Curved or 3D iPhone.
Similarly, while Amazon, Google, Microsoft, and Samsung have worked to "outdo" Siri with expansive voice-first platforms of their own, and have tried to give away WiFi mics as a way to build a voice platform among affluent users, Apple has instead simply made Siri a feature across all of its hardware, from Macs to Apple Watch to iOS devices and Apple TV. Apple now has, far and away, the world's largest voice assistant platform, spread across the most languages and countries, and focused on valuable, affluent buyers of premium hardware.
Apple now offers HomePod as a premium home speaker with Siri features and HomeKit integration. Media critics and analysts have tried desperately to portray it as a failure, despite it selling in the millions at a premium price point and earning profits that no other voice speakers appear capable of making. Apple's HomePod is already doing business in the magnitude of Sonos, establishing itself as a significant player in the home as a base for HomeKit accessories and streaming integration with Apple Music and iOS devices.
This again suggests that Apple could broaden its move into game controllers with a stand-out new model, paired with advanced silicon that makes it more reliable, easy to pair, automatic to use between your devices, and compelling for third parties to build gaming experiences around. It could even make use of wireless charging like Apple Watch and the latest generation of AirPods, meaning it needs no exposed ports at all. It could be gloves, a handheld device or some other form of wearable.
Apple is serious about gaming software, and you don't have to look any farther than Apple Arcade and ARKit to see that. It should really think about making game controller hardware.
Oculus developed wireless controllers for Rift VR, but Apple could sell advanced controllers even before bringing AR glasses to market
AirPads for wireless kinetic gameplay anywhere
Image a wireless gaming controller designed to use motion gestures, buttons, directional joysticks, touch, and haptic feedback. You don't have to imagine, because Microsoft, Nintendo, and Sony have been working to perfect the versatile home console gaming controller for decades. However, these three companies have only tied their controllers to their own, latest generation consoles. Each generation of those consoles has typically sold in the ballpark of around 100 million units. And only recently have these companies embraced Bluetooth for their wireless connectivity.Apple has an installed base of 1.5 billion devices globally, most of them iPhones that are always wirelessly connected to a mobile data network, and they all support Bluetooth connectivity. Apple has previously floated plans for Made For iOS (MFI) game controllers, but the types of games that are popular on iOS devices and Macs have somewhat limited the demand for third-party game controllers.
However, a few things are changing. First, Apple launched gaming on its Apple TV box two generations ago. And it has increasingly made games a bigger part of its Mac and iOS App Stores. And this year, it unveiled Apple Arcade as new gaming subscription to monetize the development of better games across its platforms-- games that are not trying to support themselves with annoying ads or In-App Purchases.
Apple Arcade not only hints at a new Apple TV, but also a new controller optimized for cross platform play
On the hardware side, back in 2013, Apple acquired PrimeSense, the motion-based gaming camera maker that powered Microsoft's original Xbox Kinect. Apple turned the technology into TrueDepth for iPhone X, launching new vocabulary into the world including Animoji and Memoji, along with Portraits and Portrait Lighting. It also developed all kinds of AR features including face-based graphics augmentation that was put into use by Snapchat and other third-party developers to turn participants into graphically enhanced avatars. Yet the original intent of PrimeSense involved motion-based gaming, something that Apple hasn't yet turned into a product.
Additionally, the advanced custom silicon powering Apple's mobile and home devices has made incredible leaps over just the past four years, accelerated by its own development of the new custom Apple GPU, as well as the maturing of its Metal framework for optimizing the performance and blurring the differences between its hardware platforms.
Finally, this summer Apple announced that it would be adding support in its Game Controller framework for Sony's Playstation 4 Dual Shock and Microsoft's Xbox One S and Elite 2 controllers. That required work to abstract the differences between the two standards and Apple's own MFi design so that existing games would work seamlessly with any of these third-party controllers.
Apple announced support for other game controllers iOS 13 and tvOS, but it can also make its own
What's stopping Apple from developing a game controller? In the past, the company has delegated this task to third parties-- perhaps because it thought that the market was too small to be concerned with. When it shipped the first tvOS-based Apple TV in 2015, it paired it with the Jony Ive-designed Siri Remote, effectively a simplified TV remote designed to act as a basic game controller. While the Siri Remote does okay as a playback controller, it is a terrible game controller.
But the elegantly bespoke thinness of Ive's Siri Remote isn't the only work Apple has put into input controllers. In a recent patent filing the company detailed the concepts behind a wearable interaction tool in the form of a "finger-mounted device with sensors and haptics."
An Apple patent application image showing potential mounting points for finger sensor units
Add up all of the above changes, advancements, and research and it creates the picture that Apple appears likely to enter the game controller business. In part, to sell new hardware to its Apple TV and iOS buyers, but also because it now has advanced technology capable of significantly improving the game controller as a product: the custom silicon driving proprietary advances of AirPods and Beats wireless headphones.
Imagine getting a set of "AirPads" which allow you to play games on Apple TV, your Mac, and your iOS devices, but which can also interact with AirPods to play audio-based AR games. You could virtually fight your friends or other players "Mortal Combat" style, with nothing more than your handheld or wearable AirPad controllers. Or you could challenge other players to a "DDR" dance-off, or something related to "Guitar Hero," "Tetris," "Jenga," "Angry Birds," "Pokemon," or any number of other arcade titles designed to make use of wireless mobile game controllers.
The next step in the future could be adding AR glasses for visuals that make your wireless gaming even more engaging. But Apple is still working on the building blocks of ARKit and the tools to make the development of AR worlds easy, such as its new Reality Composer tool. Game controllers already have immediate use and market, and the content for using them already exists in the App Store.
Apple is still building the foundation of AR tech and content, but gaming controllers already have a use and apps to use them with
Setting up a future launch
This strategy would have a lot in common with AirPods. Apple embraced Bluetooth and BLE early on, giving it head start over Android and Windows Phone. Apple then acquired Beats, giving it a huge retail headphones business. It subsequently launched its own AirPods, leveraging the custom W1 silicon it developed to erase some of the pain-points of existing wireless headphones, facilitating easy pairing, movement of headphones between devices, improving latency, enhancing sound quality, and boosting signal resiliency.These silicon and software advances that Apple made to the existing category of wireless headphones turned AirPods into a huge hit, one that is still raging years later. Competitors have been unable to successfully copy Apple's success, despite nearly everyone trying to do so. Many analysts kept telling us AirPods weren't going to "move the needle" until it was obvious that Wearables were indeed becoming a huge business segment for Apple, one that is growing rapidly.
With a wireless gaming controller taking advantage of the same or similar custom Apple silicon, the company could create another hit with an advanced controller that acts as a unique "anywhere" gaming system on its own, paired with a Watch or iPhone and linked up with a pair of AirPods. Apple could gamify exercise and fitness; create server-based multiplayer gaming for physical challenges such as virtual sword fights, capoeira, laser tag, or other exercises; and enable new ways to interact at meetings, presentations, and conferences.
It's not hard to imagine that the company that introduced and popularized the mouse, the trackpad, the touchwheel, and multitouch gestures could introduce a versatile new wireless gaming controller input device that would work everywhere from legacy game consoles to modern iOS devices to future AR systems. And now that Ive is no longer calling all the shots at Apple, such a game controller doesn't have to look like a super-thin rectangle with rounded corners.
This all happened before
Apple has regularly lined up the prerequisites for a secret new product in public view. As detailed above, it could be doing this again with gaming controllers. But consider how many times it has done this in the past.Apple's pioneering support for FireWire and USB and its acquisition of what would become iTunes foreshadowed its venture into iPods, something that nobody in the Apple universe anticipated, even though the market for MP3 players was not brand new and Apple had been making the "rip, mix and burn" of music an increasingly important element on its Macs.
Years later, while the media imagined that the future would be played out by mobile phones playing MP3s-- largely because this was the biggest threat they could imagine for Apple's iPod business--Apple instead aimed higher to deliver desktop-class computing on its new mobile iPhone, tethered to a data plan. Apple had developed a series of things to make this possible, ranging from mobile hardware savvy with iPods to building its own Safari web browser and sophisticated support for device data sync and digital software app sales. Yet months before iPhone was released, many still thought Apple would just license Symbian from Nokia or repeat its effort to add iTunes to a Motorola handset.
Back in 2006 I actually imagined Apple could build a full screen, touch iPod Pro in two models, one oriented to business users with messaging, email, a browser, device-synced contacts and calendar apps, and another designed for consumers that would let them take photos, videos, and voice clips and blog their experiences or listen to music and show off their photo albums. As it turned out, Apple made both devices in one, opening up the world to Instagram, iMessage, Safari, Keynote and Angry Birds on a handheld device.
Imagine an iPod playing media and running apps. Apple delivered it.
Around 2008, at a time when many thought the future of PCs were represented by cheap netbooks, Apple introduced iPad, a new tablet based on multitouch and aiming at simplicity, not just cheapness and miniaturization of yesterday's PCs. Apple had already done the work to deliver such a device, but had to wait until the market for mobile tablet devices was opened up by sales of iPhones subsidized by mobile data plans. There had been many stabs at tablet PCs by Microsoft, Samsung, and others, but the iPad arrived and took over the entire market because Apple spent time thinking about how to launch it correctly.
Conversely, while HTC Vive, Samsung Gear VR, Facebook Oculus, Google Glass and Cardboard, and Microsoft Hololens all attempted to deliver the far-off future of VR headsets, Apple instead aimed at what was really deliverable, focusing its efforts on AirPods to deliver always-there audio AR instead. This far more practical approach has turned into a multi-billion dollar business for Apple while the rest of the industry diddled way billions to deliver something nobody could use or afford, and for which there wasn't any compelling content.
Across 2016, pundits demanded to know where Apple "VR" product was, after earlier demanding a Mac netbook and a Curved or 3D iPhone.
Similarly, while Amazon, Google, Microsoft, and Samsung have worked to "outdo" Siri with expansive voice-first platforms of their own, and have tried to give away WiFi mics as a way to build a voice platform among affluent users, Apple has instead simply made Siri a feature across all of its hardware, from Macs to Apple Watch to iOS devices and Apple TV. Apple now has, far and away, the world's largest voice assistant platform, spread across the most languages and countries, and focused on valuable, affluent buyers of premium hardware.
Apple now offers HomePod as a premium home speaker with Siri features and HomeKit integration. Media critics and analysts have tried desperately to portray it as a failure, despite it selling in the millions at a premium price point and earning profits that no other voice speakers appear capable of making. Apple's HomePod is already doing business in the magnitude of Sonos, establishing itself as a significant player in the home as a base for HomeKit accessories and streaming integration with Apple Music and iOS devices.
This again suggests that Apple could broaden its move into game controllers with a stand-out new model, paired with advanced silicon that makes it more reliable, easy to pair, automatic to use between your devices, and compelling for third parties to build gaming experiences around. It could even make use of wireless charging like Apple Watch and the latest generation of AirPods, meaning it needs no exposed ports at all. It could be gloves, a handheld device or some other form of wearable.
Apple is serious about gaming software, and you don't have to look any farther than Apple Arcade and ARKit to see that. It should really think about making game controller hardware.
Comments
The idea is that Stereo AR will launch with basic 3rd party hardware and/or a Google Cardboard-like build it yourself kit. Developers will be able to use this to start making Stereo AR apps and games and users will be able to get started with it right away.
By the time Apple releases its hardware there will already be apps that support Stereo AR. This will allow the new hardware to hit the ground running.
Apple’s hardware will unlock new capabilities and take the experience to a much higher level.
The basis for this speculation is the fact that Apple already have Stereo AR running in non-public builds of iOS13.
The brand controller will allow the games to be played the same across all of Apple’s hardware, so you’d think they’d do it. Just like thinking that have to have a more powerful Apple TV (now rumored) to support Arcade games, the controller seems like a no brainer. Since they are support Sony and MS controllers, sounds like it is not coming for a long while, if at all.
There won’t be a controller for the AR glasses. At least, it will be like AirPods with an accelerometer and maybe the additional of capacitive surface or even buttons, plus Siri. At most, it will have the “Face ID” sensor cluster enabling users to user their hands and fingers to directly manipulate the virtual objects being displayed in the glasses. Maybe there will be a glove or fingernail tags to make easier to recognize hand positions, but that would be a fail in my and their opinion I bet.
For VR? A Face ID sensor cluster with maybe a controller, but I’m pretty negative on the consumer market for VR. The gloves are an artifact of history when visual object recognition was basically non-existent 20 years ago. The most natural is simply through bare hand gestures.
What's truly insecure -- coming to an Apple site, reading an Apple opinion columnist with a known beat, and then whining about it. Are you lost, bro?
I thought the point being made was quite clear -- Google, Facebook, and friends are doing their typical absurd moonshot projects of little actual value. Inversely, Apple set out to produce AirPods, which had very tangible value the moment they hit the market, but are arguably also laying the groundwork for a more-encompassing wearable platform (or at least use cases) in the future. Adding always-on Siri was another brick, as is the upcoming messaging upgrade to AirPods.
Seemed pretty clear, and pretty interesting to discuss.
Why? Doesn't the support for Sony and MS controllers already deliver this use case?
It's like claiming Apple needs to make its own printer if they're serious about being a desktop publishing platform. Well they did offer a printer, and then once third-party options became available they stopped. I don't see much difference here -- plenty of controllers available?
but this is another lamebrained idea, for Apple to have its own controllers. That idea is way too late. First you need a very popular game system for advanced gaming. Apple doesn’t really have that yet. Yes, I’ve got over 100 games on my iPad Pro 12.9”. I try almost every higher end new game that comes out, and some are pretty good. But—they could be great with a Microsoft or Sony controller fully supported.
honestly, do people really want to have to learn a totally new controller? The last time that happened, years ago, was with the Wii. But most of the sales of the Wii was to young kids, who bounce around even when they aren’t game playing. The Microsoft and Sony mostly work the same way, so going between them is pretty easy. It’s what people are now used to, and like, a lot. someday, if Apple and others come out with a really good VR system, which nobody really does have now, then a totally new controller for that will be required. But that’s not now, and definitely not useful for a phone, iPad or Mac.
It basically says Apple made money with AirPods, but others didn't make as much money with VR. But to make a tenuous connection, I'll call the AirPods Audio AR. In an article about AR/VR and gaming, are you seriously saying that makes any sense to you?
Throughout the last few decades, Apple has been quite a leading edge in the input devices/methods area. I won’t be surprise to see another leap forward on input methods (hence device) coming from Apple in the not too distance future.
Looking back at Wii, at X-Box Kinect, and a few other technologies as references in attempts to make input as nature as human instincts or realistic movements (e.g. tennis and golf games on Wii). Perhaps within Apple, there are dreamers who look at historical sci-fi movies/shows and attempt to make them a reality. Perhaps the next big thing is a glove-like input device which we can wear and use with nature gestures in the air instead of on a hard surface like screen of mobile devices or trackpad (e.g. not sure if I recollect the movie titles correctly - The Matrix or some futuristic show where the lead actor swipes in the air to search for surveillance archive or memory archive to look for clue).
Perhaps I am much a dreamer too. IMHO, I think it is highly possible but the timing isn’t quite right yet to release such a product. IMO, one key element missing in the analysis or anticipation of this article is how good Apple is at ‘timing’ (though they may not been always spot on but overall has been amazingly good timing).
AirPods (and its Beats equivalents) is just another one component (piece of a bigger picture) to form a really mass-market AR experience. For AR to become mainstream and meaningful in enhancing our daily lives, audio and visual are a big part of that journey. And to be mass market acceptance, the approach has to be as convenient, natural, comfortable.
IMHO, for any AR/VR to turn mainstream and mass market, all the components that make up the solution has to fall in place/line. Wireless earpiece is one of it as light-weight = able to wear for long duration, cordless = freedom of movement (eg I can even wear AirPods with head comfortable lying on a pillow, catching up some shows and fall asleep until next morning without feeling discomfort or injure my ears).
The missing pieces up next is a new way of visual (i.e. it’s damn tiring to hold an iPad Pro while lying in bed, with AirPods on, catching up on shows). Other than visual, it also needs new input method.
Perhaps I am dreaming up what the future I would like to solve some of the pain points for a part of my daily life. Isn’t that the DNA of Apple (at least in my own interpretation). Just as how Apple solves a few pain points I had with Touch ID iPhones while driving (eg before CarPlay was even available, it’s difficult to unlock iPhone esp with sweaty fingers and trying to get to Wife’s incoming message in a safe manner such as at traffic lights).
Just for illustration: It’s just not making sense for auto-makers to make the entertainment system’s output (aka screen) as an input method (using touch screen). As a driver without passenger, it’s almost useless to have that touch screen instead of just normal screen. Even using the buttons and knobs or touch pad in the center console area is a bit of distraction while driving.
Dreaming Mode On: What if the new input method can also be used with CarPlay in a vehicle, to simply swipe to navigate menus/actions and pinch to zoom in/out of maps, etc? Won’t that also form part of the bigger picture of Apple’s autonomous vehicle system currently in years of development?
Apple selling a branded controller in retail and shipping one with Apple TV pucks ensures control scheme compatibility across Apple’s hardware. That has to be a big plus for developers.
For Apple TV, I’ve already said I want a controller or remote with QWERTY capability.