Apple wants you to be able to just point at things to learn about them
Future Apple Watch or iPhone models could detect when you're pointing at a location, or want to buy something from a store window, and show you details about them.
Any Apple device with an accelerometer could determine whether you were pointing at something
Apple has applied for patents over shopping services before, with the idea of buyers being guided through product demonstrations online. However, one newly-revealed patent application wants to cut out that guidance and let you just point as you say "I want that one."
That's officially the point of this patent application, and it's even in the title, "Smart Shopping And Information Collection By Natural Gesture Using UWB." But patents always try to be as broad as possible, and this time it's the outer limits of the application's aims that are the most interesting.
For really the job of Apple's proposed system is to register that the user is pointing, then determine what they're pointing at. While most examples are to do with shopping, "the smart gesturing may be associated with indicating grocery store items in the grocery store," there's no reason it can only be used in this way.
The drawings including with the patent application, for instance, include one of an Apple Watch wearer pointing at a painting in a gallery. The system then offers to provide more details about the picture.
It isn't all about shopping
Or in another drawn example, a user is pointing at the Pyramids in Egypt. Again, an Apple Watch registers where they are pointing, and offers information.
"[The] electronic device may be a wearable device (e.g., a smart watch)," says the patent application, "though in other embodiments, the electronic device may include other wearable devices (e.g., a smart band, smart ring, or smart glasses) or the electronic device may include a non-wearable user device (e.g., a smartphone)."
So the process is the same regardless of the device used, and the end result is the delivery of at least some detail about what the user is pointing at. Even if that detail is a receipt for buying it.
To achieve all of this, and not achieve accidentally high bills, the device must be certain what a gesture is intended for, if indeed it's intended for anything.
"The smart gesture may be identified at least by the one or more motion sensors of the electronic device," says Apple. "The motion sensors (e.g., an accelerometer or a gyroscope) may identify the smart gesture by sensing a series of upward, downward, outward, inward, twisting, turning, and/or other similar motions indicative of a smart gesture (e.g., first position, second position, and third position)."
If the Apple Watch or other device decides that yes, this gesture is intentional and not just the user scratching their nose, then it has to work out what the intention is. Part of that is guided by where the user is, such as whether they are in a store, or whether they are at a tourist location.
There could then be some image recognition performed if the device is an Apple Vision Pro, but with the Apple Watch or really any other device, the clue is in the title of the patent application. It revolves around the use of Apple's UWB implementation, or Ultra Wide Band.
It is often about shopping, though.
Previous patent applications have seen this same idea applied to the Apple TV remote so that you can just wave it in the right direction.
But for most situations, Apple's proposal requires establishments to include UWB anchors. These are devices included in, for instance, shopping items or the Pyramids, which detects signals from nearby UWB devices.
As ever with a patent application, the descriptions are more about the how than the why. But once a device has been triggered by a UWB signal, that device can then do whatever it needs to fulfil the gesture that's been detected.
All of this means that potentially, our device can be used with real-world objects that send information. Or that sends a "Buy" link.
The target that user points at could be an enclosed system like a window shelf or a vending machine style of kiosk
Nonetheless, it is the process of pointing and identifying what is being pointed at that is the key part of this patent application. But alongside the many examples of smart shopping, there are some odd concerns -- it's like Apple is trying to solve problems we may not have.
Specifically, there are repeated descriptions of how a kiosk raises problems, "such as contact by and reuse of the device by multiple patrons (thus providing a transmission point of diseases that may be spread by contact), lines forming to use the device, crowd-building, or other undesirable conditions that may disrupt the smart shopping experience."
In Japan, it's commonplace to have vending machines selling any possible devices -- at least once, the stock in a Japanese vending machine included a car -- but that doesn't seem like something of concern in the US.
This patent application is credited for four inventors. The four have previously worked on two other iterations of the same patent application, possibly suggesting that it has been rejected before, but definitely implying that Apple is pursuing it.
Read on AppleInsider
Comments
This patent disclosure reminds me of going through the chow line in the Navy. To get 300+ people fed in one hour or less everyone going through the chow line would simply point to the food item they wanted and the sailor serving that item on the other side of the glass would slop-it or fork-it on to your tray. Adding a little grunt or the food name was acceptable too, e.g., point followed very quickly by "meat loaf", but pointing alone was usually more than enough to keep things moving along. The galley noise on the other side of the glass was not conducive to striking up a conversation - just point, and grunt if you want to. Believe it or not, while using zero technology the point & grunt worked quite well as long as servers behind the glass didn't run out of food, which could happen and usually meant that you were having ham again. For whatever reason, they never ran out of ham. There must be limitless stockpiles of ham in the Navy's food service supply chain.
I just hope Apple spends some time and effort coming up with a reliable meat recognition algorithm. Maybe skip the ham.