Apple details 'Siri for Mac' desktop virtual assistant in new patent application
While Apple's Siri has been an iOS staple since 2011, the tech has yet to make the jump to OS X. An exhaustive new patent filing, however, details a very similar desktop version of the virtual assistant that can perform dictation, high-level system commands and even act as a "third hand" for Mac users.
As published by the U.S. patent and Trademark Office on Thursday, Apple's massive 92-page patent application for an "Intelligent digital assistant in a desktop environment" describes a technology that goes far beyond Mac's current voice dictation feature, and can easily be considered a "Siri for Mac."
Like the current iteration of Siri, limited to iOS, Apple's desktop version is able to process natural speech and text input to perform actions like completing tasks, inputting and retrieving data, conducting searches and more. Further, the filing points out that commands are to be taken in context based on deduced user intent. In other words, Siri for desktop will use speech recognition to decipher and remembering contextual clues.
Source: USPTO
Driven in cooperation with an off-site server, Siri for Mac has an identical backend to Siri for iOS. The user-facing Siri asset can be integrated into a standalone app invoked by a special mouse or keyboard gesture, or may be tasked to run in the background, listening for a special activation phrase like the "Hey, Siri" implementation in Apple's forthcoming iOS 8.
In some embodiments, an icon of the virtual assistant can reside in the dock, notifying the user that it is active and ready for service. This and other modes allow the user to speak commands and start dictations from anywhere in the operating system.
Voice input is of specific importance to the patent's disclosures, as the technology is looking to augment keyboard and mouse input, or in some cases replace the physical tools altogether. The usual answer/response method seen with Siri for iOS is applied to the desktop variant, though more advanced operations can be performed given the extra computing power afforded by a proper computer.
Apple's desktop Siri version uses "focus selector," or mouse cursor, to help users apply context to a command. For example, the assistant can perform a copy operation for an asset under a hovering cursor, or may "hold" a batch of highlighted files for later retrieval. In essence, the focus selector plays the same role as an iOS device user's finger, except with substantially more options for control thanks to the Mac's GUI.
With mouse access, users can drag and drop files, folders and other assets onto the Siri dock icon for processing. For example, a folder full of pictures can be sorted by telling the assistant, "Sort by date," or "Sort by location." Following completion of the operation, a secondary command may be issued, or the icon may throw up a list of extra options in text or voice form.
Multiple assets can be dragged and dropped onto the assistant for advanced collation, merging and advanced comparison operations. A number of other operations are also described in the full patent document.
A particularly interesting function describes "Using Digital Assistant as a Third Hand." Apple notes that current graphical user interfaces are somewhat limiting in areas like multitasking. For example, while multiple applications may be running in a desktop environment, users are usually restricted to interacting with only one window or asset at a time.
In some cases, a user may be performing one task and find that a second or third action is required to facilitate unbroken focus on the primary operation. For example, while typing in a text editor, a user may want to access a picture or piece of information from the Web. Invoking a digital assistant by voice to perform this secondary task saves mouse clicks and allows continuous focus be paid to the text editing task at hand.
Alternative embodiments include creating folders while having multiple files selected on the desktop, asking the assistant to resize pictures, sending emails and changing speech into text, among many other functions. In theory, Siri for Mac would be able to perform any operation otherwise available to the user.
It is unclear whether Apple plans to roll out a version of "Siri for Mac" in the near future, though the latest OS X 10.10 Yosemite betas do not have such functionality built in. Rumors have long said Apple is planning to release a desktop virtual assistant -- going as far back as Siri's introduction on iPhone -- but there has yet to be any hard evidence of a working OS X solution.
Apple's Siri for Mac patent application was first filed for in 2013 and credits Julian K. Missig, Jeffrey Traer Bernstein, Avi E. Cieplinski, May-Li Khoe, David J. Hart, Bianca C. Costanzo, Nicholas Zambetti and Matthew I. Brown as its inventors.
As published by the U.S. patent and Trademark Office on Thursday, Apple's massive 92-page patent application for an "Intelligent digital assistant in a desktop environment" describes a technology that goes far beyond Mac's current voice dictation feature, and can easily be considered a "Siri for Mac."
Like the current iteration of Siri, limited to iOS, Apple's desktop version is able to process natural speech and text input to perform actions like completing tasks, inputting and retrieving data, conducting searches and more. Further, the filing points out that commands are to be taken in context based on deduced user intent. In other words, Siri for desktop will use speech recognition to decipher and remembering contextual clues.
Source: USPTO
Driven in cooperation with an off-site server, Siri for Mac has an identical backend to Siri for iOS. The user-facing Siri asset can be integrated into a standalone app invoked by a special mouse or keyboard gesture, or may be tasked to run in the background, listening for a special activation phrase like the "Hey, Siri" implementation in Apple's forthcoming iOS 8.
In some embodiments, an icon of the virtual assistant can reside in the dock, notifying the user that it is active and ready for service. This and other modes allow the user to speak commands and start dictations from anywhere in the operating system.
Voice input is of specific importance to the patent's disclosures, as the technology is looking to augment keyboard and mouse input, or in some cases replace the physical tools altogether. The usual answer/response method seen with Siri for iOS is applied to the desktop variant, though more advanced operations can be performed given the extra computing power afforded by a proper computer.
Apple's desktop Siri version uses "focus selector," or mouse cursor, to help users apply context to a command. For example, the assistant can perform a copy operation for an asset under a hovering cursor, or may "hold" a batch of highlighted files for later retrieval. In essence, the focus selector plays the same role as an iOS device user's finger, except with substantially more options for control thanks to the Mac's GUI.
With mouse access, users can drag and drop files, folders and other assets onto the Siri dock icon for processing. For example, a folder full of pictures can be sorted by telling the assistant, "Sort by date," or "Sort by location." Following completion of the operation, a secondary command may be issued, or the icon may throw up a list of extra options in text or voice form.
Multiple assets can be dragged and dropped onto the assistant for advanced collation, merging and advanced comparison operations. A number of other operations are also described in the full patent document.
A particularly interesting function describes "Using Digital Assistant as a Third Hand." Apple notes that current graphical user interfaces are somewhat limiting in areas like multitasking. For example, while multiple applications may be running in a desktop environment, users are usually restricted to interacting with only one window or asset at a time.
In some cases, a user may be performing one task and find that a second or third action is required to facilitate unbroken focus on the primary operation. For example, while typing in a text editor, a user may want to access a picture or piece of information from the Web. Invoking a digital assistant by voice to perform this secondary task saves mouse clicks and allows continuous focus be paid to the text editing task at hand.
Alternative embodiments include creating folders while having multiple files selected on the desktop, asking the assistant to resize pictures, sending emails and changing speech into text, among many other functions. In theory, Siri for Mac would be able to perform any operation otherwise available to the user.
It is unclear whether Apple plans to roll out a version of "Siri for Mac" in the near future, though the latest OS X 10.10 Yosemite betas do not have such functionality built in. Rumors have long said Apple is planning to release a desktop virtual assistant -- going as far back as Siri's introduction on iPhone -- but there has yet to be any hard evidence of a working OS X solution.
Apple's Siri for Mac patent application was first filed for in 2013 and credits Julian K. Missig, Jeffrey Traer Bernstein, Avi E. Cieplinski, May-Li Khoe, David J. Hart, Bianca C. Costanzo, Nicholas Zambetti and Matthew I. Brown as its inventors.
Comments
Lots of questions. But that knowledge navigator video still entices. That level of assistance seems like it would be so useful.
For privacy (and sanity) reasons, I’ve never understood the “always on” concept as it has been implemented using real technology.
However, I really like the idea itself: my hardware should just immediately respond to me, and in the case of something like, oh, a wise home, having a system that is always ready to respond to your voice commands makes a lot of sense. Have a mic/speaker combo installed in every room and just say what you want when you want it, Star Trek style.
But let’s assume the tech can be secure and protect your privacy. Say… the processing is done client-side, which is where it BELONGS. What then?
Well, then the only problem becomes power draw, which is a real concern on mobile devices. But on the desktop? Not so much. So an always-listening Siri for OS X is a wonderful idea. And on the desktop, client-side processing will happen FAR sooner than on mobile devices. Heck, Apple could very probably do it FROM THE GET-GO.
They won’t, but it would be the right thing to do.
What do you mean? Siri should be intelligent enough to offer contextual suggestions (and, as in the case of the Knowledge Navigator, be able to link current data to past data to help people draw conclusions), but at NO time should it be allowed to make ANY changes of ANY sort without approval from its user. Same with a wise home, same with anything like this.
I'm expecting a few 'Dave, I can't let you do that" jokes in the thread though ...
I mean can Siri write emails and messages for you -- should it be allowed to? When? How directed must it be? Post on social media? Initiate phone calls? Rename files? Trash files?
If so, it better be 100% accurate. And not hackable under any circumstances.
And even assuming the above to be true, what level of power will freak people out anyhow? If Siri can do all the above will they be concerned that it'll go renegade and start emailing your boss and sending pictures to the local newspaper, erasing your files and locking you out from them.
I can imagine those memes, those ads, those worries. So how do you mitigate them? Start with a weak assistant and build from there? Blitz the truth from the start and hope the fears never catch on?
Just lots of things to think through.
No, of course not. There’s a system in place already, for example, that will automatically send out an automated “I’m away” IM or e-mail, and you set that up manually. Therefore you could tell Siri what you want to have the message say (instead of typing it manually) and you could tell her when you want to have it begin sending (instead of ticking the tickbox manually), but Siri should NOT be able to, say, look at your location and activities, notice that you’re on vacation, and decide on her own to send out that message.
These, of course, should also not be allowed without user input. NOTHING should be allowed without user input.
To get a better feel of what I mean, once battery tech gets to a certain point, we’ll have general purpose humanoid robotics: the dream of nearly a century now. A personal robotic assistant at home, in human form.
I want one. So much.
BUT, these things are generally shown to have varying degrees of autonomy in fiction. In reality, they will have NO autonomy. You’ll be able to load them with programs (clean the house, gardening, etc.), and their sensors will be certainly intelligent enough to react to new information on the fly (can’t clean the house if the program expects exactly this item in exactly this place to pick up and put away, so it will be able to recognize objects for what they are, where they are, know how to pick them up, know where they go, and know how to walk differently to put them where they belong), but they won’t be able to make any decisions whatsoever without input from the user.
Wow there! You got the Tour? Anything else you'd like to share with total strangers here? :smokey:
A "Virtual Assistant"? Looking at those screens, with the little microphone icon asking "Jeff, How can I help you", it just looks a lot like Clippy, which we still poke fun at Microsoft for thinking was ever a good idea. But I have faith that Apple will think of a way to present it that doesn't have the annoying downsides that Clippy had, but we'll see.
Not a tour in that sense, as an authorized dealer MD I was at Apple HQ a lot for meetings, monthly at least. This was for product launches, sales meetings and so on. This was shown prior to the launch of some new product but I can't remember what, 1987 I think. In fact, I just remembered, John Scully, the guy behind the making of that video, actually came to our premises shortly after the Apple Center was opened at that time part of the McQueen group, a printing company that was tied in with John Warnock and Paul Brainerd the bosses of Adobe and Aldus. McQueen were the European distributors for Adobe and Aldus products. It was a hectic time indeed! My management team did a buyout of the Apple side of the business.
Well you will have more chance of help with a female assistant at least!
1. Interesting! Would love to read more on you reminiscing those days. I like that word, reminiscing.
2. Buyout paid off, after which you moved to the US? I remember you are there now for, what, 25 years? Probably don't miss the UK weather since you're in the Sunny State. Then again, now you're having the occasional hurricane, which isn't good either.
Wow there! You got the Tour? Anything else you'd like to share with total strangers here?
What Chuck Norris wants, Chuck Norris gets! :P
Oh great, the last thing I want is my computer judging me on the type of porn sites I visit. Siri: "really, you want me to search for, male Philipeneo naked jello wrestling, you have children Relic", "ugh, shut up Siri"! "Where's the uninstaller for this bitch".
Finally a porn reference in regards to the "third hand" mode, I can believe it took this many posts before it happened
Had to LOL at your post, you have a fantastic sense of humor, especially with what you're going through.
I'm hopeful that Apple is using the remote processing to learn as much as they can and refine the processing before they push larger and larger portions of it local (whether desktop or mobile).
Do you worry that your boyfriend/girlfriend/husband/lover/secretary/workmates/wife/children/dog observe you?
?If you do, why are you posting on this website? You've just added your computer, your router, the ISP, the MSP, the AI firewall, the AI server, and just about anyone who cares to infiltrate your inner being.
Siri on a Mac is what we all want.
Skynet is what we all fear.
Something in the middle is what we deserve.
Ok, I know this wasn't the main point of this article but anyone else get thrown for a loop with the touch screen on a mac that seems to be presented here? Is this the potential future iPad pro or 12" macbook air or is the "touch screen" more limited - i.e. just something that waits for the input to activate Siri and not an actual multitouch interface like iOS.
Would love this. Walk into the room & have Siri listening for commands. Less creepy than Google Chrome's 'always on' feature.
Would love this. Walk into the room & have Siri listening for commands. Less creepy than Google Chrome's 'always on' feature.
At least Google is quite, Siri gives me the heebie jeebies.
If it didn't the feature wouldn't be very useful. Probably never. At least not anythign that wasn't agreed to before hand. Good question. As it works today on an iPhone it isn't especially useful. If the AI's powering these things can evolve even part way to what we see in science fiction then yeah it could be useful. Very useful in fact. You need to learn to walk before you can run. I suspect it will take decades to get these sorts of features to the point that they satisfy the needs of most users. The real trick here is to get the AI and a good part of its intelligence to sit locally on the computer. The AI should be able to do useful work without hitting the net.
Sure does! However don't expect a Knowledge Navigator type capability with the next Mac OS release. Rather expect something that can be built upon.