Future HomeKit could track you through your house and predict your needs

Posted:
in General Discussion edited June 18

A future version of HomeKit may keep track of where the people in the house are, and learn user habits to figure out when to automatically take actions, without you having to ask Siri.

HomeKit



The temperature and humidity sensors in the new HomePod, and now enabled in the HomePod mini, are designed to be used as part of automated systems. Right now, you can create a Shortcut that says if the indoor temperature falls below a certain point, you want your heater turned on.

That does require a heater you can control remotely, or perhaps just a smart outlet, plus the HomePod. But it also requires you to set up that Shortcut, and "Using In-home Location Awareness," a newly-granted patent, suggests that Apple wants to move away from that.

Rather than you having to create a Shortcut or to set up a HomeKit automation, Apple wants it all to just happen for you. By itself.

"Users often perform the same or repeated actions with accessory devices while in a particular location," says the patent. "For example, every time a user comes home from work, they may close the garage door when they are in the kitchen."

"Therefore, certain activities with respect to devices in a home may be performed regularly and repeatedly (e.g., daily, several times throughout a day) while the user is in a certain location," it continues. "This can be a time consuming and tedious task for a user since these tasks are performed regularly or several times throughout the day."

"Thus... it is desired for the home application on the mobile device to be able to determine a location of the user," says Apple, "and suggest a corresponding accessory device that a user may want to control or automatically operate a corresponding accessory device based on the location of the mobile device of the user."

Detail from the patent showing a range of connected devices
Detail from the patent showing a range of connected devices



Apple doesn't want to remove the user from the decision to turn on a heater, or a light, but rather to remove any effort.

"[When] a user is at a determined location according to detected sensor information, one or more associated accessory devices can be suggested to a user, thereby enhancing a user's experience," says the patent. "Also, rooms and accessory devices associated with the room can be suggested to a user."

"In addition, scenes and accessory devices associated with the scenes can be suggested to the user," it continues. "The suggestions regarding accessory devices and scenes can be learned or can be rule based."

The key word there is learned. Currently an iOS widget can offer you different information at different times of the day, and it learns that, for instance, you check your calendar first thing, but you like news headlines at the end of the day.

That's exactly what Apple wants with your smart home devices. They're smart, you use them all the time, they should just work.

"It would be beneficial if accessory devices and/or scenes can be automatically suggested to a user based on their current location," says the patent, "or if an accessory device is automatically controlled based on a current location of the mobile device based on the user's history of activity with an accessory device."

The patent runs to more than 17,000 words, and while naturally a large part of that is to do with privacy, the majority is about precisely detecting a user's presence.

As is typical for a patent trying to cover as broad an idea as possible, it's a little vague on the specifics. But if Machine Learning can already provide services such as presenting a calendar in the morning, it is this issue of precision location finding that is what's new and key.

Geolocation is not yet precise enough



Currently, HomeKit is able to have an automation that, for instance, stops music playing when everyone has left the house. That's chiefly done by determining whether iPhones are connected to the home's Wi-Fi network, though.

Or there are motion sensors which can pick up when you enter a room. But the problem there is that, normally, they can only detect motion such as entering or leaving, they don't actually sense a presence.

Aqara has begun selling what it calls a "human presence" sensor, which is able to detect minute body movements, but its device is not yet commonly available.

Rather than look to any one specific new technology, Apple's patent suggests combining information from many sources, and then making calculations.

"For example, the mobile device can include sensors to measure the distance that a user is walking or stepping as they turn on lights in, for example, a room or a house," says Apple. "Motion detectors and motion sensors in, for example, the accessory devices, can also be used to aggregate data regarding a user's movement in a room or area."

The patent doesn't directly refer to the U1 ultra wideband chip, but if it's in a user's devices, it will clearly be used.

So if you enter a room and also head toward your desk lamp instead of the overhead light, HomeKit could know to turn on only that lamp.

Apple is adding the U1 processor to more devices, plus HomeKit can already get information from sources such as motion sensors. Plus, Apple's support of Matter should mean such plans could extend to many more smart home devices.

This patent is one that was previously granted. While no patent is a guarantee of an eventual product, it being re-applied for means Apple is at least refining and updating its research.



Read on AppleInsider

Comments

  • Reply 1 of 19
    lkrupplkrupp Posts: 10,557member
    And I just found out from a TidBits article that Siri now supports rebooting your iPhone/iPad. “Hey Siri, Reboot” It works
    watto_cobraAlex1NVictorMortimerwilliamlondondewme
  • Reply 2 of 19
    This is one of those things that makes sense as research but would likely be impractical/annoying in real world use. 
    boxcatcherFileMakerFellerwatto_cobraAlex1NVictorMortimerCrossPlatformFrogger
  • Reply 3 of 19
    Rather than trying to install a vast array of sensors across an entire house and attempt to infer where someone is + automate what they want, I have to think it’ll end up being a lot easier to just use “Apple Glass”.

    *Walk into a room wearing Apple Glass*

    *Look at a light* Turn on that light

    *Light turns on*

    Basically: why not use the one sophisticated sensor a person is wearing + explicit intent vs. tens of fixed sensors + inferred intent?
    edited February 2023 watto_cobrawilliamlondon
  • Reply 4 of 19
    Yeah, we don’t need this.

    At least I certainly don’t want to be tracked throughout my home just for the possibility that my home automation might be a wee bit helpful but in all likelihood it would just be annoying or wrong.
    FileMakerFellerdecoderringwatto_cobraAlex1NDAalsethVictorMortimerCrossPlatformFrogger
  • Reply 5 of 19
    I was thinking about investing in Homekit but if there was one thing that would turn me away from it forever, that would be it in a nutshell. If Apple implements this then you can count me out forever investing in any home automation system which is kinda ironic as my Masters degree is in Control Engineering.

    I hate all forms of tracking and adverts but as I'm a grumpy old boomer, I don't count do I?
    FileMakerFellerdecoderringboboliciousmuthuk_vanalingamwatto_cobraAlex1N
  • Reply 6 of 19
    After IOS 16.2 update wrecked HomeKit, I can't even use it to control lights that have worked for years. Thank God for Alexa still working. I would appreciate a fix for HomeKit before adventures into new frontiers.
    itinj24jd143cnwatto_cobraAlex1N
  • Reply 7 of 19
    dewmedewme Posts: 5,486member
    My expectations with respect to ambient computing aren’t aligned around Siri ordering more toilet paper, you know, just in case the meal in progress had some serious aftershocks. Geez. 

    I’d be happy with some assistance around energy usage based on approaching weather. 
    FileMakerFellerwatto_cobraAlex1N
  • Reply 8 of 19
    Hmm, it's just HomeKit making suggestions, just more suggestions.

    I actually have presence detection in my home right now (Not motion detection) and it is truly great that lights follow you as you go through the house. Also things like if the last person leaves a room whilst tv is running, to pause it, and then restart it when they return.

    Wish I could get music to follow me.

    Siobhan
    watto_cobraAlex1Nwilliamlondon
  • Reply 9 of 19
    Much as I would love to have Siri be as competent as Jarvis in being my virtual assistant, I’d rather be able to ask Siri first instead of having the automation anticipate my needs. In no way, would I ever let automation control the door to my home, or control a heating device in my home. If I tell Siri to turn the heat on, that’s one thing, for it to just assume that I want it turned on because the temperature drops below 65° at my house is another. I am an elderly lady, so having these devices carry out my needs is very attractive, and I have smart plugs to get Siri to tell them to do that, but I definitely, don’t want to do it on an anticipatory basis. 
    FileMakerFellerdecoderringwatto_cobraAlex1Nwilliamlondon
  • Reply 10 of 19
    Hmm, it's just HomeKit making suggestions, just more suggestions.

    I actually have presence detection in my home right now (Not motion detection) and it is truly great that lights follow you as you go through the house. Also things like if the last person leaves a room whilst tv is running, to pause it, and then restart it when they return.

    Wish I could get music to follow me.

    Siobhan
    Even in my advancing years, I can remember enough of the music I like to have access to it wherever I go. I'm surprised more people don't cultivate this skill.
    beowulfschmidtwatto_cobraAlex1NVictorMortimer
  • Reply 11 of 19
    I was thinking about investing in Homekit but if there was one thing that would turn me away from it forever, that would be it in a nutshell. If Apple implements this then you can count me out forever investing in any home automation system which is kinda ironic as my Masters degree is in Control Engineering.

    I hate all forms of tracking and adverts but as I'm a grumpy old boomer, I don't count do I?
    For your consideration - 12 and counting: gizmodo.com/apple-iphone-privacy-analytics-12-lawsuits-statement-1850077715

    "Open up your iPhone’s settings, tap “Privacy & Security,” and scroll down to “Analytics & Improvements.” There you’ll find a setting label “Share iPhone Analytics.” Toggle it on and off all you want. Tests from Mysk found that Apple collects device analytics data, no matter how you adjust the control"

    Alex1Nwilliamlondon
  • Reply 12 of 19
     Tests from Mysk found that Apple collects device analytics data, no matter how you adjust the control"
    That level of tracking does not bother me for the following :-
    I don't do social media
    I don't use email on my phone.
    I don't use Safari that much either.
    When I'm driving, my phone is usually in Airplane mode.
    Most of the apps I use are related to charging my EV. (different charging networks) or you know... speaking to someone on the phone.
    The opportunity to fire ads at me is limited and I never respond to the few that I see anyway.
    My phone is not the device that runs my life, it is a tool that is used when needed.
     
    Alex1N
  • Reply 13 of 19
    22july201322july2013 Posts: 3,615member
    I've been trying to implement my own track-me-in-the-home feature since X10 technology in the 1990s. I'm almost there now with motion sensors that turn my lights off and on based on my presence. This will probably be the main feature of a $99 Apple Ring if they ever get around to it.
  • Reply 14 of 19
    22july201322july2013 Posts: 3,615member
    I was thinking about investing in Homekit but if there was one thing that would turn me away from it forever, that would be it in a nutshell. If Apple implements this then you can count me out forever investing in any home automation system which is kinda ironic as my Masters degree is in Control Engineering.

    I hate all forms of tracking and adverts but as I'm a grumpy old boomer, I don't count do I?
    I see you graduated from Elon Musk's school of Control Engineering.
  • Reply 15 of 19
    Nope.  Big nope.

    I don't even allow Siri to listen.  I'd rather push a button.
  • Reply 16 of 19
    AppleZuluAppleZulu Posts: 2,073member
    Seems likely that any such features will be very much optional. I wouldn’t get into a pearl-clutching panic yet. 
  • Reply 17 of 19
    dewmedewme Posts: 5,486member
    An implantable AirTag would solve a lot of the human tracking challenges. Mom, Dad, all the kids, and pets too.

    I've always found the user accessible temperature and humidity sensors in a speaker to be bizarre world level stuff. Putting them in a smart thermostat or indoor air quality monitor would absolutely make sense. Imagine the confusion that would ensue if the temp sensor on a HomePod or Echo was tied to the temperature controller in your house and something went wrong with the sensing function and your heat control got whacked out. You call out an HVAC service technician and they finally figure out where the disconnect is and tell you something like "Sorry ma'm, but you have to replace the speaker in your family room for the heat to come back on." I can imagine the home owner's  response would be something like: "WTF, and you're probably going to tell me next that I need to replace the fan belt in my Subaru to get my ice maker to work."

    I can totally understand where temperature and humidity sensors in a speaker would help with acoustic beamforming because both factors affect sound propagation. I suppose if those sensors were already there for sound processing, so exposing them to users would be low hanging fruit to add a couple of bullet points to the speaker's feature list. But from a smart home system perspective it is simply weird, especially for those folks who subscribe to the single responsibility principle for software design and architecture. Maybe I'm too much of a purest. I will go back to my cave, turn on my TV / lawn sprinkler controller. 
  • Reply 18 of 19
    AppleZuluAppleZulu Posts: 2,073member
    dewme said:
    An implantable AirTag would solve a lot of the human tracking challenges. Mom, Dad, all the kids, and pets too.

    I've always found the user accessible temperature and humidity sensors in a speaker to be bizarre world level stuff. Putting them in a smart thermostat or indoor air quality monitor would absolutely make sense. Imagine the confusion that would ensue if the temp sensor on a HomePod or Echo was tied to the temperature controller in your house and something went wrong with the sensing function and your heat control got whacked out. You call out an HVAC service technician and they finally figure out where the disconnect is and tell you something like "Sorry ma'm, but you have to replace the speaker in your family room for the heat to come back on." I can imagine the home owner's  response would be something like: "WTF, and you're probably going to tell me next that I need to replace the fan belt in my Subaru to get my ice maker to work."

    I can totally understand where temperature and humidity sensors in a speaker would help with acoustic beamforming because both factors affect sound propagation. I suppose if those sensors were already there for sound processing, so exposing them to users would be low hanging fruit to add a couple of bullet points to the speaker's feature list. But from a smart home system perspective it is simply weird, especially for those folks who subscribe to the single responsibility principle for software design and architecture. Maybe I'm too much of a purest. I will go back to my cave, turn on my TV / lawn sprinkler controller. 
    Temperature and humidity sensors are cheap. If you’re already going to have HomePods in various rooms around the house, why not include them? 

    As for your nightmare HVAC scenario, you’d have to be pretty daft to use this optional feature and set up HomeKit automations linking sensors to control your heat and air and then forget to check on that before you call an HVAC service tech. 

    P.S. I sometimes use the My Home function on my Apple TV to turn on the sprinkler. You don’t have to use it, but that’s what HomeKit is for. 
  • Reply 19 of 19
    dewmedewme Posts: 5,486member
    AppleZulu said:
    dewme said:
    An implantable AirTag would solve a lot of the human tracking challenges. Mom, Dad, all the kids, and pets too.

    I've always found the user accessible temperature and humidity sensors in a speaker to be bizarre world level stuff. Putting them in a smart thermostat or indoor air quality monitor would absolutely make sense. Imagine the confusion that would ensue if the temp sensor on a HomePod or Echo was tied to the temperature controller in your house and something went wrong with the sensing function and your heat control got whacked out. You call out an HVAC service technician and they finally figure out where the disconnect is and tell you something like "Sorry ma'm, but you have to replace the speaker in your family room for the heat to come back on." I can imagine the home owner's  response would be something like: "WTF, and you're probably going to tell me next that I need to replace the fan belt in my Subaru to get my ice maker to work."

    I can totally understand where temperature and humidity sensors in a speaker would help with acoustic beamforming because both factors affect sound propagation. I suppose if those sensors were already there for sound processing, so exposing them to users would be low hanging fruit to add a couple of bullet points to the speaker's feature list. But from a smart home system perspective it is simply weird, especially for those folks who subscribe to the single responsibility principle for software design and architecture. Maybe I'm too much of a purest. I will go back to my cave, turn on my TV / lawn sprinkler controller. 
    Temperature and humidity sensors are cheap. If you’re already going to have HomePods in various rooms around the house, why not include them? 

    As for your nightmare HVAC scenario, you’d have to be pretty daft to use this optional feature and set up HomeKit automations linking sensors to control your heat and air and then forget to check on that before you call an HVAC service tech. 

    P.S. I sometimes use the My Home function on my Apple TV to turn on the sprinkler. You don’t have to use it, but that’s what HomeKit is for. 
    Sure, if it’s cheap just stick it everywhere. I suppose that makes sense for Amazon, but Apple? I think the temperature and humidity sensors in a speaker illustrate how little Apple currently has to contribute to home automation beyond clients and gateways. 

    Things like sensors and actuators and device management of the automation devices isn’t really where Apple’s domain expertise lies. This isn’t a knock on Apple’s business focus, but it does expose a weakness in some fundamental things that they are happy to relegate to partners and third party vendors. For example, it makes no sense to me why I can’t centrally manage the configuration and software/firmware updates on all Apple devices that are part of the Home system, most notably Apple TVs. 

    Apple’s Home app feels like a parking garage for a bunch of devices that have various degrees of connectedness and are very loosely coupled or standalone. Apple Home isn’t really part of a cohesive system and it is not a system management console by any stretch of the imagination. I’ve been hoping it would eventually grow into a system management console. As far as I know there isn't a way to search in the Home app.

    I understand that Apple doesn't really want to be a system house for anything other than its own products and services. That makes sense for the way they've been doing business for decades, which is device centric, product centric, and with service cloud that brings a high level of connectedness and interoperability between their devices, products, and services. Having to physically touch every Apple TV to manage it does not seem like a stretch under their current model. But for home users or business users who have multiple Apple TVs, having to to physically visit every one of them to configure settings or force an update is kind of a pain. I only have 6 Apple TVs and it would be great to update any or all of them from one client or configure one of them to my liking, save the settings, and deploy the same configuration to the rest of them. Same deal with Time Machine configuration.

    My hope is that the Matter standard will provide a way to build the kind of managed system I would like to have. I just hope that Apple plays along and doesn't lock us into using their Home client the way it currently works. If Matter provides a common, vendor independent, and platform independent way to create device profiles that can be used by any Matter compliant console/client to manage any Matter compatible device this could happen. I just don't think Apple is able to put up with the typical timelines and analysis paralysis that sometimes slows standards development to a crawl. We will see.
    edited June 20
Sign In or Register to comment.