An Apple Patent reveals an advanced Machine Learning System for Home apps using Microlocations' Tagged Data

Image: MAVRiC – Click on image to Enlarge

Today the US Patent & Trademark Office published a patent application from Apple that relates to providing improvements in determining the user's position within their home and identify an application that the user is likely to use in that particular room. It could be to provide the user with their Apple TV app to turn on the TV when they enter the family room or provide a garage door opener app as you enter the garage and much more.

Apple notes in their patent background As modern mobile devices become more integrated with modern day life, the number of applications stored on the mobile devices increases. It is not uncommon for modern mobile phones to have hundreds of applications. Having numerous applications may allow the mobile device to be particularly useful to the user; however, it may be difficult, and time consuming for the user to find and run a desired application amongst all of the available applications.

Apple's invention provides improvements to determining the user's position within their home and identifies an application for a user based upon their determined position within the home.

An application (e.g., home application) on a mobile device may be used to control other devices, such as accessory devices (e.g., kitchen appliances, lightings, thermostat, smart locks on doors, window shades, etc.), throughout a home. A user of the home application may be in the same room as the accessory device that is controlled or may be in a different room from the accessory device that is being controlled. For example, a user may be in their kitchen when they use the home application on their mobile device to close the garage door.

An "accessory device" can be a device that is in or in the vicinity of a particular environment, region, or location, such as in a home, apartment, or office. An accessory device can include a garage door, door locks, fans, lighting device (e.g., lamps), a thermometer, windows, window blinds, kitchen appliances, and any other devices that are configured to be controlled by an application, such as a home application.

An accessory device can be determined or associated with a home by the home application. An accessory device can be determined by, for example, a mobile device automatically scanning an environment for any accessory devices, or a user may manually enter accessory device information via, for example, the home application.

Users often perform the same or repeated actions with accessory devices while in a particular location. For example, every time a user comes home from work, they may close the garage door when they are in the kitchen. In addition, when it is dark outside, the user may turn on a lamp in the living room or change the temperature on a thermostat while in the living room. Therefore, certain activities with respect to devices in a home may be performed regularly and repeatedly (e.g., daily, several times throughout a day) while the user is in a certain location. This can be a time consuming and tedious task for a user since these tasks are performed regularly or several times throughout the day.

Embodiments provide improved mobile devices and methods for recommending applications and/or accessory devices, or automatically performing an action with the application based on historical usage of the application at identifiable locations (which may be referred to as microlocations) using sensor measurements.

Sensor(s) on the mobile device (e.g., an antenna and associated circuitry) can measure sensor values from wireless signals emitted by one or more signal sources that are essentially stationary, e.g., a wireless router in the home or a network enabled appliance. These sensor values are reproducible at a same physical position of the mobile device, and thus the sensor values can be used as a proxy for physical position. In this manner, the sensor values can form a sensor position, although in sensor space, as opposed to physical space.

A "sensor position" may be a multi-dimensional data point defined by a separate sensor value for each dimension. In various embodiments, a parameter of a wireless signal can be a signal property (e.g., signal strength or time-of-flight, such as round-trip time (RTT)), or other sensor values measured by a sensor of a mobile device, e.g., relating to data conveyed in the one or more wireless signals.

A "cluster" corresponds to a group of sensor positions (e.g., scalar data points, multi-dimensional data points, etc.) at which measurements have been made. Sensor positions can be determined to lie in a cluster according to embodiments described herein. For example, the sensor positions of a cluster can have parameters that are within a threshold distance of each other or from a centroid of a cluster. When viewed in sensor space, a cluster of sensor positions appears as a group of sensor positions that are close to one another. A cluster of sensor positions can be located, for example, in a room of a house or in a particular area (e.g., hallway, front door area) of a house.

A location in a house or building can also be referred to as a "microlocation." A location can be referred to as a microlocation because the location refers to a specific area in, for example, the user's home. In addition, a location or microlocation can also be referred to as a cluster of locations. The terms location, microlocation, and cluster of locations may refer to a same area or region. A home may have a number of locations. A location can correspond to a room in a house or other areas in a house. For example, a location can be a backyard area, a front door area or a hallway area.

Apple's patent FIG. 4 shows a block diagram of a system for identifying an application for a user based on a sensor position; FIG. 5 illustrates an example of microlocations using unsupervised machine learning.

Apple's patent FIG. 7 below illustrates a simplified block diagram of a semi-supervised machine learning model; FIG. 8 illustrates an example outcome produced by a semi-supervised machine learning model.

Apple's patent FIG. 10B above illustrates . a simplified block diagram of a prediction system for an application including an application-specific microlocation machine learning model.

According to some embodiments, an application can automatically generate tagged samples without the user actively requesting them. For example, when a user opens the front door using a home application on the mobile device while in the driveway (assuming that the front door is equipped with a smart lock), the home application can automatically generate a tagged sample by measuring signal values at that location, and labeling that tagged sample with "front door."

Subsequently, after the machine learning model has been trained, the home application can provide "opening front door" as a recommendation on the user interface or automatically opens the front door, when the machine learning model predicts that the user is in the drive way (e.g., when the machine learning model determines that the data point is "similar" to the cluster of data points associated with the driveway).

As another example, a wireless streaming application can use a semi-supervised machining learning model for predicting target devices to project video or audio. After the machine learning model has been trained, the wireless streaming application can provide the living room TV as a recommendation when the machine learning model predicts that the user is in the living room.

Apple's patent covers the following topics:

For engineers and enthusiasts wishing to dive into the finer details of this invention, review Apple's patent application number 20230179671.

Apple Inventors

Posted by Jack Purcher on June 08, 2023 at 05:38 AM in 1A. Patent Applications, Maps, Indoor Location | Permalink | Comments (0)