Apple Car cameras may identify landmarks & objects of interest, detect broken windows

in Future Apple Hardware
Beyond self-driving, the Apple Car may use its cameras to identify objects of interest and other vehicles, and for detecting damage to the car's windows.

A van by Apple-acquired startup bearing external camera sensors
A van by Apple-acquired startup bearing external camera sensors

Digital imaging is an important part of any self-driving car, as it is used in concert with other systems to read the road and to spot potential hazards. Having the collection of cameras could be beneficial in other ways, which Apple has considered in a pair of related vehicular imaging patents.

Object Recognition

For the first patent, "Method and system for providing at least one image captured by a scene camera of a vehicle," Apple suggests using the onboard cameras to capture a photograph of a nearby object. While this would typically be relied upon for object recognition for driving, such as identifying bollards, traffic cones, or pedestrians, Apple's proposal is a little more left-field in purpose.

The filing reasons the driver or a passenger may be interested in identifying an object of interest for themselves, potentially to find other types of the same object. Furthermore, a user may want to purchase that object, if possible, and could be directed to a store offering it for sale, or for "hands-on checking."

It may also be impractical for a person in a vehicle to take a photograph of something outside while on the move, such as if they are pressed for time and cannot stop the car.

Apple's description of the patent basically revolves around telling the car to take a photograph of something outside the vehicle, triggering the car's outward-facing cameras to capture an image or multiple images. These photographs can then be used either as a record of a moment in a scene or for an object-recognition system.

The patent suggests a user's gaze could tell the external cameras where to photograph.
The patent suggests a user's gaze could tell the external cameras where to photograph.

The filing suggests that the Apple Car photo system could monitor the user's gaze to determine where they are looking at the time of a vocal request. For passengers, a gesture could be detected to perform the same task.

In the case of object recognition, the images can be passed to a system to recognize objects in a scene, determine the one of interest to the user, and then to acquire more information about that item.

The patent lists its inventors as Lejing Wang and Thomas Alt, and it was filed on June 21, 2019.

The idea of using external cameras for alternative purposes has been explored before, with the 2019 patent application of "Automated capture of image data for points of interest" used to take photographs of areas of an environment that may be worth knowing about to a user. That same application suggested that a user could ask for a photograph to be taken of a scene.

Broken Windows

The second patent, "Window defect sensing and image processing" is more straightforward, but revolves around the use of cameras inside the vehicle, rather than those facing out. This could include cameras used for gesture recognition or eye gaze detection, which may be used for other car functions.

In short, the patent suggests illuminating car windows, and using imaging sensors to capture a photograph of the window. The image is then used to determine if there are any cracks or defects in the window, which could form a safety hazard for the passenger or driver.

Apple proposes the use of edge lighting to put light through the inside of the window pane, with surrounding modules able to illuminate different sections. These light modules can transmit either typically unfiltered light or graze lighting, namely directional light at an acute angle, which may be better at highlighting cracks and other issues.

Multiple in-car imaging sensors and light modules could help detect cracks and window issues.
Multiple in-car imaging sensors and light modules could help detect cracks and window issues.

Rather than relying on one imaging sensor, Apple suggests the use of multiple sensors to take images, which can then be compared. As each sensor is located in a different known position, this can enable enhanced processing to be performed on the images, such as determining the external view of the vehicle and eliminating that from the data to leave just the window and defects.

Once an issue has been determined, the findings can be communicated to the driver or other users through a notification or alert.

The system isn't necessarily limited to just cracks and scratches, as it could also be used for more everyday features. It is suggested a window cleaning system could be produced to spot clean specific points of the window itself, rather than cleaning the entire window each time.

The patent's inventors are listed as Clarisse Mazuir, Malcolm J. Northcott, Jack E. Graves, James R. Wilson, Christopher D. Jones, and Martin Melcher. It was filed on May 15, 2018.

Apple has come up with other ways to detect cracks in windows, such as one revealed in early August that relied on an infrared light-blocking layer, ohmic heating currents, and resistive measurements to determine crack sizes and locations.

Apple files numerous patent applications on a weekly basis, but while the existence of a patent indicates areas of interest for the company's research and development efforts, they do not necessarily mean the idea will appear in a future product or service.


  • Reply 1 of 4
    rozhasi said:
    There is NO Apple Car. There will NEVER be an Apple Car. 

    Maybe Apple will have a self-driving EV framework that they will license out to other car manufacturers, but Apple will NEVER manufacture their own EV. They are way too late to the game. If they wanted to manufacture their own car, they would have bought TSLA when it was worth $30 billion by offering Musk $60 billion for it. If he refused, they would have purchased some other startup. At this point, there is too much competition, including the recent startups that are about to release their EVs to the market in 2021. Apple can't compete with the established car manufacturers anymore, and neither can they offer anything compelling to the customer apart from what Tesla and other EV manufacturers are working on. 
    Please explain how it is “too late” for Apple to enter the car market but it isn’t too late for “recent startups” to enter.

    Also, what do you base all your beliefs on? Typically, when someone presents an assertion they offer something to back up their claims. So, why is Apple too late? Why can’t they offer anything compelling? How come they can’t compete with established manufacturers?
    edited August 2020
  • Reply 2 of 4
    Digging the stick figures.  Always impressed by the art department within Apple Legal.
  • Reply 3 of 4
    lojacklojack Posts: 28member
    I'm sure there's a very specific artistic skill set to be descriptive enough to explain and yet vague enough to not give too much away.  :)
  • Reply 4 of 4
    dewmedewme Posts: 5,282member
    That van looks surprisingly clean. They have done an excellent job of washing all of the ectoplasm off of it from its last service call. Good choice of color too, since it contrasts with all known ectoplasmic color variations. Now we know who to call.
    edited August 2020
Sign In or Register to comment.