Google has a company-wide commitment to AI, especially when it comes to innovative use cases, and one of the best demonstrations of that is a new tool being worked on by a couple of researchers within the company that's made to preserve a user's privacy by detecting when somebody is peering at a phone over the user's shoulder. The project leverages TensorFlow Lite for onboard AI operations, cutting the cloud out entirely and thus reducing detection time to around 2 milliseconds. Google researchers Florian Schroff and Hee Jung Ryu developed the system, and are planning to show it off at the upcoming Neural Information Processing Systems in Long Beach, California. The conference will run from December 4 to December 9.
Using the front-facing camera on a device, the system is able to find another pair of eyes in its view besides the primary user, then determine whether those eyes are gazing at the user's phone screen. From there, of course, app developers can program their apps to react however they choose, such as interrupting the app to show the stranger to the user. The whole thing is powered by deep facial recognition algorithms that require many machine learning processes to run simultaneously. Thanks to the onboard machine learning feature in Qualcomm's Snapdragon 835, however, researchers were able to process everything on device to cut down on the time it takes to detect an onlooker. Doing the same thing offsite would require streaming the video to an outside source for processing, which would be slowed not only by sending and processing the video, but also by the response to a trigger like a pair of eyes being sent from the offsite server to the phone.
This feature is in very early stages of development, and there's no telling exactly when it may become available to consumers and in what form, if it will become available at all. The demo seen in the video is conducted on a Google Pixel 2, which is powered by the aforementioned Qualcomm Snapdragon 835 processor. The Snapdragon 835 is not the only mobile processor out there with onboard machine learning features, but it is the only one with out-of-the-box TensorFlow integration, which means that this feature and others developed directly by Google may take a while to reach other flagship processors like those from Huawei and Samsung, unless those other processor makers decide to work with Google on TensorFlow integration in their own processors.