In May 2023, Google’s AI team published a report titled “Enabling conversational interaction on mobile with LLMs,” which focused on testing large language models with a phone’s user interface. The report discussed incorporating large language models into graphical user interfaces (GUIs), such as apps and software displayed on a phone’s screen. It highlighted four main areas of application, including summarizing on-screen content, answering questions based on displayed content, and, most notably, assigning UI functions to language prompts.

For instance, the language model can analyze the UI to generate contextual questions automatically. It can then convert the gathered information into questions that the model can promptly answer when asked by the user. Another noteworthy capability is “screen question answering,” where the AI can provide details, such as the headline, author’s name, and publishing date when a blog post is open in a web browser.

However, the most promising application area is “mapping instruction to UI action.” This essentially means controlling your phone using prompts, whether through voice or text. The enhanced conversational abilities of the virtual assistant enable it to perform tasks like opening an app or adjusting phone settings, such as the cellular network mode. Although the exact arrival date of the supercharged Google Assistant is unknown, it would greatly expand its capabilities. Interestingly, it appears that Apple is also exploring generative AI tools, internally referred to as AppleGPT, to enhance Siri.