Back
Technology

Google Expands Gemini Capabilities Across Android, Web, and Desktop with New AI Features and Automation Frameworks

Generated on: Last updated: 5 sources
View source

Google is significantly enhancing its Gemini AI assistant, rolling out a suite of new features across its Android mobile application, web application, and a forthcoming Android desktop interface. These upgrades introduce experimental "Labs" features for advanced AI interactions, expand screen automation for Android tasks, and lay frameworks for deeper application integration. Updated privacy guidelines for automated actions are also part of this extensive rollout.

Gemini is set to become a more integrated and capable assistant, spanning mobile, web, and desktop environments with advanced AI and automation capabilities.

Gemini Live Upgrades and Experimental Features

The Google app beta, specifically version 17.2, indicates substantial development for Gemini Live. Currently operating on Gemini 2.5 Flash, Gemini Live is anticipated to transition to being powered by the more advanced Gemini 3. Users of the Gemini app on Android will soon be able to opt into "Labs" features to test future capabilities:

Android "Labs" Features:

  • Live Thinking Mode: This mode is designed to take more time to process requests, aiming to provide more detailed responses. It is described as potentially utilizing either the Thinking or Pro models of Gemini.
  • Live Experimental Features: These features are expected to include multimodal memory, enhanced noise handling, the ability for Gemini to respond when it visually detects something, and personalized results derived from integrated Google applications. These capabilities are linked to Gemini 3 Flash and Pro, with the visual detection feature potentially related to Project Astra.
  • UI Control: This feature will enable the Gemini agent to control the phone to complete specific tasks.
  • Deep Research: This function will allow users to delegate complex research tasks to Gemini.

Gemini Web Application Updates:

Separately, the Google Gemini web application (gemini.google.com) has updated its Tools menu to include an "Experimental Labs" section. The prompt box dropdown is now organized into two distinct sections:

  • Tools Section: This includes features such as Deep Research, Create videos (available with AI Plus), Create images, Canvas, Guided Learning, and Deep Think (available with AI Ultra).
  • Experimental Features Section (Labs): Distinguished by a "Labs" beaker badge, this section contains features under active development like Agent (available with AI Ultra), Dynamic view or Visual layout (available to all users), and Personal Intelligence (available to all paid subscribers).

Expanded Android Automation and Agent Capabilities

Google is developing "screen automation" for Gemini on Android devices, identified in the Google app 17.4 beta under the codename "bonobo." This functionality aims to enable Gemini to assist with tasks such as placing orders or booking rides directly within specific applications. Android 16 QPR3 is reportedly laying the groundwork for this integration. The broader "Computer Use" integration is anticipated for Gemini Agent functionality on Android.

Early-Stage Developer Capabilities:

Google is also introducing early-stage developer capabilities designed to connect applications with agentic apps and personalized assistants like Google Gemini. These features prioritize privacy and security and include two primary approaches:

  1. AppFunctions: An Android 16 platform feature and accompanying Jetpack library, AppFunctions allows applications to expose specific functions that agent apps and AI assistants can access and execute directly on the device. Google notes that these functions execute locally on the Android device.

    • Example Use Cases:
      • Creating tasks in management apps.
      • Invoking playlist creation in music apps.
      • Performing cross-app workflows (e.g., searching email and populating a shopping list).
      • Creating calendar events without manual user interaction.
    • Current Implementations: The Samsung Gallery app on the Galaxy S26 and other Samsung devices running OneUI 8.5 and higher utilize AppFunctions. The Gemini app currently employs AppFunctions for its Calendar, Notes, and Tasks integrations within Google applications and OEM defaults.

  2. UI Automation: For scenarios where dedicated app integrations are not available, Google is developing a UI automation framework. This framework enables AI agents and assistants to intelligently execute generic tasks on users' installed applications. Android 17 is expected to broaden these capabilities. Google is collaborating with app developers on these experiences, with further details anticipated later in the year.

Gemini on Android Desktop Interface

New information has emerged regarding Gemini's integration into an Android desktop interface, with strings found in the Google app beta (version 17.5). These details suggest users will be able to access Gemini for assistance with tasks such as writing, planning, and brainstorming. Access methods include selecting a Gemini icon from the top-right corner of the screen or using a keyboard shortcut involving a Google Key and the Spacebar. The Google app is expected to power this Gemini experience, which is anticipated to launch as an overlay.

Privacy and User Responsibility

Google has issued advisories regarding Gemini's potential for errors when performing automated actions.

Users are informed that they are accountable for actions performed on their behalf and are recommended to supervise Gemini's automation closely.

Users retain the ability to interrupt Gemini's automation and assume manual control.

Key Privacy Guidelines:

  • Screenshots may be subject to review by trained personnel for the purpose of improving Google services, provided "Keep Activity" is enabled.
  • Users are cautioned against inputting login or payment information into Gemini chats.
  • It is advised to avoid using screen automation for emergencies or tasks involving sensitive data.

Other Noted Features

An additional "Personalize chat when helpful" toggle has been added to the Gemini web app, allowing users to manage whether Gemini utilizes their Connected apps for the current conversation. This setting is temporary and will automatically re-enable when a new chat is initiated.

The beta also contains references to a feature or integration named "Likeness," codenamed "wasabi." This designation is associated with how Android XR employs 3D avatars, currently utilized in Google Meet calls, with related strings suggesting potential accessibility of this feature for prompts.