Google just gave us a tantalizing glimpse into the future of AI agents

Google is trying to turn AI chatbots to intelligent systems that can reason, plan, and remember. Project Astra is a window into this potential future.

• At Google IO conference, CEO Sundar Pichai teased where AI is headed next.
• It's going to be all about agents that are better at reasoning and can act on our behalf.
• What if Google did the searching for you?

What if the Google Assistant was actually… an assistant?

It's a question the company finally started answering at the 2024 Google IO conference on Tuesday, where the search giant fired off AI announcements by the dozen.

With Google and other AI companies making advances in how these systems can ingest images, videos, sound, and text, we're beginning to see how these systems are evolving from smart chatbots to more sophisticated tools that can do more of the hard work.

It's something Google CEO Sundar Pichai is thinking a lot about right now.

"I think about AI agents as intelligent systems that show reasoning planning and memory," Pichai said in a roundtable with reporters ahead of the developer show this week.

"They're able to think multiple steps ahead and work across software and systems all to get something done on your behalf and most importantly, with your supervision."

In short: AI agents are what have the best shot at taking this technology from "nice to have" to "need to have."

Project Astra

Pichai said Google is "in the very early days" of developing this, but promised we'd see glimpses of "agentic direction" across Google's products at IO. And there were several, but the big one people will be talking about is Project Astra.

Astra is a vision of what the Google Assistant should have been all along. You can also think of it a smarter version of Google Lens, one that uses real-time computer-vision capabilities to let you ask it questions about what you can see and hear around you.

"We've always wanted to build a universal agent that will be useful in everyday life," Google DeepMind chief Demis Hassabis said. "Imagine agents that can see and hear what we do better, understand the context we're in and respond quickly in conversation making the pace and quality of interaction feel much more natural."

Google showed a demo of someone holding their phone up with the camera on and asking the AI voice assistant questions about what it was seeing. For example, they pointed it out the window and asked "What neighborhood do you think I'm in?" Correctly, it located Google's King's Cross office in London.

Hassabis stressed that this demo video was recorded in "real time." Google got a lot of blowback in December after a Gemini AI model demo turned out to be edited, so Google needed to emphasize it can really do this — especially after OpenAI showed off a similar demo on Monday.

If there really is no manipulation here, Astra is certainly impressive, and will probably be the big takeaway of the show. But there are other ways these AI agents will emerge in the nearer term.

Google teased a combination of updates that will soon make its Gemini AI chatbot more capable and proactive. Some of this is being unlocked as Google continues to increase the context window, which is the amount of information a large language model can ingest at a time.

Say you want to know something buried deep in a series of very long documents that you don't want to spend hours sifting through. With a large context window, you can share all the documents with Gemini and then ask questions. The model can answer quickly, based on all the information it just ingested.

Tentacles and tailoring

But it's in its legacy products that Google really has an edge when it comes to enabling agent-like qualities.

With its tentacles already in many aspects of our lives, from email to search to maps, Google can synthesize all its knowledge about users and the world around them to not just answer queries, but tailor responses.

"My Gemini should really be different than your Gemini," said Sissie Hsiao, head of Gemini and Google Assistant.

Later this year, Gemini will be able to plan your vacation with a much more granular level of detail, according to Hsiao. The idea is you'll be able to plug in all your specific demands (you like to hike, you hate it when it's too hot, and you're allergic to shellfish) and Gemini will return a detailed itinerary. Chatbots can already do this type of thing, but, if permitted, Gemini will have access to your flight information, travel confirmations in Gmail, and perhaps your hotel, and can use this to inform its answers.

"An AI assistant should be able to solve complex problems, should be able to take actions for you, and also feel very natural and fluid when you engage with it," Hsiao said.

Much of what agents will do is remove steps and shorten tasks. Google is also thinking about this when it comes to Search, as the company evolves its most precious product to spit out answers made using generative AI.

It's rolling out a custom version of Gemini built specifically for Search, combining its knowledge of the web with the AI model's multimodal abilities and giant context window.

Liz Reid, Google's head of Search, showed an example of Google Lens being used to take a video of a record player. The user asks Gemini why the arm isn't staying in place. Google responds with exact instructions for that specific turntable.

As ever, Google has some confusing branding to solve: Lens, Assistant, Gemini, Astra. Ultimately, though, a lot of this eventually merges together. On Tuesday, Google gave us clues on what this logical conclusion will look like.

Or, as Reid put it: "This is a way for Google to do the searching for you."