Scaling Productivity with Ava — Instacart’s Internal AI Assistant
Zain Adil, Kevin Lei and Ada Cohen
Overview
Over the past few months, we’ve been building an internal AI assistant powered by OpenAI’s GPT-4 and GPT-3.5 models called Ava. Ava has seen accelerated adoption at Instacart, with over half of Instacart employees using it every month and more than 900 using it every week. Ava’s web interface, similar to ChatGPT, also has additional features like conversation search, automatic model upgrades and a “Prompt Exchange” that allows users to share and reuse prompts. Users use Ava to write, to review and debug code, to improve comms, to learn faster and to build AI-enabled internal tools on top of our APIs. So, how did we get here?
From Hackathon Project to Internal Tool
Earlier this year, Instacart hosted a company-wide hackathon. While hacking on a few different projects, our team was able to produce nearly twice as many features as initially planned with the help of ChatGPT. ChatGPT, specifically GPT-4, proved to be an amazing companion for brainstorming, coding, debugging, and generating tests. As an engineering team obsessed with productivity, we quickly realized that we had to give ChatGPT access to all our developers, as fast as possible.
At the time, we were working closely with OpenAI and had early access to GPT-4 (the 32K context model) with APIs that had custom data privacy, security, and quota guarantees. We decided to leverage these and quickly built and launched Ava for our engineers!
As part of our initial launch, we prioritized product features that engineers would find the most useful: convenient keyboard shortcuts, single-click code copying and automatic upgrades between GPT-4 models as the conversation context grew. Post launch, we quickly got widespread adoption faster than we expected and noticed that users were spending 20+ minutes per session and were producing and copying a significant amount of code with Ava as a chat companion. With the largest GPT-4 context model, developers were also using Ava to create, debug and review full code files and to summarize documents and ask related questions.
Expanding Ava to users beyond Engineering
Ava took off more quickly than we expected. Post launch, we conducted user interviews and identified two major areas of investment. First, it was clear that not just the Engineering team, but other Instacart employees, including those from Ops, Recruiting, Brand Marketing and HR, were eager to try out Ava. Second, we wanted to enhance Ava’s capabilities by providing it access to our codebase, internal documentation, and data.
To prepare Ava for a broader release, we worked toward making Ava more ergonomic and easier to use rather than building developer-centric features. As we focused on non-engineering users, we knew that starting with a blank text box would be a barrier to entry. To address this, we added some well crafted prompts which we called “templates” that allowed users to quickly start conversations. We also added a full-text conversation search and conversation sharing to allow users to share their conversations with Ava with others.
We knew Ava conversations would primarily be shared over Slack, and we wanted to make sure that shared conversation links would provide users enough context that they could decide whether to follow the link or not. So we added “unfurling” on Slack to provide users a preview of the conversation. This not only increased visibility but also significantly aided adoption and overall product awareness.
The Ava Prompt Exchange
After releasing Ava to the entire organization, we saw that a large proportion of conversations started with our prompt templates. We could have continued to add an increasing number of templates, but setting aside the amount of effort that would entail, our small team simply didn’t have the domain-specific expertise to create great templates for every team’s use case. Instead, we decided that it would be far more valuable to allow our users to create and share templates based on their own needs and experience.
So, collaborating with stakeholders from across the organization, we launched the Ava Prompt Exchange, a library of user-created prompts which can get Ava working on a specific task with just a few clicks. Users can browse popular prompts, search for something specific, or create their own and share them with the rest of the company. And if they find a prompt particularly useful, they can star it so that they can get back to it later.
Bringing Ava Closer to users
By this point, nearly a third of the organization was already using Ava every month, but we wanted to reach more users and make Ava even more accessible. After all, navigating to a specific web page and pasting in context is a lot of friction if you’re trying to do something small. Building a Slack app was an obvious way to make sure that Ava was always a few keystrokes away for our users. But we didn’t want our Slack integration to just be a less-featured clone of the web experience, so we looked for specific features that would be especially valuable within the Slack app.
When we released the first set of templates with Ava, “Fast Breakdown” was one of the more popular templates, summarizing a conversation with facts, open questions, and action items. We saw our users applying this template to meeting notes, emails and, most importantly, Slack conversations.
So “Fast Breakdown” was a clear choice for a first-class feature in our Slack app. In addition to conversing normally with Ava, we allow users to easily summarize a thread or channel by simply typing “@Ava summarize.” The summary is then posted publicly, which makes it easy for other participants to verify that the summary is accurate and note any corrections.
It’s hard to convey just how big of a time-saver this feature is, particularly when you’re called late into a thread and have seventy messages to slog through. Once Ava has summarized a thread, a five-to-ten minute context grind is reduced to just a few seconds of skimming.
Of course, Ava also functions as a normal chatbot on Slack, both in DMs and in the channels which the bot has been invited to. Ava has access to the conversation context, which gives it the ability to infer answers and participate normally in the conversation. When building Ava’s Slack App, we were careful to ensure that the user experience feels similar to chatting with a colleague.
What’s Next?
Although Ava is available on multiple interfaces (Web & Slack), there is still much to do. We see a bright future ahead for Ava in debugging and reviewing code as well as enhancing meetings and incident management. Our next priority is to make Ava even more powerful. The Achilles’ heel of LLMs is the data they are trained/tuned on or have access to, so we are working on knowledge retrieval and code execution, so that Ava can not only access Instacart’s wealth of internal knowledge, but use it to accomplish complex tasks. Levelling up Ava this way will not only immediately benefit our users, but will also serve as a foundation for even more sophisticated future applications.
We also don’t want to be the gatekeepers of Ava’s power. We know that other teams at Instacart will find uses for this technology within their own workflows and processes, and we plan to unlock that potential by exposing Ava’s APIs company-wide.
Collaborating on the Ava project has been an absolutely delightful experience thus far! We want to give special thanks to folks on the team: Ben Bernard, Kye Hohenberger, Mike Deeks, Robert Bowering, Rodrigo Cordeiro, and Greg Lyons. Thank you all for your tireless efforts and remarkable contributions.
