Inside Instacart's AI-Powered Smart Shopping Cart | NVIDIA AI Podcast Ep. 302

Instacart acquired Caper for 350 million dollars and has deployed Caper carts across 100 cities, tripling year over year, with roughly 20 percent of all Wakefern stores equipped. International deployments include Coles in Australia and a announced rollout with Morrisons in the UK.

The Caper cart is a sensor fusion system built around an NVIDIA Jetson board in every cart. It combines a weights-and-measures certified scale covering the entire basket, multiple cameras facing the basket and the shelf, and location sensors using a SLAM approach augmented by visual shelf data. Side-facing cameras serve a dual purpose, resolving SLAM ambiguity between adjacent aisles and identifying what is on the shelf to inform recommendations. Edge computing is central to the design because consumer expectation for responsiveness is in the hundreds of milliseconds, faster than cloud-based systems that typically respond in seconds. An edge encoder processes sensor signals locally while longer session data is analyzed in the cloud, with outputs combined in a shopping experience decoder.

Camera input alone is insufficient because items can be pulled out at varying speeds, the camera can be blocked, and the basket can become full. The weight sensor acts as a ground truth layer while camera inputs inform it. Shopper behavior such as leaning on the cart, arms moving in and out, and cart movement over bumps complicates scale-based detection. The system is also designed to handle spotty Wi-Fi and poor cell reception common inside grocery stores. Grocery environments compound recognition challenges through variable lighting, tens of thousands of SKUs, catalog differences store to store within the same retail banner, seasonal SKU changes, and subtle size differences between items. Most retailers lack an accurate planogram, and inaccurate planograms cause wrong recommendations that erode user trust and reduce engagement with the cart screen over time.

Retailers are seeing double-digit sales lift from Caper carts. A did-you-forget feature that surfaces items a customer normally buys but skipped on a given trip drove nearly a one percent absolute increase in sales lift, and a new recommendation algorithm delivered more than one percent absolute improvement on top of prior gains. The did-you-forget recommendation is personalized to individual purchase history rather than being a generic aisle-based or promotional suggestion. The top reasons customers use Caper carts are running total tracking, deals and discounts, and the convenience of bagging as they go. Customers who bag as they shop have shifted from store-provided plastic bags to reusable bags because they no longer need to unload items at checkout.

Instacart draws on over 1.6 billion lifetime online grocery delivery orders and a two billion item catalog to inform in-store recommendation algorithms. The company is building a grocery foundation model that ingests online orders, item catalog data, and in-store data including clickstream behavior such as where users pause and what causes them to add or remove items. More than half a million Instacart shoppers enter stores every day and contribute to real-time store understanding. Caper cart side-facing cameras continuously scan shelves to build shelf inventory understanding and can proactively notify store employees about out-of-stock items, with some store models refreshing as frequently as every 15 minutes. Instacart also announced StoreView, a technology where Instacart shoppers scan store shelves with their phones to build shelf inventory understanding.

Instacart connected the Caper cart to its deli digitization product so shoppers can order from the deli directly from the cart, and integrated electronic shelf labels so carts can light up shelf tags to locate items. Checkout can be completed on the cart via tap payment or Apple Pay, or transferred to an existing payment terminal for cash or alternative payment methods. Kroger, Sprouts, and Quigga announced rollouts of Cart Assist, which lets users plan a store trip online and sync that plan to the Caper cart in store. Users interacting with an AI assistant for grocery planning provide complex natural language requests such as multi-week meal plans with budget and allergy constraints, behavior that differs from traditional online grocery browsing. Migrating ad workflows from CPUs to GPUs reduced latency significantly and increased click-through rates in Instacart experiments.

The 3D item reconstructions and store mapping data generated by the Caper cart could eventually support robotics workflows in stores. David McIntosh described Instacart's five-to-ten-year vision as one where customers do not have to think about whether they are shopping in-store or online, with both unified into one mode powered by a continuously learning AI system. A near-term version of that hybrid experience could involve a pre-picked partial order waiting in a personalized cart while the customer selects remaining items such as produce themselves. McIntosh noted that digitizing the store makes it measurable and allows it to be optimized like software, and that in-store behavior captured by smart carts feeds back to improve online grocery recommendations and experiences.