Snap’s Secret to Processing 10 Petabytes a Day: GPU-Accelerated Spark | NVIDIA AI Podcast Ep. 298

Snap's experimentation platform processes more than 10 petabytes of data daily under a morning SLA, delivering A-B test results to developers, product managers, and data scientists before the start of each workday. The platform serves close to 940 million monthly active users and approximately 500 million daily active users, and incorporates statistical methods including heterogeneous treatment effects detection, variance reduction for power-user distribution imbalances, and sample size mismatch correction for user churn during experiments.

Prudhvi Vatala, head of engineering platforms at Snap, framed the core engineering problem as flattening the cost-scaling curve rather than continuously adding CPU capacity as user count and feature complexity grew.

Snap discovered the NVIDIA RAPIDS Accelerator for Apache Spark through a blog post citing a 3.6x performance improvement and roughly 50 percent runtime reduction for PySpark workloads, then ran its own benchmarks on Google Cloud Dataproc across three job types. Join-heavy jobs with repartitions and shuffles achieved more than 3x speedup, union jobs achieved close to 2x, and aggregation jobs achieved just over 1.5x, with the lower aggregation gain attributed to CPUs already being efficient at that workload type.

The full production migration reduced job costs by approximately 76 percent, cut required CPU core count by roughly 62 percent, reduced memory footprint by approximately 80 percent, and eliminated around 120 terabytes of disk and memory spill from Snap's pipelines. The migration from prototyping to full production took approximately eight to nine months, with a three-way partnership among Snap, NVIDIA, and Google Cloud cited as essential to that timeline.

Migrating one production shard using 300 GPUs produced strong initial results, but on-demand GPU capacity on Google Cloud became a bottleneck when plans expanded to a 10-shard migration requiring approximately 3,000 GPUs. Snap resolved this by identifying idle online inference GPU capacity available between roughly 1 a.m. and 5 a.m., exploiting the cyclical usage pattern of Snapchat's global audience, and migrating its batch Spark workloads to a Kubernetes-based Spark runtime on GKE to access those inference GPUs.

Because the inference GPUs were tied to Google Kubernetes Engine, Snap built a new data platform on GKE that any internal team could use. Preemption logic ensures that a spike in user traffic requiring online inference immediately reclaims GPU resources from batch workloads, and a graceful fallback chain drops workloads to CPUs if GPU capacity is unavailable, then to Dataproc clusters if shared GKE resources are constrained.

Snap migrated its hourly pipeline to GKE plus Spark RAPIDS before the daily pipeline because speed improvements there carried greater operational value for guardrailing user experience. NVIDIA Spark RAPIDS required zero code changes to existing Spark jobs, and NVIDIA Ether provided out-of-the-box Spark tuning that maintained consistent parameter settings across the GPU, CPU, and Dataproc fallback environments.