Inside AI Tokenomics: How to Profitably Turn Tokens Into Business Value | NVIDIA AI Podcast Ep. 299

Shruti Kopakar defines tokenomics as four pillars: token utility, token supply, token demand, and token monetization. Token utility is shaped by the intelligence embedded in a token and the speed at which it arrives, measured as tokens per second per user. Token intelligence depends on the model chosen and the context length allowed, with longer context generally improving accuracy, though excessive context can degrade output quality. More complex models produce more valuable tokens in absolute terms, but a use case may not require that level of intelligence, making the added capability useless. Domain-specific applications can use fine-tuned small language models that match or exceed large model accuracy for narrow tasks.

Token demand starts with users multiplied by requests per user per time period multiplied by tokens per request. Reasoning models introduce thinking tokens never seen by the end user, and deployments can cap how many thinking tokens are allowed per interaction. Agentic workflows are a significant demand multiplier because multiple turns and loops occur within a single use case, and even a few milliseconds of additional latency per turn can accumulate into several seconds of total delay. KV cache acts as short-term memory, storing previously computed input requests to reduce effective token demand. Demand variability also includes time-of-day patterns, seasonal spikes such as retail holiday surges, and projected user growth.

Kopakar argues that input metrics like cost per GPU hour or flops per dollar do not reflect actual delivered token output and are insufficient for evaluating AI infrastructure ROI. Cost per token incorporates both input costs and output delivery and is the more revealing metric. NVIDIA Blackwell carries approximately twice the hourly GPU cost of Hopper but delivers twice the flops per dollar, 50 times more tokens per watt, and 35 times lower token cost on the NVL72 configuration. The 50 times tokens per watt advantage is not captured by flops per dollar, which illustrates precisely why cost per token is the more useful measure.

Kopakar distinguishes co-design from integration by defining co-design as building multiple system components simultaneously from the ground up toward a single outcome, specifically lowest token cost. NVIDIA's co-design spans compute, memory, storage, networking, and software from CUDA kernels to serving software, and extends to silicon partners, OEMs, cloud providers, and open source frameworks. Blackwell NVL72 is well suited for mixture of experts models because it optimizes inter-GPU communication. Techniques including disaggregated serving via Dynamo and wide expert parallelism using TensorRT, VLLM, and SGLang reduce cost per token for those models. The Vera Rubin platform, which contains seven chips, is built for agentic AI and requires the Rubin GPU for LLM acceleration, Groq 3LPX for ultra-low latency, and the Vera CPU for tool calling and code sandboxing.

Software is described as the difference between what a spec sheet shows and what is actually delivered in token output and cost. Stacking all software optimizations simultaneously produces 50 times more throughput on Blackwell and 35 times lower token cost, and piecemeal optimization is insufficient to reach those gains. VLLM and SGLang inference runtimes have seen 8 times more performance in approximately six months, with gains attributed to the broader ecosystem including open source frameworks, partners, customers, and the developer community. The CMX platform using Bluefield DPUs with SpectrumX enables KV cache offloading and retrieval for matching incoming requests.

Kopakar applies Jevons paradox to GPU efficiency, arguing that lower cost per token unlocks new use cases that absorb efficiency gains rather than reducing GPU demand. When generative AI lowered cost per token, demand for more GPUs and tokens increased because test time scaling and reasoning became viable. The agentic revolution is described as another inflection point enabled by deploying mixture of experts and reasoning models at lower cost per token.

Token monetization requires understanding the cost to produce a token, the token's utility for value-based pricing, and the distribution of demand across token value tiers. Four primary business models exist for turning tokens into business value: selling tokens directly, building AI-native products, enhancing existing products with AI, and improving internal operations and employee productivity. Companies selling tokens directly include Fireworks, Base10, TogetherAI, and Deep Infra, while AI-native companies building products from the ground up include Perplexity and Cursor, and companies enhancing existing products include Shopify, Airbnb, and Adobe. Kopakar recommends that business leaders start with the final outcome and customer need before making any infrastructure or tokenomics decisions, defining token utility and demand first, then token supply, and then monetization strategy.