Microsoft's Rani Borkar: AI Infrastructure Needs a Yield Mindset

The semiconductor lesson for AI infrastructure

Microsoft's head of Azure infrastructure is calling for a fundamental shift in how the technology industry thinks about artificial intelligence investment. Rani Borkar, who oversees hardware development and deployment for Microsoft's cloud platform, argues that AI needs to adopt the semiconductor industry's decades-old discipline of yield—measuring success not by inputs but by useful output produced.

Writing on the official Microsoft blog, Borkar notes that while AI adoption has spread faster than the internet or smartphones, global penetration stands at just 18% of the working population, with most usage limited to chat interactions. As systems evolve toward agentic workflows that reason, plan, and execute complex tasks, the infrastructure equation changes dramatically: a single agentic task can consume more than 3,400 times as many tokens as a typical chat interaction.

The industry's default response has been to add more of everything—more silicon, memory, power, and connectivity. But Borkar contends this approach puts the sector on a treadmill that moves only as long as resources keep flowing in.

Why it matters

The AI infrastructure buildout represents capital deployment on a scale once reserved for nations, with power consumption measured in gigawatts. If efficiency gains don't keep pace with capability demands, cost and energy constraints will limit who can access AI tools—turning what could be a broadly enabling technology into a resource available only to well-funded organizations. The yield framework offers a concrete alternative to simply scaling up: optimize across the entire stack to extract more intelligence from existing resources.

Cross-layer optimization in practice

Borkar's team has learned that the biggest constraints are rarely solved in the layer where they appear. Microsoft's experience building the Azure Maia AI accelerator and Cobalt CPU platforms demonstrates this principle across three critical domains.

For memory, the company addressed inference bottlenecks not by simply adding capacity but through coordinated changes in model architecture, compression techniques, software memory management, silicon optimization for data movement, and compiler improvements. Together, these interventions increased the useful intelligence extracted from the same memory resources.

In networking, Microsoft designed Maia's platform from the outcome backward—efficient inference at fleet scale—rather than starting with existing network architectures. The team built a two-tier scale-up network, integrated NIC functionality directly into the chip, and developed a custom transport layer. The result: scalable performance with less hardware and lower total system cost.

For power distribution, Azure Cobalt 200 implements per-core voltage and frequency controls paired with per-virtual-machine power capping in software. This granular control lets Microsoft run more servers within the same power envelope while protecting critical workload performance.

From tokens to accessibility

Borkar emphasizes that tokens and raw intelligence aren't the finish line. Full yield means AI becomes accessible enough that every person and organization can use it, build on it, and create value—whether that's a scientist accelerating discovery, a clinician identifying early signals, or a small business finding growth opportunities.

At scale, accessibility depends on efficiency. It's the only path to deploying enough intelligence at a cost that allows broad reach.

The breakthroughs ahead will require collaboration across the ecosystem, Borkar argues: hyperscalers and silicon providers, equipment makers and materials innovators, utilities and datacenter operators, model builders and software developers working across traditional boundaries.

These details were first reported by Microsoft in a blog post authored by Rani Borkar, who leads hardware and infrastructure development for Azure.