Inference Bottlenecks Push Optical Interconnects Into AI Chips

The infrastructure challenge in artificial intelligence is shifting. While training massive models still requires enormous compute resources, inference—the phase where deployed models respond to user queries—is emerging as a distinct bottleneck defined not just by processing power but by memory bandwidth, network latency, and energy consumption.

"Inference is the moment when users use the model," Peter Ossieur, portfolio director at imec and professor at Ghent University, told EE Times. "So, it needs to be reactive."

That responsiveness requirement creates a fundamentally different systems problem. Inference must generate tokens rapidly and continuously for many concurrent users, often involving extended context windows, retrieval-augmented generation, and multi-agent architectures. The result is intensive communication between accelerators that increasingly cannot be solved with copper alone.

Scale-up networking becomes the constraint

The critical distinction lies between scale-out and scale-up networking. Scale-out connects systems across data center racks and already relies heavily on optical transceivers. Scale-up networking, by contrast, links GPUs or accelerators so tightly they function as a unified processor from the application's perspective.

Today's scale-up fabrics remain largely copper-based and rack-constrained. But as AI systems coordinate hundreds or thousands of accelerators simultaneously, that architecture hits physical limits. Frontier models routinely exceed single-GPU memory capacity, forcing workloads to split across multiple chips with constant inter-accelerator traffic. The problem intensifies as AI systems become more agentic, with specialized models exchanging data among themselves and with larger foundation models.

For imec researchers, this creates a system-level interconnect challenge requiring co-design across software, architecture, and physical layers. High-radix switches and potentially optical circuit switching become necessary to minimize communication hops when thousands of GPUs must coordinate with minimal latency.

Beyond co-packaged optics

Co-packaged optics—integrating optical engines directly into processor packages rather than using separate modules—represents the industry's current direction for bringing photonics closer to compute. Shortening electrical paths improves both bandwidth and energy efficiency.

But Imene Jadli, portfolio manager for optical interconnect at imec, argues co-packaged optics won't suffice for future AI systems. The constraint is power. A future processor requiring approximately 250 terabits per second of bandwidth could see its co-packaged optics alone consume around 1.25 kilowatts—on top of multi-kilowatt processors. That creates unsustainable thermal and packaging challenges.

Imec's proposed solution is 2.5D optical I/O, integrating optics at the interposer or substrate level. The approach uses a "wide and slow" architecture: many moderate-speed optical lanes rather than fewer extremely high-speed channels requiring sophisticated signal processing. This maintains aggregate bandwidth while dramatically reducing energy per bit—potentially dropping optical power consumption below 200 watts in the same system projection.

The longer-term vision extends to 3D optical I/O, where photonics becomes native to the 3D compute stack, potentially enabling optical communication between high-bandwidth memory and stacked processors. But moving optics closer to silicon amplifies integration challenges across materials compatibility, manufacturing yield, optical coupling, and thermal management.

Imec is exploring materials including barium titanate and III-V compounds alongside silicon photonics, seeking integration schemes compatible with CMOS manufacturing. Jadli identified thermal management as among the hardest unresolved problems for 3D optical I/O.

Why it matters

As inference becomes the dominant AI workload, the question is no longer whether optical interconnects will play a role in AI infrastructure, but how deeply integrated with processors they must become. The bandwidth and power requirements of large-scale inference may force the industry to solve photonics integration challenges that have remained theoretical—or accept fundamental limits on AI system scaling.

These details were first reported by EE Times.