AI is consuming the output of entire power plants to answer people’s questions. Chinese researchers have built a chip that targets where most of the waste happens. The waste starts with an exhausting loop that a computer repeats every time it does anything, whether or not it runs AI.
A computer grabs data from memory and runs it over to the processor, which does the calculation and sends the result back to memory. The loop repeats millions of times per second. That is why machines overheat under heavy load, and why AI computing keeps running into limits on power and cooling.
This back and forth is known as the von Neumann bottleneck. Chips have no trouble calculating, but each calculation pays a tax in energy and time to move the data in and out. When the work is data heavy, the movement becomes the work. The limit is physical, so the biggest lever is a design that moves less data.
The researchers’ design goes further and stops moving the data at all. It uses an approach called in-memory computing, in which the memory that stores information also performs the calculation. As a result, the data does not have to make the trip to the processor and back.
Their chip, described in Nature Electronics, applies that approach to AI, whose basic math is calculation across large tables of numbers. The chip does that math inside the tiny memory cells that each store a single bit, and the paper reports clear performance figures tied to the hardware design.
Its magnetic memory is what makes the design credible. Data is stored as tiny magnetic directions rather than electrical charges, and it stays there when the power is off. Switching a direction changes the bit, so the same memory can both hold data and calculate with it.
With less data in motion, more of the computing could happen on people’s own machines, and fewer steps would need a connection to the cloud. That makes in-memory computing a power strategy for AI, and a path to running more capable models on tighter power budgets, including on phones and laptops.
