Google Research says it has found a way to make AI models up to 8x faster on Nvidia’s H100 chips, the high-end AI processors used in data centres. Its method, TurboQuant, shrinks the short-term working memory a model needs every time it answers. Google reports that it cut this memory by at least 6x, while scores on the standard tests used to compare models stayed close to the original model’s. That memory is one of the biggest cost pressures in modern AI, and the pressure may begin to ease.
TurboQuant makes a model cheaper to use once it exists, and for most companies that is where the AI bill compounds. Building a model, a step called training, is a one-time run, while using it is a daily cost across employees, products and customer interactions. The underlying paper draws the same line. TurboQuant leaves training, with its heavy memory needs, as expensive as before.
When AI becomes cheaper to use, businesses do not always spend less. In many cases, they run more of it, because more uses start to make economic sense.
Cheaper running also weakens an old assumption about where AI belongs. Through 2024 and 2025, the best AI stayed in the cloud, because that is where companies could absorb the cost of running it. If the memory needed during use falls this sharply without much loss in quality, some of that work can leave the cloud for the machine beside the employee. That machine starts to look more capable than the old cloud-first view assumed.
A separate study of Apple’s own chips points the same way. It found that Apple’s hardware can run very large models efficiently on the device itself. Companies could then choose where to run AI: the cloud, local machines or smaller internal setups. Running AI closer to the people using it could cut its cost and speed up responses.
The heaviest training runs belong in the cloud, where shared data centres offer a scale that local machines cannot match. TurboQuant’s results also come from Google’s own tests rather than from companies running it in their daily work.
