AI models read and write text in small chunks called tokens, and standard models write them one at a time. Nvidia researchers have built a design, called TiDAR, that produces four to six times more tokens per second. In their paper, the quality of its writing stays close to that of the leading one-at-a-time models.
Running a model to produce answers is called inference, and the one-at-a-time process is its bottleneck. Each token has to wait for the one before it, which caps how much text a model can produce each second. The wait also leaves the chips that run the model partly idle, and for companies serving heavy traffic, that idle time is a direct cost.
Another approach drafts several tokens at once, a method known as diffusion, and earlier research showed that it can produce text faster. Its weakness has been accuracy. Diffusion models have struggled to match established models on long or precise text, which left companies choosing between speed and reliability.
Nvidia’s answer is to use each method where it works best, so TiDAR uses diffusion to think and one-at-a-time writing to talk. The diffusion stage quickly drafts several possible next tokens. A second stage then picks the final output, using the same one-at-a-time logic as standard models.
In the paper’s tests, TiDAR beat diffusion-only models on both speed and quality. It also beat speculative decoding, a common speed-up technique that many companies rely on. That technique’s benefits taper off as models grow larger. TiDAR’s drafting can cope better with that growth, because it puts more of each chip’s spare capacity to work.
The business effect is shorter waits and smaller bills. When text arrives that much faster, delays drop in customer service tools, developer tools and internal systems. Slow tools become responsive, and people use AI more when the delays disappear. A fixed workload also needs less computing power, which means fewer hours of chip time, lower cloud costs and shorter queues at peak times.
Those savings depend on TiDAR moving from a research paper into the systems companies run. Until then, the speed-up rests on the tests in Nvidia’s own paper.
