AI agents fill headlines, product demonstrations and corporate strategies. The promise is software that can plan, carry out and deliver complex work on its own. When researchers handed leading agents real freelance jobs worth $143,991, the work that met professional standards was worth $1,810. Counted job by job, the agents’ success rate was only 2.5%.
The test, called the Remote Labor Index, was released in October 2025 by the Center for AI Safety and the AI company Scale AI. Its assignments were the kind of work companies already pay freelancers to do, including 3D modeling, video editing, building dashboards and graphic design.
That result cuts against the story of rapid progress, even as academic tests keep showing big improvements. Stanford’s 2025 AI Index reported large gains in reasoning at graduate level and in making sense of images and other material alongside text. Those tests measure what AI can do in isolation, under controlled conditions. They cannot show whether an agent turns out consistent, high-quality work on the vague, many-step tasks that businesses face.
When the agents in the freelance test failed, they failed badly. Many handed in work that missed the brief entirely. A request for a promotional video might come back as a three-second clip, and a product design as a stick-figure sketch. Technical standards often went unmet as well, which left the work unusable.
For a company hoping to replace freelance or junior staff with agents, that is a serious risk. The software may be cheap, but the hours spent reviewing, correcting or redoing its output can wipe out the savings. With so little of that output usable, an agent adds inefficiency and drags productivity down. Sound use of agents therefore depends on pilot tests, output measured against human standards and clear thresholds for reliability, all in place before any large investment.
Agents do far better with people beside them. AI then speeds up the work or automates specific steps, while people stay in charge of quality and judgment. In an evaluation at Upwork, the freelance marketplace, projects where human professionals guided the agents and checked their work reached success rates of up to 70%.
