Many people have probably already helped build the data that robots learn from, without realizing it. For a decade, Pokémon Go players pointed their phones at buildings and street corners while chasing Pikachu and other characters on screen. Humanoid robots and self-driving cars, machines built to work away from a screen, need exactly that kind of data.
Leading AI systems write, code and solve complex problems with ease. Asked to move through a room or fold a shirt, the same systems fall apart. That is because this kind of AI, called a language model, trains on text, images and audio. It can read every account of how fabric folds without ever touching cloth.
The main fix has been simulation, where engineers train robots in virtual worlds with set rules of physics. The real world is far messier, full of small variations no engineer coded in advance. Robots trained in those clean settings often fail the moment they meet it.
A different kind of system is getting past that problem. A language model predicts the next word in a sentence, but this system is trained to predict how a scene will change next. It is called a world model. Doing that well means knowing where things are, how they behave and how nearby people or objects are likely to move. Such a model learns that from hours of video and images of real places.
Researchers at Harvard and MIT showed what happens when an AI lacks that understanding. They trained a language model to give directions in Manhattan, and it produced near-perfect routes between almost any two points in the city. Then they closed 1 percent of the streets at random. The share of correct directions fell from near 100 percent to 67 percent. The model had memorized routes without ever building an understanding of the city that adapts when conditions change.
Building that understanding takes data on a scale that is hard to picture, and the Pokémon Go players had supplied it. Their phones took roughly 30 billion images, each tagged with a precise location. Niantic Spatial, an AI spinout from Niantic, built a model of the physical world on them. The model can tell where a robot is, to within a few centimeters, from a handful of camera images. It guides delivery robots through real city streets.
