From data scarcity to abundance: Dr. Fei-Fei Li on how AI actually got here

We hosted Dwarkesh Unplugged: a live interview between Dwarkesh Patel, host of the Dwarkesh Podcast, and Dr. Fei-Fei Li, co-founder and CEO of World Labs and the researcher often called the Godmother of AI. It was a conversation about how modern AI actually came to be — told by someone who was in the room for most of it. A few ideas stuck with us long after the room cleared out.

‍

The bottleneck was data, not cleverness

The through-line of the evening was deceptively simple: the field was held back not by a shortage of good ideas, but by a shortage of data. Li's own turning point came from asking what it means for a machine to see — not to collect pixels or RGB values, but to make sense of what's in front of it.

The machine learning of the mid-2000s ran on tiny datasets and increasingly elaborate models. Li's insight was that this had it backwards. Humans and animals learn from a massive, continuous stream of experience, and no amount of model-tweaking could substitute for that. So instead of building a cleverer algorithm, she built a bigger world for algorithms to learn from. Her team spent three years assembling ImageNet — 15 million images across 22,000 categories — and then open-sourced it. The public challenge that followed narrowed things to a curated 1,000 categories across a couple million images.

The irony she pointed out: the architectures were already sitting on the shelf. Convolutional neural networks date back to 1989. But neural networks weren't fashionable in either computer vision or NLP at the time, and the people training them didn't have data to feed them. When the two finally met, everything changed.

Breakthroughs are usually convergences

That meeting happened in public. For the first couple of years, the ImageNet challenge was won by support vector machines. Then in 2012, Geoff Hinton's team entered a convolutional network — AlexNet — and the field pivoted.

What made it work wasn't a single genius stroke. It was a stack of things landing at once: a model architecture that had already proven itself, a couple of training tweaks like dropout and ReLU, and Alex Krizhevsky physically wiring two GPUs together to get enough parallel compute. Big data, a high-capacity model, and a new kind of chip — none sufficient alone, decisive together.

The progress after that was steep. AlexNet's error rate was close to 20 percent; by around 2017, computer vision methods were matching or beating human performance on the benchmark. And there's a nice footnote about what "human performance" even meant: Andrej Karpathy, then a student in Li's lab, hand-labeled 1,000 categories himself to establish the baseline, landing around a 4–5% error rate.

AI turned into a team science

One of the sharper points of the night was about how the practice of research has changed. In the 2010s, a single grad student with one GPU could try a new activation function or a regularization trick and move the field. Li argued that era is largely over. Modern frontier work is systems-building — tens of ideas converging on one ambitious model, tested by teams with enormous compute.

Her analogy was high-energy physics. A century ago, a discovery could come from Rutherford alone in a Cambridge lab with a glass tube. Today it takes the hundreds of scientists and engineers at CERN. AI, she suggested, has crossed the same threshold — which is worth sitting with if you're a builder wondering where individual contribution still fits. (Her own lab is a good answer: her former students went on to co-found World Labs with her and to lead NVIDIA's robotics team, and what she credited most wasn't raw talent but intellectual fearlessness.)

We overcorrected on data — and that reframes the frontier

The scaling law — the systematic push to make models bigger and feed them more — only became widely appreciated around 2020, near the GPT-2 era. And it worked so well that the original problem inverted. Li recounted a conversation with mathematician Terence Tao, who described the shift from information scarcity to information "obesity". A frontier model now trains on roughly a millionfold more tokens than a person will encounter in a lifetime.

That abundance is genuinely superhuman in one dimension: recall. Tao's estimate, which Dr. Li shared, is that today's models might help solve around 5% of open math problems — precisely the ones whose solutions already exist in known methods that no single human happens to remember. That's real value. But it also draws a sharp line. What a model cannot do is produce an idea that has never been written down. The empty space in the training data is the genuine frontier of intelligence, and it's exactly where models can't yet go.

Dr. Li reached for one more comparison to make the point: planes and birds both obey aerodynamics, but their engineering diverges completely, and so do their capabilities. Machines and minds are diverging the same way. One is not a replacement for the other.

Which is why the closing question — team human or team AI — didn't feel rhetorical. Dr. Li's answer was that she's still team human. After an hour tracing how far the machines have come, it landed less as sentiment and more as a description of where the interesting work still is.

‍

Special thanks to Dwarkesh Patel and Dr. Fei-Fei Li for joining us at SFJazz for Dwarkesh Unplugged.