Yann LeCun: Meta ‘fudged a little bit’ when benchmark-testing Llama 4 model

Yann LeCun, Meta’s outgoing chief AI scientist, says his employer tested its latest Llama model in a way that may have made the model look better than it really was. 

In a recent Financial Times interview, LeCun says Meta researchers “fudged a little bit” by using different versions of Llama 4 Maverick and Llama 4 Scout models on different benchmarks to improve test results. Normally researchers use a single version of a new model for all benchmarks, instead of choosing a variant that will score best on a given benchmark. 

Prior to the launch of the Llama 4 models, Meta had begun to fall behind rivals Anthropic, OpenAI, and Google in pushing the envelope. The company was under pressure to reassert Llama’s prowess, especially in an environment where stock prices can turn on the latest model benchmarks.

After Meta released the Llama 4 models, third-party researchers and independent testers tried to verify the company’s benchmark claims by running their own evaluations. But many found that their results didn’t align with Meta’s. Some doubted that the models it used in the benchmark testing were the same as the models released to the public. 

Ahmad Al-Dahle, Meta’s vice president of generative AI, denied that charge, and attributed the discrepancies in model performance to differences in the models’ cloud implementations. 

The benchmark-fudging, LeCun said, contributed to internal frustration about the progress of the Llama models and led to a loss of confidence among Meta leadership, including CEO Mark Zuckerberg.

In June, Zuckerberg announced an overhaul of Meta’s AI organization, which included the establishment of a division called Meta Superintelligence Labs (MSL). Meta also paid between $14.3 billion and $15 billion to buy 49% of AI training data company Scale AI, and tapped Scale’s CEO, Alexandr Wang, to lead MSL. On paper, at least, LeCun, who won the coveted Turing Award for his pioneering work on neural networks, reported to the 28-year-old Wang. 

LeCun told FT’s Melissa Heikkilä that while Wang is a quick learner and is aware of what he doesn’t know, he’s also young and inexperienced. “There’s no experience with research or how you practice research, how you do it. Or what would be attractive or repulsive to a researcher,” LeCun said.

The division LeCun ran at Meta for a decade, FAIR (Fundamental Artificial Intelligence Research), was a pure research organization that picked its own areas of inquiry. An adjacent “applied AI” group worked closely with the lab to find ways to use the research in Meta’s own products.

But the organizational changes weren’t the only reason LeCun wanted to leave Meta. He has long expressed doubts that the current thrust of Meta’s AI research—large language models—will lead to human-level intelligence because such models can’t learn fast and continuously. LLMs can learn a certain amount about the world through words and images, but the models of the future will also have an understanding of the real world through physics. 

And it’s those “world models” that LeCun hopes to invent at his new company, Advanced Machine Intelligence. LeCun will act as executive chair, which will allow him to spend much of his time doing research. Alex LeBrun, CEO of French healthcare AI startup Nabla, will become CEO of AMI.

“I’m a scientist, a visionary. . . . I can inspire people to work on interesting things,” LeCun told Heikkilä. “I’m pretty good at guessing what type of technology will work or not.”

View the full article