Wanted: Human experts to help train AI

About a year ago, an advertisement caught the attention of Ashleigh Ruane, a PhD student in physics at the University of Cambridge. The ad was simple but unusual: Teach AI about physics.

Curious, she clicked. She learned that experts across fields—from physics and finance to healthcare and law—were now being paid to help train AI models to think, reason, and problem-solve like domain specialists. She applied, was accepted, and now logs about 50 hours a week providing data for Mercor, a platform that connects AI labs with domain experts.

Ruane is part of a fast-growing cohort of professionals who are shaping how AI models learn. According to Freelancer, thousands of new AI data training and annotation roles have appeared on their marketplace, with most of the growth taking hold in just the past 18 months. These roles range from highly technical expert tasks, like evaluating complex reasoning or diagnosing model errors, to nuanced judgment calls that large models still struggle with.

“We’re entering a really interesting time period,” says Freelancer CEO Matt Barrie. “AI models need more and more data. We’re seeing professionals from every field in every part of the world taking part in this AI data training work.” 

The trend raises bigger questions: If AI models have already been trained on the open internet and vast corporate datasets, why do they still need human experts? What exactly are these experts doing? And how long will this new kind of work be around?

AI has ‘read the whole internet’—and still needs real experts

There’s a common assumption that today’s largest AI models already know everything they need to know. After all, they’ve been trained on millions of books, articles, papers, and posts. But industry leaders say domain experts are now more important than ever.

“Models trained on the entire internet can get you to an 80% answer, but in legal or tax, 80% isn’t useful,” explains Joel Hron, CTO of Thomson Reuters. “Our customers demand a high level of accuracy and trust. Leveraging experts ensures accuracy to the highest degree that we can.”

Ana Price, vice president of supply at Prolific, which provides human data for AI labs, agrees that experts are becoming even more important as AI models move into regulated, high-stakes domains. 

“The demand for human expertise and domain specific feedback from AI models is growing and growing and growing,” says Price. “As these models have gotten bigger, the errors are becoming harder to spot. Real expertise is needed to judge the substance of what models are producing, and not just the surface level correctness.”

In other words, the internet alone is not a substitute for structured professional knowledge. The more organizations rely on AI for serious, high-stakes work, the more they need experts to show models how real professionals think.

What expert AI trainers actually do

Linda Yu spent the last decade as an investor, deploying $4 billion of investments into technology enabled businesses. She started working with Mercor as an expert contributor a year ago, where typical projects involve coaching AI models to think like an investment professional.

“My role as a domain expert is to evaluate whether the model response is not just technically correct, but whether the complex reasoning behind the response is accurate—including assumptions the model made, where it may have overreached, where it missed, and what a better answer would be,” shares Yu. “The work feels less like training an AI model, and more like mentoring a junior analyst.”

Experts like Yu say the work varies from project to project, and is being applied across industries from law, medicine, engineering, and beyond. Participants are typically paid hourly—$85 per hour on average—and may be asked to evaluate a model’s reasoning on a technical question, rewrite incorrect answers into correct, step-by-step explanations, and compare multiple model outputs and choose which best reflects real-world practice. 

The output isn’t generic content, but high-fidelity reasoning data designed to shape how AI systems operate.

AI interviewers interviewing AI trainers

The work requires real expertise, which means AI labs need data from experts who are vetted. To assist with the vetting, some platforms rely on AI interviewers to assess the actual expertise of potential AI trainers.

“Experts jump on a call, and they interview with AI,” says Arsham Ghahramani, founder of Ribbon, an AI interviewer with more than 500 customers, including an AI training data provider who is interviewing more than 15,000 experts a month. “You’ll likely be asked the best interview questions you’ve ever been asked.” 

AI interviewers assess experts for signals that would indicate red flags around expertise, like irregular response cadence, whether they respond naturally, and of course, whether they have the required expertise for a given domain. 

“It was actually my first interview with not a real person,” says Yu. “It scanned my resume and came up with really relevant questions. After each answer, the AI interviewer acted like a real person and summarized what I said and asked a question that was a natural extension of our conversation topic. I was fascinated by the technology.” 

AI now evaluates the humans teaching it, a reflection of just how far people have advanced model capabilities.

The ‘last mile of information’ still belongs to humans

One of the clearest explanations for why expert data remains essential comes from Mark Quinn, senior director of AI operations at Pearl and former head of Waymo engineering operations. He draws a connection between today’s AI challenges and autonomous driving.

“At Waymo, we worked towards the last mile of autonomous mobility. Now, we’re working towards the last mile of information,” Quinn says. “Even though AI systems are being developed to close the last mile of information, the reality is that people may still prefer human expert validation if they need an answer on what to do if their dog ate some chocolate.”

The metaphor resonates across the industry. Even as models get smarter and larger, there’s a world full of edge cases—situations that require judgment, ethical reasoning, or domain-specific logic that isn’t easily captured in general datasets.

Some leaders believe the last mile will shrink but never disappear entirely.

Hron of Thomson Reuters notes, “The base models still have a long way to go to be truly deep. Expert systems and expert knowledge will help models climb to the next level.”

Price of Prolific adds, “We’ve only scratched the surface in terms of what AI can do. Humans are a critical piece of the puzzle, especially in niche domains.”

In other words, the future isn’t about replacing experts. It’s about scaling the expertise that’s essential to making AI models better and safer.

A new kind of knowledge work

For Ruane, the physics PhD student, expert data work has become a significant source of income. She recently accepted a full-time position, but notes that her new job will only be 38 hours per week—leaving time to continue contributing to AI training projects.

What she’s experiencing is quickly becoming common: skilled professionals treating AI training work as a supplemental career path, flexible side hustle, or even full-time job.

The work plays an increasingly central role in how AI systems operate. As models get more capable, the value of real-world expertise is being redefined, not diminished.

Experts aren’t just using AI. They’re teaching it how to reason, think, and act like an expert.

View the full article