Traditional forecasting still beats AI for the most extreme weather

AI is being touted as the future of weather forecasting—faster and more precise. But new research shows a major blind spot: it often fails at predicting extreme weather. Traditional physics-based models still do better.

“They do perform well on a lot of tasks, but for very extreme events—that are the most important for society—they still struggle,” says Sebastian Engelke, a statistics professor at the University of Geneva and one of the authors of a new study in Science that pitted some of the leading AI weather models, including GraphCast and Pangu-Weather, against a database of recent extreme events.

For record-breaking heat, like a heat wave in Siberia in early 2020 that led to wildfires and melting permafrost, AI predictions tend to underestimate high temperatures. (The heat wave would have been almost impossible without climate change; another study found that global warming made it 600 times more likely to occur.) They’re also less accurate than older models at predicting extreme wind or record-breaking cold.

That’s because they’re trained using decades of past data. “They try to empirically understand, if I see a certain type of weather today, what is the weather tomorrow?” says Engelke. “Essentially, they are reproducing what has happened in the past. If we’re looking at extreme weather, and especially record-breaking events, then this has not been observed in the past. It’s really the lack of information in their training data that makes it almost impossible for them to forecast it.”

The study looked at models a year ago, so they’ve already improved; some have added probabilistic models that predict multiple outcomes to try to become more accurate. But the fundamental problem still exists, because they’re still based on training data from the past. Traditional physics-based forecasting uses complex mathematical models to represent the physical world instead, and can more readily adapt to new conditions. (Traditional models aren’t perfect at predicting extreme weather, either, but still do a better job.)

For more typical weather forecasting, or extreme weather that isn’t wildly outside the range of past events, AI can outperform traditional models. When Nvidia released its AI forecasting model Atlas earlier this year, it ran a study showing how well it performed on an extreme event it had not been trained on: Storm Dennis, a rapidly intensifying cyclone that impacted the U.K.

“You can see just clearly by visualizing the magnitude of the wind and the magnitude of the pressure gradient that the model was able to capture realistically intense wind events and really intense cyclones that cause damage,” says Mike Pritchard, director of climate simulation research at Nvidia. The models can also accurately predict the path of hurricanes. They’re already used alongside traditional models by weather agencies, weather data companies like the Weather Company, and insurance companies.

Researchers are exploring ways to improve the accuracy of forecasting the most extreme of extreme weather. One option, for example, is to add data to training sets that shows what record-breaking events could look like. “There’s ways to kind of coerce physics weather models to produce especially extreme events, and you can sprinkle these into the training data set alongside reality in order to prepare the weather models to extrapolate,” says Pritchard.

The technology is rapidly improving. Engelke argues that as new models roll out, they should all undergo the type of testing laid out in the new study. “Most of these models come from tech companies, and benchmarking and independent evaluation [is important] because they’ll have a critical impact on our lives,” he says.

For now, it’s likely that traditional forecasting won’t go away anytime soon.

View the full article