Large language models struggle with climate policy questions and user pressure
A study by University College London and UNESCO tested 33 large language models on climate questions. The models performed well on simple factual questions but failed when asked how climate policies interact or when users pushed back against correct answers.

Key takeaways
- Large language models score around 90 percent on simple climate facts but fail on policy interaction questions.
- All 15 models tested dropped their accuracy by 30 to 35 percent when users pushed back against correct answers.
- Accuracy fell by seven percentage points in Urdu and ten in Swahili compared with English.
- About half of the questions received different answers from the same model depending on the language.
In this article
What the study tested
Researchers from University College London and UNESCO tested 33 large language models, including frontier models and open-weight models. The models were asked 600 questions drawn from several benchmark sets, including multiple-choice textbook questions, reading comprehension questions based on scientific text snippets, and questions about how confident climate scientists are in different conclusions from IPCC reports.
Where models performed well

Frontier models and larger open-weight models generally scored around 90 percent on the simpler factual benchmarks. Small open-weight models scored in the 70s. The latest frontier models tended to score better than earlier generations of the same model, and larger open-weight models scored better than smaller ones.
Where models failed
Questions about how confident climate scientists are in IPCC conclusions were the most difficult. Flagship models answered around 78 percent correctly, while the smallest open-weight models scored around 47 percent. When asked to judge whether evidence supported or refuted specific claims, the best models scored around 70 percent and the worst scored below 40 percent. Questions about how pairs of climate policies combine were answered no better than random guessing, and questions about how much a policy would change future warming returned even worse results.
User pressure changes answers
In another experiment, researchers argued with 15 models over several rounds, presenting sustained disagreement and counter-arguments. All 15 models reduced their accuracy by 30 to 35 percent. For most cases where the model changed its answer, this happened at the first round of pushback, when the user simply said they did not think the answer was right.
Language differences
The researchers translated three factual benchmarks into 12 other languages and tested five models. Average accuracy dropped in non-English languages, with Urdu down by seven percentage points and Swahili down by ten. About half the questions received different answers from the same model depending on the language used.
Sources
Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.



