Many companies offer detailed climate projections tailored to property-level precision. These tools can allow owners and investors to better anticipate risks of flooding, wind, wildfire and other disasters. Insurers have been among the adopters of climate-risk models, in a sign that these forecasts of vulnerability will inform decisions about whose premiums go up as the planet warms.
But these risk assessments are happening inside what amounts to a bunch of black boxes. There’s almost no way to compare or review the growing number of risk projections, which are typically guarded from inspection as the intellectual property of private firms. Now there’s new evidence that risk models often disagree with each other on fundamental assessments of vulnerability.
In a review using properties in Los Angeles County as a benchmark, there was widespread disagreement between one private risk model that evaluates future flood risk and an open-source alternative created by academics to measure the same thing. The two risk models assigned the similar vulnerability scores to only about 1 in 5 properties in the Los Angeles area.
Other comparisons made by researchers have run up against this same lack of agreement between models. Analysts at the climate nonprofit CarbonPlan found substantial differences in the data and projections across two private risk models, according to a report published Friday on the group’s website. A city lot that one company labels high-risk may not be risky according to another, raising questions about the reliability and transparency of tools that are now widely used in many sectors of the economy.
In December, CarbonPlan researchers asked nine climate-analytics companies to participate in a “limited data review” by sharing information on 342 U.S. locations. The researchers specifically requested small data samples out of millions that are potentially available to avoid burdening the providers. Of the nine they approached, only two fulfilled the request: Jupiter Intelligence and XDI.
Working with data from two firms was enough to allow comparisons that are otherwise unavailable to anyone who hasn’t purchased access to both tools. CarbonPlan found major variations in the risk estimates, both looking back to results from the mid-1990s and in forecasts out to the end of this century. In one telling example, the sample results from Jupiter and XDI agreed on only 12% of locations where fire risk will increase in the decades ahead after comparing sample data for 128 California post offices.
The researchers also asked for coastal flood data for 90 New York City schools, prompted by the clear risk of sea-level rise in the region. Both models concurred that vulnerable properties are concentrated near the coast. But the share of locations where both Jupiter and XDI agree on rising flood risk is only 21%.
A third comparison used river and rainfall flood data for 124 New York State post offices. Both companies agree on the broad overlap among locations facing low risk of river flooding out to the end of the century. But among the locations assessed with higher risk levels, “there is only minimal consistency” across the two models, according to the CarbonPlan report.
Some of the variation comes down to the fact that Jupiter and XDI have different approaches and proprietary models. CarbonPlan analyzed only each firm’s risk scores, not their underlying methodologies. The purpose of the study was not to evaluate whose model is better, only to see if or where the tools differ.
“(W)hat is perhaps most striking about this endeavor is how little data we received” from most of the climate-analytics companies, CarbonPlan noted in their report. First Street Technologies, Verisk Analytics, ClimateCheck, ZestyAI, Riskthinking.AI, Carbon4 Finance and Climate X either declined to participate or did not respond to the researchers.
In an interview before the release of CarbonPlan’s report, XDI co-founder Karl Mallon said he expected the comparison between different risk models would prove useful. He said an advantage companies have over academic researchers is the ability to build a team drawing on diverse areas of expertise, such as climate science, building codes, insurance, engineering and hydrology.
“When we have very big differences, it sort of indicates we’re doing different things,” Mallon said. Even if modelers agree on key fire-weather drivers, like temperature, humidity and wind, they may diverge over whether they’re also including grass fires.
Josh Hacker, co-founder and chief scientist of Jupiter, said it was hard to comment on CarbonPlan’s findings without seeing the full analysis. He disagreed in general with the notion that private climate models are black boxes. “When you’re in business, you have to protect some things,” Hacker said, noting that Jupiter’s team publishes peer-reviewed papers and presents at conferences when possible.
Hacker also noted that Jupiter works with third-party experts to help validate its work and is careful to caveat the results for customers, which Jupiter says include Norwegian oil company Equinor as well as insurance giants Aon and Zurich Insurance Group. “We communicate the uncertainty,” he said. “We communicate where we have less confidence. We communicate where we have more confidence. But it’s against our business interest to communicate everything broadly.”
Many of XDI’s clients, which the company says include the world’s largest money manager BlackRock and banking giant HSBC Holdings, want to stay ahead of insurers’ decisions, and that encourages XDI to think like an insurer. “In a way, part of our challenge is trying to think how an actuary will think,” Mallon said. “And they’re pretty tight-lipped about how they do stuff.”
CarbonPlan’s mission is to promote transparency, and its new report on risk models echoes that imperative. Its findings emphasized that risk models may agree over a region while diverging substantially at the property level. In other words, different risk models might determine that 5% of houses in a state at high risk of a disaster — just not the same 5%.
Another takeaway concerns how people, companies and governments respond to this kind of information. With climate adaptation a growing priority, “risk estimates from analytics companies are likely to affect billions of lives and trillions of dollars,” CarbonPlan warns in its findings. “If the climate-risk industry continues in the private sector, without any requirements for transparency, black-box models that cannot be compared will become the norm across many industries, including housing and insurance.”
Climate science for decades has progressed by comparing models on a global scale. In a sense, CarbonPlan is now calling for similar practices to compare private risk analyses on a very local level.
“There’s a lot of room for there to be variation — even room for there to be variation and not have anyone be doing it wrong,” said Oriana Chegwidden, a research scientist at CarbonPlan. But the tighter the geographic focus, the more uncertainty in the prediction, she noted. “And we just don’t know how much there is.”