20250510_Anomaly_Correlation

When Anomaly Correlations Are on Social Media.

Update 20250626:

This is an update on June 26, 2025. The previous posts are below.

Please note all of this information is from public websites. I do my best to construct a robust story.

At the end of the previous post I wrote:

In mid June a substantial number of satellite observations will no longer be available. It is likely that the forecast system can be recalibrated to accommodate this loss of data. However, this requires personnel, expertise, and computer time. This transition would be high on my mind.

June 16 has come and gone. Observations from the POES Satellites have been turned off. I have seen no posts of degradation or successfully navigating this transition.

In this update, I show that the GFS anomaly correlation skill scores were well maintained during the transition. All of the anomaly correlation graphs are from Operational GFS Verification.

For more background on what is shown in the figures, see the Original Post 20250510, below.

Shown in Update 20250626 Figure 1 is the anomaly correlation at 0 (zero) GMT for a 5-day forecast in the northern hemisphere for the past 31 days. The GFS is in black and ECMWF is in red.  Also shown in the figure are the U.K. MetOffice, the CMC from the Canadian Meteorological Center, the U.S. Climate Forecast System (CFS), the U.S. Navy model (FNMOC, Fleet Numerical Meteorology and Oceanography Center), and IMD from the Indian Meteorological Department.

Update 20250626 Figure 1: from Operational GFS Verification.

All of the systems made it through the June 16 data cutoff with stable behavior. There have been days with GFS outperforming ECMWF.

For the record, from the NASA Global Modeling and Assimilation Office (GMAO) Observation Impact Monitoring, I include a graph of the AMSUA observations count from NOAA-15. This is one of the POES satellites. The GMAO and GFS observation streams are similar.

You see the hard cutoff on June 16, 2025. Note, there are other AMSUA instruments on satellites that remain operational.

Update 20250626 Figure 2: from NASA Global Modeling and Assimilation Office (GMAO) Observation Impact Monitoring.

The next transition challenge will be when SSMIS is turned of at the end of June. I expect this transition to be smooth as well for the global forecast systems.

Update 20250526: (A note added in proof?)

In my original post on 20250510 (available below this update), I presented and discussed a couple of figures that were circulating in social media. Some were using the figures to claim that they represented a degradation of the GFS (Global Forecast System, the U.S. global medium range weather application).  I maintained in my discussion that the degradation in skill score was not likely to be related to fewer weather balloon launches. I noted that forecast skill dropouts in GFS were a recurring feature of the system.

In this update, I will show that the GFS anomaly correlation skill score recovered, and describe more of its past behavior. All of these graphs are from Operational GFS Verification.

For more background on what is shown in the figures, see the Original Post 20250510, below.

Shown in Update 20250526 Figure 1 is the anomaly correlation at 0 (zero) GMT for a 5-day forecast in the northern hemisphere for the past 31 days. The GFS is in black and ECMWF is in red.  Also shown in the figure are the U.K. MetOffice, the CMC from the Canadian Meteorological Center, the U.S. Climate Forecast System (CFS), the U.S. Navy model (FNMOC, Fleet Numerical Meteorology and Oceanography Center), and IMD from the Indian Meteorological Department.

Update 20250526 Figure 1: from Operational GFS Verification.

The image shows both the high scores and the stability of the ECMWF integrated forecast system. It is the state of the art. There are numerous times when GFS scores dip.  For example on May 4 and 5 there are two days in a row of low scores.  From April 30 – May 3, GFS and ECMWF each show a decline. On May 4 ECMWF recovers its high skill, while GFS shows a large decline.

Shown in Update 20250526 Figure 2 is the the anomaly correlation at 0 (zero) GMT for a 5-day forecast in the northern hemisphere for the past 90 days. This documents the lower and more variable score of the GFS relative to ECMWF as, simply, a fact of life. It is interesting to look, starting April 17, 2025, which was part of the Original Post 20250510, that all of the systems show a decline in quality until about April 26. Then they stabilize and decline a bit more. On May 4, ECMWF recovers and on May 6 GFS has also recovered. An extended period of low(er) scores is often indicative of a difficult pattern to predict, which is realized by all of the systems.

Update 20250526 Figure 2: from Operational GFS Verification.

Shown in Update 20250526 Figure 3 is the the anomaly correlation at 0 (zero) GMT for a 5-day forecast in the northern hemisphere from 1996 onwards. The figure establishes the robust ranking of ECMWF as best, with the UK MetOffice next, followed by GFS. The periodic cycle in the figure shows that the winter is easier to forecast than the summer. April and May are transition months that contain difficult to predict patterns.

Update 20250526 Figure 3: from Operational GFS Verification.

Shown in Update 20250526 Figure 4 is the the anomaly correlation at 0 (zero) GMT for a 5-day forecast in the southern hemisphere for the past 90 days. I include this plot for the southern hemisphere to point out the large differences between the two hemispheres and the greater gap between ECMWF and other systems.  The southern hemisphere has far fewer weather balloons, and hence relies more on satellite observations. ECMWF appears to be far better in using satellite observations. GFS is more erratic in the southern hemisphere than in the northern hemisphere.

The collected graphs suggest that GFS is, in general, more reliant on balloon observations than ECMWF.

Update 20250526 Figure 4: from Operational GFS Verification.

These figures suggest that the loss of the balloon observations over the U.S. have not caused a general degradation of the hemispheric skill scores. All of the systems recovered in May.

If I were a manager, facing a loss of personnel and expertise, for this situation, I would be concerned about the ability to analyze the dropouts, having the personnel to address dropouts in real time, and the ability to perform the research to address the underlying deficiencies in GFS, and transfer that research to operations .

In mid June a substantial number of satellite observations will no longer be available. It is likely that the forecast system can be recalibrated to accommodate this loss of data. However, this requires personnel, expertise, and computer time. This transition would be high on my mind.

More generally, I would be discouraged by my inability to maintain quality relative to my competitors, for example, being surpassed by the Canadian forecast system in recent years (see bottom image in Update 20250526 Figure 3).

In total, I would be concerned about the ability to maintain current quality. The loss of key personnel and the increasing erosion of federal capabilities and capacity by a fragmenting private sector amplify the risk of structural decline in skill scores.

On a larger stage, I would be concerned about a decline in service and the ability to address the mission of protecting lives and property.

Here is the –>

Original Post 20250510:

Recently the two plots, below, have been circulating on social media.

Many are pointing to them as a degradation in weather forecasting due to the Trump Administration’s decisions to reduce NOAA funding, and specifically to the loss of weather balloon launches (i.e. radiosondes)

With no desire to defend the  Administration’s decisions related to NOAA, it is premature, perhaps incorrect, to attribute what is seen in these plots as due, directly, to cuts to NOAA.

Why?

Figure 1: Anomaly correlation from social media (various @RyanMaue )

Figure 2: Anomaly correlation from social media (various Météo Côte D’Or )

What are these plots?  They are anomaly correlations, one of the “headline” skill scores used to quickly assess the quality of numerical weather forecasts¹For a discussion of skill scores see Chapter 2 of UFS Strategic Plan 2021 – 2025.  This document also lists all of the applications in the operational suite, as well as a strategy to improve forecast quality (if only).. They are a measure of how well did the predicted pattern of highs and lows match what was observed. The plot on top (Figure 1) is for a 7-day forecast and the plot on the bottom (Figure 2) a 5-day forecast. There is, normally, a significant decline in the score from 5 to 7 days.

If forecast and verification data matched perfectly, the anomaly correlation would be 1.

0.9 is a match 90% of the time.

Somewhere between 0.9 and 1.0 is the state of the art.

The different lines are from different forecast centers.  GFS is the U.S.  system.

ECMWF usually has the highest scores. AIFS is the AI model used at ECMWF.

—

There are two items to note in the figures.

1) is the decline across all of the forecast systems in mid to late April.

2) is that GFS, NOAA’s Global Forecast System (GFS), is performing worse that ECMWF, and its decline seems dramatic.

Taking #1 first, it is normal to see a decline in skill score as we move from winter to spring. It is a time that is more difficult to forecast.  Currently, the transition from El Niño to La Niña likely, also, contributes to reduced forecast skill scores.

It is difficult to attribute this general decline to changes in U.S. data.

As for #2, a common social media claim is that this is a consequence of the reduction of weather balloon launches in the U.S.

There has been the reduction or elimination of balloon launches at 13 stations across the U.S. (as of March 31, 2025, update and analysis May 26, 2025). This is an important loss of some of the most accurate data.  The reduction of balloons is clustered in Wyoming, Colorado, South Dakota, and Nebraska, and leaves a large area that is important to the generation of severe local storms without a key data source.

However, the loss of these ~20 ballon flights per day over the U.S. would not have a large impact on the entire northern hemisphere, which is what is shown in the figures. 20 balloon flights are a small fraction of the global balloon observations, and the balloon observations are a small fraction of the observations used in the forecast.

I do not want to diminish the loss of balloon observations. They are very important to forecasters, especially for severe storms. They are also important for calibrating satellite observations.

—

Based on my experience, here is what I would, first, look at.  All systems have times of underperformance, and the GFS has more of them than ECMWF. I suspect that GFS has a particularly difficult time with the blocking patterns that are present during this time period. That is, we have a weather pattern that the GFS finds difficult to predict. I would look at a multi-year plot and examine previous times of dropouts (i.e., reduction of  forecast skill).

To look forward and, perhaps, more consequential, in mid June, several NOAA satellites will be “retired.” I expect some NASA satellites will, soon, be retired as well. This will be a far larger number of quality observations than the balloon data. This will be an important time to monitor the global consequences.

I expect the cuts to NOAA to have multiple negative impacts on U.S. forecasting and associated services.  We may be seeing early consequences of this, but it will take some time to determine. I think the observed reduction in hemispheric skill scores is unlikely to be due to the reduction of the balloon launches.

Attributing this degradation in skill score to policy decisions based on a couple of plots of one bulk metric does not well serve the cause of science and the work of scientists.

—

For the record, this plot from, (Mass, 2023 see also ) shows the historical difference in anomaly correlation between GFS and ECMWF from 1984 to 2021.  There is a persistent gap, that has been a matter of concern to U.S. researchers for years.

Figure 3: Historic anomaly correlations performance for Mass (2023)

The point here, however, is that what is shown in the figures above (Figures 1 and 2) is, first, consistent with history, in the sense that GFS does not score as well as ECMWF.

A second point, the recent performance of GFS in Figures 1 and 2, with scores well above 0.9 and occasionally better than ECMWF, are, in fact, consistent with the state of the art.

—

A worrisome discussion: The anomaly correlation is a statistical measure that is correctly developed over time spans of a year or longer.  A year, for example, would get you some idea of the seasonal cycle.  An El Niñ0 cycle would get you different states of global ability to predict. To carry on a public analysis of a couple of weeks is not good science.

Dropouts in quality do occur, and they have, historically, been a notorious characteristic of the GFS.  They do sometimes have a relation to the particular weather pattern – other times a disruption to a data stream.

It is, also, important to do an apples-to-apples comparison. GFS and ECMWF have quite different input data streams for observations.  At least historically, ECMWF had a 12-hour observation window – GFS 6. The balloon launches have a strong daily cycle, with 0 (zero) GMT having the most launches.  Therefore, a better comparison would use only the 0 (zero) GMT forecasts. There are numerous other discrepancies that need to be accounted for.

This is an example of the type of discussion that normally takes place inside of a weather center or a research lab – “what’s happening with the forecast?”

A casualty of budget cuts is likely the ability to have those discussions and take corrective actions.

It is not good science to leap on these occurrences as political commentary – even if they turn out to be true.