A short delay explanation: I did say “further details following“, but following a minor stroke that “following” has had to become a flexible term. Not forgotten, but further down the list of priorities.
Placeholder for now. Discussion, explanations and further details following.
Image settings have been updated so that images can be opened enlarged in a new tab or window.
There will be a short (I hope) pause while I investigate the stations used by GHCN to adjust Valentia. Complicated by the not uncommon situation in climate science that things are not quite what they seem. In this case the downloadable code dates from 2012, which should correspond to v3.2.0 — but in fact it seems to be code specially modified for a dataset for Peter Thorne, now of Maynooth. When reconfigured for GHCN input and run with GHCN-M v3.2.0 raw data the output is close to, but does not quite match, the corresponding GHCN-M adjusted data. Whether this is due to some hard-coded parameter or parameters which were not placed in the documented configuration files, or whether the modified basic code was frozen in time before all the modifications for v3.2.0 were completed, remains for the moment a matter of conjecture.
As the tutorial material I am adding will be excessive for those already familiar with some topics, I will finally shorten the post by moving out some expanded explanations to linked secondary posts.
This post started out to bring together a Twitter conversation:
The remaining corrupted Valentia GHCN-M data, dealt with further below, is relatively minor. But the facts that even after notification and ‘correction’ any such errors remain, and that 17 months later the nature of their errors in MCDW still seems to elude them, as evidenced by the absence of a code correction, can hardly inspire confidence in their claim that the problem lies just with ‘some mean temperature data for select stations in Ireland’.
This post now also provides an opportunity to clarify some common misunderstandings regarding surface temperature series as well as discussing the specific question above.
Owen M seems sound on energy, but has fallen into the trap of straying from the area he is familiar with and accepting what seems at face value an authoritative analysis, but one by someone else who has also strayed from the area they were familiar with, and not understood the source data they were dealing with. This is the reason I try to confine my own blog posts to Surface Temperature data sets, and in particular Gistemp and GHCN. It is only too easy to make errors when accepting other peoples work at face value without checking for yourself.
I dealt with Ewert’s errors in November 2015 when blogs picked up his material. At Not a Lot of People Know That (1), Not a Lot of People Know That (2), Not a Lot of People Know That (3) and NoTricksZone. At NoTricksZone unfortunately a flood of comments from one David Appell, although he was correct on this occasion, seems likely to have caused the substantive issue to become lost.
The station cited by Owen M on his blog is 62103953000 Valentia (Ireland), which is clearly a rural station. NOAA do adjust Valentia data, even though it is a long rural record. But this is not an adjustment for any urban effect (the NASA Gistemp confusion – see below) but rather because the station has been relocated a number of times during this long station history. As one of these relocations was to a site averaging 0.3°C cooler than the prior location, as verified by 24 month parallel observations at the old and new sites before the February 2001 station move, either the data before the station relocation or the data after the station relocation must be adjusted (homogenised) to maintain a coherent long term record.
Unfortunately the NOAA adjustment is an automated one for stations outside the USA. The GHCN adjustment made for Valentia, shown now below, does not correspond to any known station relocations. This does not affect the question above regarding possible NOAA modification of Met Éireann data since it is the unadjusted GHCN data, not the adjusted data, which should be compared to the Met Éireann record. An adjustment based on the known station relocation can be found later in this post.
We can see that the unadjusted GHCN record does generally match the Met Éireann record (apart from the corrupted data mentioned in my tweet):
whereas the adjusted GHCN record below does not match the Met Éireann record.
Note that the adjusted GHCN record in the lower figure is identical to the Gistemp (Raw/Adjusted) record in the upper figure, a point to which we will return shortly.
Before returning to this point, a short note on the Met Éireann data is in order. The complete Met Éireann record above has been digitised from a figure at Climate of Ireland – Air Temperature. The numeric data for Valentia from October 1939 onward can be found at Historical Data from current stations (or, while the old website remains available, at Historical Data from current stations (old website). I have provided both locations here as the complete record does not yet appear to have migrated to the new website).
The NASA Gistemp confusion
From the publicly available data, Ewert made an unbelievable discovery: Between the years 2010 and 2012 the data measured since 1881 were altered so that they showed a significant warming, especially after 1950. […] A comparison of the data from 2010 with the data of 2012 shows that NASA-GISS had altered its own datasets so that especially after WWII a clear warming appears – although it never existed.
But the datasets were not “NASA-GISS’s own. They were the NOAA GHCN-M datasets which NASA-GISS used as their “raw” data. And as NASA-GISS made clear, in 2010 NASA-GISS used raw GHCN-M v2 data as the input raw data for Gistemp, while in 2012 NASA-GISS instead used adjusted GHCN-M v3 data as the input “raw” data for Gistemp. Ewert was comparing apples and oranges. Whether the decision to use adjusted GHCN-M v3 data as Gistemp input was appropriate or not is worthy of examination. But there was no “massive” altering of data.
GHCN data and code availability, and methods
NOAA does not provide a convenient display of GHCN data. Here is what can be found at the NOAA FTP site:
NASA GISS does however provide a more user-friendly interface to both NASA Gistemp and (if used carefully) NOAA GHCN data:
“GHCN V3 adj – homogenized” is the Gistemp adjusted data, not NOAA GHCN data at all, and is displayed here together with the GHCN V3 Unadjusted data (by hovering the mouse over this choice).
NASA Gistemp data and code availability, and methods
NASA GISS describe their analysis:
GISS Homogenization (Urban Adjustment)
One of the improvements — introduced in 1998 — was the implementation of a method to address the problem of urban warming: The urban and peri-urban (i.e., other than rural) stations are adjusted so that their long-term trend matches that of the mean of neighboring rural stations. Urban stations without nearby rural stations are dropped. This preserves local short-term variability without affecting long term trends. Originally, the classification of stations was based on population size near that station; the current analysis uses satellite-observed night lights to determine which stations are located in urban and peri-urban areas.
The differences between GHCN-M and Gistemp consist of this explicit adjustment for urban effects, together with the addition of a small number of southern hemisphere stations not included in the GHCN-M inventory, and a different (and more transparent) method of deriving gridded cell data and global time series.
With the change to use adjusted GHCN-M v3 data as Gistemp input however it is no longer the case that Gistemp is purely matching the trend of the mean of neighbouring rural stations. GHCN-M adjusts the records of both rural and urban stations, and so urban stations influence the adjustment of rural stations. A further problem arises as this also presupposes that GISS can correctly classify stations as urban or rural using satellite-observed night lights.
more on this last problem to be added here
GHCN adjusted record based on known station history
The February 2001 station move took place after parallel observations at old and new locations indicated that the new location ran 0.3°C cooler. To obtain a coherent record for the full period 0.3°C is subtracted from the observations prior to the station move. This is effectively the best estimate of what would have been measured at the new location if these observations could have been made at the new location instead of at the old location where they were actually made. The adjustment is made to the older observations so that the latest observation remains unchanged. (Note that this alignment does not necessarily apply to USHCN stations, where known station histories are used as an additional input to the adjustment process, as well as an additional FILNET step, not discussed here. Outside the US station histories are not used even where, as for Valentia, such station histories could be obtained).
The record adjusted for the February 2001 station move is closer to the raw GHCN (or Met Eireann) data than to the record as adjusted by GHCN. The greater warming shown by the GHCN adjusted record arises as a result of a number of ‘change points’ detected by the automated GHCN PHA adjustment algorithm which do not correspond to any known events in the station history (and failure to detect the known February 2001 station move which did have a measured effect). By contrast, the automated PHA algorithm without knowledge of station history did for example detect a known station move at Dublin Airport.
So what stations are used to adjust Valentia? PHA (Pairwise Homogenization Analysis) starts by selecting the 100 nearest stations, then examines these 100 nearest stations and selects the 40 stations most highly correlated with Valentia. In the maps on the left and below the forty most highly correlated stations are shown as large squares, the rest of the 100 nearest stations as medium squares, and, as a bonus the remainder of the 445 nearest stations by small squares. Although none of the additional 345 stations are used to adjust Valentia, Valentia is among the 40 stations identified for use to adjust some of these, including for example the two stations at Ponta Delgada in the Azores, for which Valentia, almost 2000 km away, is one of the 100 nearest and 40 most highly correlated stations. Stations which would be classified as urban by the GISS night-time luminance criterion are indicated in red, those which would be classified as rural in green. The dashed circles (distorted by map projection) are at 500 km, 1000 km, 1500 km and 2000 km from Valentia.
Anything else I may think of later
to be added, if I have forgotten something else for now
Making sure all details are accurate is unfortunately a slow process. How much simpler life would be if we all could adopt the approach to accuracy of John Gibbons and the merry band of the An Taisce “Climate Change Committee”:
(For non-Irish readers, An Taisce is an Irish NGO, John Gibbons is the spokesman for that committee, with an extensive track record of scientific nonsense and misrepresentation. As you can search for examples of his work on this blog I have chosen another member of that committee to feature this time. Both, although they may not like the comparison, share with President Trump the practice of blocking critical comments on Twitter. There are other members of that committee who should, and presumably do, know better. But I have yet to see any of them surface to put a stop to such nonsense.