GHCN Metadata (v2.inv), Gistemp and Night light Radiance

The draft journal article “Current GISS Global Surface Temperature Analysis” posted at http://data.giss.nasa.gov/gistemp/paper/gistemp2010_draft0319.pdf provides a link to the nightlight radiance data set [Imhoff et al., 1997] used [http://www.ngdc.noaa.gov/dmsp/download_rad_cal_96-97.html] (measurements made between March 1996 and February 1997)] at a resolution of 30″ x 30″.

While I was unable to simultaneously view the terrain detail and the NASA nightlight layer which comes with Google Earth, the nightlight radiance data from the TIFF files in this data set can be added to the Google Earth tours described in my earlier post.

The image below shows my “home station” with the radiance values added in green. I have also added a yellow rectangle centred on the station and extending 0.005 degrees N/S and E/W, showing the locations which when rounded to 0.01 degrees will give this pushpin location. The radiance index in v2.inv (33) matches the radiance value closest to the station pushpin (as did the radiance values for the other Irish stations).

Metadata (lat/lon and radiance) for Dublin Airport

Metadata (lat/lon and radiance) for Dublin Airport

For comparison, here is the same station, but using radiance values from the lower resolution TIFF image downloaded.

Radiance for Dublin Airport from lower resolution TIFF

Radiance for Dublin Airport from lower resolution TIFF

[ Code to follow. I suspect the radiance values may be slightly shifted north-west, as the file description for the radiance seems to suggest such a shift (although it is also possible that I have introduced this slight shift myself!). For now this is a “draft” post which will be corrected or expanded later. ]

[ Expansion March 27th 2010 ]

The question of a possible shift arises as the dimensions of the downloaded 96-97 TIFF file are 43200 pixels by 21600 pixels, one short of the number required to give complete 360 degree by 180 degree coverage at 30″ by 30″ (for comparison the version 4 DMSP-OLS TIFF files are 43201 pixels wide by 16801 pixels deep [latitude coverage from -65 to 70 degrees only]). If it is assumed that the actual grid size is slightly increased from 30″ by 30″ to give full coverage, the shift involved can be seen by comparing the yellow and green figures in the two illustrations below: for Dublin Airport

Radiance location comparison: Dublin

Radiance "shift": Dublin Airport

and Saipan (a greater shift for a southern hemisphere location, further towards 180 East)

Radiance location comparison: Saipan

Raidiance "shift": Saipan

(I will try to get this point clarified by NGDC)

At http://rhinohide.wordpress.com/2010/03/24/dmsp-the-stars-at-night-they-are-so-bright/ Ron Broberg says:

I am disappointed in the degree of accuracy/precision between GISS and myself in the parsing of DMSP/RC data. While there is a visible rough correlation, I expected better.

This seems largely a question of the rounding/truncation habits of the Gistemp group (when porting Gistemp with added output information I found that the only way to achieve a perfect match to the original Gistemp output was by means of extensive special-case rounding tables, to take care of the many instances of rounding of values close to 0.5 in unexpected directions). In this case, rounding the image row calculated from latitude to the nearest row gave 1230 differences from the GISS values, whereas truncation to the next row to the north (TIFF images are stored top down) reduced this to 121 differences.

Without the GISS code to examine it is hardly worth spending more time trying to reconcile the remaining differences, particularly as it is also entirely possible that the GISS draft paper may be incorrect in citing the 96-97 data, which is the data set currently archived. Reading the cited Imhoff et al. [1997] might suggest that earlier data may have been used, which is no longer available for download.

The GISS draft paper also states:

For these reasons, beginning in January 2010 the standard GISS analysis employs global
nightlights in choosing stations to be adjusted for urban effects.  Use of nightlights is a well-
defined objective approach for urban adjustment, and the nightlight data set used for the
adjustment is readily available (see http address above).

and

The standard nightlight adjustment defines rural as nightlight radiance less than 32
µW/m2/sr/µm

From which you might conclude that this standard nightlight adjustment has been employed in the standard GISS analysis since January 2010, but examination of the archived log files shows that in fact the rural definition of less than 11 which was used in the October 2009 test is still being used. Has anyone at GISS noticed? In this regard some might also consider the shifting balance between rural and urban stations a rather large elephant lurking in the room which deserves mention in the draft paper: from 2507 rural/3795 urban prior to the global application of the nightlight criterion, via 3123 rural/3179 urban for the criterion actually now applied, to 5334 rural/968 urban for the standard adjustment described in the draft paper. A minor change not worth mentioning? Possibly not of course if you also believe you have stations located with a resolution of 0.01 degrees of latitude and longitude because that is the number of digits appearing in the file.

Correction (April 14, 2010): in fact the Gistemp group have in fact followed their draft article. I missed a conversion equation relating DN values (used in the night lights TIFF file and in v2.inv) to the radiance values referred to in the draft article. The number of additional rural stations, just over 600, is still substantial, although not as extreme a change as I earlier thought.

[ code to follow when rewritten in R. At present this is still a C# console application ]

Advertisements
This entry was posted in Uncategorized and tagged , , , , , . Bookmark the permalink.

17 Responses to GHCN Metadata (v2.inv), Gistemp and Night light Radiance

  1. That’s a nice solution to the problem.

    I was playing around with Gearth last night and thinking about how you would do that.

    Let me send MarkC over here.

    The other cool thing would be to add “urban extent” and population density to the map.

  2. Yea.. I made R work on my Mac

  3. good work,

    have you thought about contacting the Guys at clearclimatecode?

    They appear to have a good working relationship with GISS. I can send Nick Barnes over here if you like
    Nick works with the Python version of GISS.

  4. Nick Barnes says:

    I don’t have time today to read the paper or the data description. Maybe later this week. But in the meantime, can you explain why a 30″ grid should be 43201×21601, not 43200×21600 ?

    • oneillp says:

      As I said, I will try to get this point clarified by NGDC. I’m comparing the pixel sizes and respective “readme.txt” files for the DMSP-RC TIFF used and for the Version 4 DMSP-OLS Nighttime Lights Time Series.

      world_avg.txt (DMSP-RC):

      The full resolution file contain byte images in a geographic
      projection with the following characteristics:

      Image Size: 43200 columns X 21600 rows
      Grid Spacing: 30 Arc Seconds
      Center of Upper Left Pixel: 90 Lat, -180 Lon

      and README_V4.txt (DMSP-OLS, 43201 x 16801 pixels):

      The products are 30 arc second grids, spanning -180 to 180 degrees
      longitude and -65 to 75 degrees latitude.

      If the centre of the Upper Left pixel is at 90 lat, -180 lon or 75 lat, -180 lon for the OLS data, it takes an additional 360 * 120 or 43200 pixels, 43201 in all, to reach a pixel with centre at +180 lon. It might seem possible to suggest that only 43200 in total are needed because +180 has wrapped around to -180, but the OLS V4 size and description indicate that the odd number of pixels is used to explicitly include +180 lon, rather than stopping 30″ short.

      With latitude it is less ambiguous: the OLS V4 data runs from 75 N to 65 S, and an odd number of pixels, 16801, is needed to include both limits at 30″ intervals. The 21600 rows for the RC data would leave the last row centred 30″ short of 90 S.

      world_avg.txt does not explicitly say that the centre of thw bottom right pixel is -90 Lat, +180 Lon, but it seems likely by analogy to the OLS-V4 description that this is so. If so, a cell spacing slightly greater than 30″ is needed for both latitude and longitude, and this gives rise to the progressive shift I have suggested.

      I’ll post any clarification received here.

      • Nick Barnes says:

        It seems vanishingly unlikely to me that the pixels are any size other than 30″ x 30″. So I suspect that end-pixels are repeated.

        Wherever pixel boundaries lie, if you have 43201 30″x30″ pixels in a W/E row, the east-most pixel and the west-most pixel of each row will coincide exactly. For instance if the west-most pixel centres are at 180W, then the east-most pixel centres will be at 180E, which is exactly the same point. Do the end-pixels of each row match?

        Regarding the rows, presumably the top row only describes the northernmost 15″ of the world, and the bottom row omits the 15″ around the south pole.

  5. Nick Barnes says:

    Also see world_avg.tfw. The cell size is given as 0.00832999963313 degrees, which is not 1/120 but is a plausible representation of 0.00833 (see also my post on floating-point).

  6. One thing that would be helpful is the following.

    1. Color coding the nightlights value (txt color) for the pixel closest to the site location ( or use a different font size)

    2. Averaging all nightlights values within 5 minutes of the location, since sometimes the location is off by 5 minutes ( ancedotal information)

    3. Averaging all nightlights within say 10km of the closest pixel to the location. and displaying that. UHI is a meso scale phenomena

  7. Zeke Hausfather says:

    Wish I had come across this blog earlier; nice work! I’ve been sticking mostly to analyzing USHCN via various urbanity proxies due to the issue of lat/lon coords being somewhat problematic in GHCN. I wonder if we could take a random sampling approach (and a lot of man hours…) to figure out both what percent of stations are mislocated and what the mean mislocation is.

    I also wonder if you could use the subjective urban classification in the GHCN metadata as a good filter to avoid misclassified; e.g. if something is classified as urban in the metadata but has dark nightlights, its probably being misclassified based on incorrect lat/lon.

    • Zeke, The thing I was hoping for is this.

      1. Getting the best lat/lon meta data for USHCN. Surface stations has a list posted but you have to copy by hand or
      or figure a way to read that data off the html.

      2. generating a pile of photo’s from Google earth.

      3. organzing a community effort to look through the pile.

      We could do the same thing with the rest of GHCN. what 7000 stations or so? WRT urban darklights, that’s a good place to start checking
      also rural bright. Another way is to just look at the picture a a 20km radius around the stated location? What are the pixels values closest.
      The photo above allows you to do that.

  8. oneillp says:

    To Steven and Zeke: looking at your suggestions

    To Nick: the tfw files are in fact “almost” also contained within the TIFFs as tagged data, but that “almost” has suggested a possible lead to follow. More later

  9. peter,

    When you get a chance you might visit here:

    http://noconsensus.wordpress.com/2010/03/28/new-surface-temperature-page/

    Jeff works in R.

  10. oneillp says:

    I have added a correction today near the end of this post.

  11. Hey hows it going, i been learning R. ha.

  12. thanks I just read your update. I need to read hansens paper a bit more carefully. At somepoint I want to put nightlights into the metadata system Im building.. thx

    • oneillp says:

      I’ll post on nightlights reconciliation with Gistemp when I can – in summary, I have been able to reconcile all but 95 stations with the values used in Gistemp, although the calculation for latitude to get these to round to the Gistemp values was unexpected, and different from that for longitude! I presume the remaining 95 stations differ because of platform dependent rounding considerations.

      I have a post on this in draft, but unfortunately the supporting material to complete it is on a new laptop which had what appears to be a hardware failure in the power supply, and has had to be returned for this to be rectified. I hope I can complete that post later in the month. It might have been completed before that hardware failure if we had not had unexpected unseasonal snow near Dublin at the beginning of the month which I availed of for skiing!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s