Surface temperatures

Initiative progress report posted

2012-02-01T09:26:00.002-05:00

The first Initiative progress report has been posted here. This details the acheivements, progress against agreed workplans, and any issues. They will be produced annually. Although primarily for reporting purposes to 'sponsors', constructive feedback from others is welcome on this blog. We will do our utmost to take any comments on board in future activities.

More on benchmarking - COST HOME paper released

2012-01-10T08:23:00.002-05:00

Continuing on from the theme of the last post the main paper of the COST Action HOME has been published today in Climate of the Past. This predominantly European effort (COST is a European cooperation mechanism) looked to assess a very large suite of approaches to homogenization efforts using a number of test cases which consisted of either real data or synthetic data which had known issues to the data creators for relatively small (compared to the size of the global data holdings) case study regions. Both surface temperature and precipitation were considered. Very many candidate approaches to homogenization were considered and assessed in a consistent manner allowing robust conclusions about the relative merits and strengths and weaknesses of different approaches.

The challenge for the surface temperature initiative is to build upon this by creating global rather than smaller regional benchmarks and to have a similar number of algorithms applied to the much larger global database. Only through looking at the problem in multiple different ways and consistently benchmarking these approaches we will be able to properly (adequately, robustly - take your pick) assess the true uncertainties in our knowledge of global and regional climate change and variability.

For more in-depth information on this study please see the lead author Victor Venema's blog post at http://variable-variability.blogspot.com/2012/01/new-article-benchmarking-homogenization.html and links therefrom.

Benchmarking and Assessment applied to USHCN

2012-01-06T11:56:00.000-05:00

Firstly, an up-front caveat, I am third (last) author on this paper.

Today the Journal of Geophysical Research has published a paper that applies the benchmarking and assessment principles of the surface temperature initiative to the USHCN dataset of land surface air temperatures.

Williams, C. N., Jr., M. J. Menne, and P. Thorne

Benchmarking the performance of pairwise homogenization of surface temperatures in the United States

J. Geophys. Res., doi:10.1029/2011JD016761, in press. ~~(behind a paywall - sorry)~~

Update 1/19/12: NCDC have provided a version commensurate with AGU copyright guidance (and copyright AGU) at ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2/monthly/williams-menne-thorne-2012.pdf. The paper is also highlighted on their "what's new" page at http://www.ncdc.noaa.gov/oa/about/temperature-algorithm.pdf .

The analysis takes the pairwise homogenization algorithm used to create the GHCN and USHCN products and does two things.

Firstly, it identifies a large number of decision points within the algorithm that do not have an absolute basis and allows these to vary. A good climate science analogy here is the perturbed climate model runs of the climateprediction.net project and other similar projects. These decision points were varied by random seeding of values to create a 100 member ensemble of solutions. This at least starts to explore the parametric uncertainties within the algorithm (and any interdependency's) and their implications for our understanding of the observed temperature record evolution. However, what it does not necessarily do is give us any better an idea as to what the true climate evolution may have been. Which brings us on to the second innovation and the focus of this post ...

Secondly, in addition to running on the observations these ensembles were run on a set of eight analogs to the USHCN network. These consisted of sets of data which directly mimicked the observational availability of the USHCN network itself through time. They were based upon climate model runs from a range of models and a range of forcing scenarios. This ensures some 'plausible' spatio-temporal coherency to the large-scale temperature fields. On top of these were super-imposed additional differences to mimic potential random and systematic influences of instrumental and operational artifacts. A set of distinct possibilities were explored across the eight worlds ranging from no systematic biases at all (highly improbable but a useful 'algorithm does no harm' test) through to a scenario where the network was bedevilled with very many largely small breaks with a sign-bias tendency - a situation which any algorithm would find hard to cope with. Unlike the real-world these analogs afford a luxury of knowing the true answer so that it is possible to actually benchmark and understand the fundamental algorithm performance and any limitations. Then it is possible to re-evaluate the real-world results afresh with these new insights gleaned from such realistic test-beds.

The results from the analogs were broadly encouraging. First and foremost when applied to the data with no breaks added virtually no adjustments were made and the impact on large-scale averaged timeseries and trends was so minuscule a magnifying glass would be required to tell the difference. So, in the implausible eventuality that the raw data are bias free the algorithm really would do no harm. For the other analogs the performance was mixed. Easier cases where breaks were bigger and metadata better it did better. Harder cases it fared worse. Where there was no overall sign bias in the applied breaks the ensemble was spread relatively evenly around the starting data. But where there was an overall bias in the raw data presented to the algorithm it consistently moved the overall data in the right direction but rarely far enough. The implication being that this uncertainty is effectively one-tailed. The chances of over-shooting the adjustments is substantially smaller than the chances of under-shooting the required adjustments.

So, what does the real-world look like when reassessed through this new understanding?

For minimum temperatures over the longest timescales the trends are spread around the raw data. But in the periods 1951 onwards and 1979 onwards it is spread distinctly either side of the raw data. Post 1951 trends in the raw data exhibit too little warming, and post-1979 too much. This is consistent with prior understanding of the biases of the change of time of observation (largely 1950s-1970s; spurious cooling) and move to MMTS sensors (early to mid-1980s and often associated with a microclimate relocation; spurious warming) on Tmin.

For maximum temperatures the ensemble consistently precludes the raw observations over all considered timescales. The raw data are almost certainly biased and show too little warming. Over all periods the ensemble of solutions show more warming than the raw data. Again, this is consistent with current understanding of the impact of time of observation biases (again a spurious cooling) and the transition to MMTS which unlike for minimum temperatures imparts a spurious cooling effect. The less than encouraging implication from the analogs, however, is that in the operational USHCN algorithm we are substantially more likely to be under-estimating the required adjustments, and hence rate of warming in maximum temperatures, than over-estimating it.

Is this the last word on the issue? Certainly not. As this was a first step along this path the analogs were perhaps not as sophisticated as would ideally be the case. And here we hope the benchmarking and assessment group can provide more realistic (and global) analogs later this year. Further, this considered solely parametric uncertainty - varying choices within one algorithm. The larger and more difficult uncertainty to understand is the structural uncertainty that would result from applying fundamentally distinct approaches to the same problem and allow a better exploration of the possible solution space. And here we have to look to you, readers, to develop new and novel approaches to homogenizing the data and submit them to the same raw data and analogs to allow consistent benchmarking and better understanding.

Finally, over coming weeks we will be hosting code, data, and metadata (including the analogs) online. I'll provide an update when it is all up there but given that its several Gb and requires fitting around other duties its not going to be instantaneous by any stretch.

Comments are welcome, but please remember that this is a strictly moderated blog and to follow the house rules.

Initiative overview paper published in the Bulletin of the American Meteorological Society

2011-12-15T09:32:00.002-05:00

The first peer-reviewed paper describing the end to end envisaged International Surface Temperature Initiative scope has been published this week by the Bulletin of the American Meteorological Society. It is an Open Access paper available at http://journals.ametsoc.org/doi/pdf/10.1175/2011BAMS3124.1. Any comments, queries or offers of effort are very welcome either through this blog or the general enquiries email general.enquiries@surfacetemperatures.org. Updates since the paper submission can be found at www,surfacetemperatures.org.

GHCN-M v3.1.0 – showing the value of engaging with software engineers

2011-11-10T09:40:00.000-05:00

NCDC have just released version 3.1.0 of the GHCN product, detailedin a tech note, as documented in the dataset paper of their globalLand Surface Air Temperature product – the Global Historical Climatology Network Monthly. This releasedoes two things.

Firstly it incorporates an array processing algorithm thatsignificantly speeds up the processing which will enable NCDC to process themuch larger databank holdings upon its first version release inearly-to-mid 2012 to form a yet more comprehensive estimate of the global LandSurface Air Temperature evolution.

Secondly, and the focus of this post, is that itincorporates a set of five process bug fixes, four of which were discovered inthe homogenization algorithm as a result of an effort undertaken by Daniel Rothenberg sponsored by the Google Summer of Code and mentored bythe Climate Code Foundation. The final bug was discovered as a result ofcarefully checking for similarly based bugs which essentially related to arraycompression / non-compression for missing values on passing between routines.That bugs exist in what is several thousand lines of code is hardly surprising.In fact it would have been far more surprising if it had been discovered thatthere were no bugs. Daniel visited NCDC as part of his project and the bugswere discussed at length with relevant NCDC staff and fixes have subsequentlybeen undertaken, extensively validated, and their impacts on the analysisdocumented.

The bottom line impact on the global mean trend is adifference of less than 0.002K/decade – below the typically quoted global mean estimateprecision of 2 decimal places and two orders of magnitude less than thereported centennial scale global-mean Land Surface Air Temperature warming ratefrom this dataset. Equally global annual means show negligible differences.Differences at the station level are almost always below 0.2K/decade witheffectively zero mean change. So, whilst the bug fixes were important from botha science and process perspective they do not significantly alter our currentunderstanding of changes in climate at the largest space and longesttimescales.

What this does provide is an example of the very realpotential value in openness and transparency, in code replication, and inworking in positive partnership to resolve the issues that arise. Daniel aimsto continue working on his port of the algorithm to python and it will beof great interest to see what other benefits may accrue.

NCDC have released the old (v.3.0.0) and new (v.3.1.0)versions of the homogenized data (in frozen form) and other relevant metadata (with ongoing additions) at ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/archives/.

WCRP OSC thoughts

2011-10-31T13:03:00.002-04:00

WCRP OSC was a very large conference, mainly poster based.The sheer volume of posters was over-whelming. Plenary talks were generally very good, whereas the parallel sessions were a mixed bag with a number of real gems. Being talked at for ten hours a day is too long though and interest inevitably wanes. Wireless connections certainly aren't a help in that regard allowing people to attend without really truly being in attendance. I was presenting four postersacross two sessions (one on my other 'hobby' - the GCOS Reference Upper Air Network) on the Tuesday morning and giving a talk on the surface temperature initiative the Tuesdayafternoon.

I warmed up for this by asking a question in the c.2000attendee observations plenary session first thing on Tuesday - nerve wracking in its own right. Two of theplenary speakers had bemoaned the lack of agreement between estimates for manyvariables and stated to be ‘scared’. I pointed out that this was an inevitableconsequence of making measurements that were not traceable to measurement standardsand that I was instead encouraged to see multiple estimates as this was theonly way we could ascertain what could / could not be said. None of thespeakers responded so either it was an awful point to make or they did not wishto respond.

I spent the majority of the poster time around the three surface temperature initiative posters. Like many of the posters they were in acorner but there was still reasonable interest and a number of potential dataleads were identified. Roughly half of the 50 data request cover letters anddata submission guidelines hardcopies were taken. Most of the discussants weresupportive although inevitably some raised the Berkeley effort and whether thisnow obviated the need for the initiative as a whole. This gave an opportunityto clarify the holistic nature of the enterprise and how the Berkeley effort,if published(!), would simply constitute one important contributing component.It was stressed that science and society are interested in more than the globalcentennial timescale trend and that differences would be greater at smallerspace and timescales. It was also stressed that consistent benchmarking wasnecessary to understand differences more robustly. See also Steve Easterbrook's take on the poster that he was presenting on benchmarking.

The afternoon talk was given in a parallel session withprobably 300-500 people (it felt like the latter!) in attendance. It was alittle bit rabbit in the headlights for the first half although better towardsthe end. There were at least three (maybe four) questions from the floor. Therewere then several people who had questions after the end of the session thatkept me busy for the full half hour coffee break and beyond. These gave achance for a much smaller audience to expand on various aspects – especiallycrowdsourcing. Questions regarding whether data holdings known to a givenindividual were already there highlighted the need to clarify that we wished toget hold of any and all data and that the databank processing will be designedto account for such redundancy in an open and transparent way. Regardless, aconcatenated master-list of current holdings at stage 2 level was requested.This has now been added to the databank prototype.

ISTI at WCRP OSC

2011-10-18T08:51:00.001-04:00

Or acronym soup?

For those who follow this effort and will be attending next week's WCRP OSC conference in Denver, Colorado, we will have a talk in Session B4 (Tuesday @15.00) and four posters in Session C13 (Tuesday morning). The posters are also posted at www.surfacetemperatures.org and the oral presentation will be uploaded there also after the event. Please do pop by and say hello.

ISTI Meeting Reports: 4th ACRE Workshop and GCOS Steering Committee meeting

2011-10-17T09:29:00.000-04:00

I've just got back from presenting the work of the International Surface Temperature Initiative at both the 4th ACRE Workshop (Utrecht, Netherlands) and to the GCOS Steering Committee meeting (ECMWF, UK). The presentations and meeting reports are now hosted at: http://www.surfacetemperatures.org/background.

Kate

You may have noticed we have a logo now ...

2011-07-27T12:31:00.000-04:00

Things are also starting to take shape in revamping the website. Any comments on this and suggestions as to how to make surfacetemperatures.org more useful gratefully appreciated.

Overarching implementation plan published

2011-07-14T13:47:00.001-04:00

We have today published an implementation plan for the initiative as a whole. The plan focuses primarily upon the steps necessary to complete the first databank version and benchmarking and assessment cycle. Comments upon this document are welcome.

Steering committee terms of reference and meeting minutes

2011-04-26T11:27:00.000-04:00

The latest steering committee minutes are available along with a first version of their terms of reference. Forthcoming soon will be an Implementation Plan and terms of reference for sub-groups. This may all seem incredibly boring (and it generally is) but it is also absolutely necessary for the initiative to function properly if it is to be a successful multi-person, multi-institution, multi-year effort. Comments welcome as always.

Prototype for the databank publicly available

2011-04-06T11:11:00.000-04:00

A very initial version of the envisaged global surface databank is available from http://www.gosic.org/GLOBAL_SURFACE_DATABANK/GBD.html. It should be stressed that this is in the very early stages of development. A full version release is not expected until early to mid 2012. This allows time to harvest additional data sources, reprocess, merge, and add provenance information so that it represents a significant delta from what has gone before in terms of both completeness and fundamental scientific value. Comments are welcome here but would be more appropriate at the new databank blog.

Two new initiative related blogs

2011-04-05T16:04:00.000-04:00

There are two new more working level blogs that have been set up recently. These are more technical discussion areas than this blog. Both allow working group members to add posts and comments (moderated) are allowed from anybody else.

http://globalsurfacedatabank.blogspot.com/ covers work towards a global surface databank. If you know of data sources please head on over and provide leads in the post comments.

http://surftempbenchmarking.blogspot.com/ covers work towards a set of benchmark analogs to the databank that algorithm creators can run their algorithms on to ascertain both absolute and relative performance.

Data provenance and versioning task team

2011-03-23T16:32:00.002-04:00

There is a data provenance and versioning task team for the databank effort - details are available from http://www.surfacetemperatures.org/databank/provenance-and-version-control-task-team. This task team has now met twice by telephone and made some initial inroads into the problem.

More generally the main surfacetemperatures.org site continues to be updated with progress as it happens. Other commitments have meant many of these have been left not noted here. My apologies.

Data bank task team on data rescue set up

2011-01-28T14:52:00.003-05:00

Further details can be found here.

For anyone wondering how they can help to improve the data holdings right now, you might want to take a look at either http://www.data-rescue-at-home.org/ for upper air and land surface records, or http://www.oldweather.org/ for world war 1 UK ship records. It is hoped that many of the substantial land data holdings currently available only in image / hard copy form can eventually be digitized by citizen scientists over the internet.

Second teleconference of steering committee

2011-01-28T13:34:00.001-05:00

Notes are posted here. Comments welcome.

First teleconference of steering committee

2010-11-30T15:00:00.002-05:00

Notes are posted here. This provides an update of initial steps undertaken thus far.

Databank working group - first teleconference

2010-11-03T10:21:00.002-04:00

The databank working group recently held their first teleconference. Notes arising from this are available here.

Comments welcome, but there will be patchy moderation, if any, until 15th, please be patient.

Some additional perspectives

2010-11-02T10:09:00.002-04:00

A couple of additional perspectives have recently been published that more formally outline outcomes of the meeting. These still fall short of full expositions of outcomes in formal publications (at least two of which and possibly more are in the works).

Significance magazine online - a new online venture by the American Statistical Association and the Royal Statistical Society.

climatecentral

In addition activities have been starting to spin up in response to the meeting. it is hoped to post information on these starting fairly shortly.

Comments are welcome but should follow the comments policy.

A few perspectives

2010-09-20T09:27:00.002-04:00

Several active bloggers were at the meeting and posts relating to their views are linked here. This list will be updated if and when I am alerted to any additional posts elsewhere but will be limited to views from workshop participants. Please comment on their comment threads as well as here.

Serendipity - Steve Easterbrook (software expert)
http://www.easterbrook.ca/steve/?p=1913

Bryan Lawrence (British atmospheric Data Centre)
http://home.badc.rl.ac.uk/lawrence/blog/2010/09/10/surface_temperature_workshop

Claerclimatecode - Nick Barnes (software expert)
http://clearclimatecode.org/surface-temperatures-workshop/

Protons for breakfast - Michael de Podesta (metrologist)
http://protonsforbreakfast.wordpress.com/2010/09/11/surface-temperature-workshop/
http://protonsforbreakfast.wordpress.com/2010/09/12/surface-temperature-data/

Exeter workshop presentations available

2010-09-13T16:03:00.003-04:00

I am in the process of uploading all presentations to the surfacetemperatures.org site at http://www.surfacetemperatures.org/exeterworkshop2010. I will also post agreed outcomes, hopefully within the week and revamp the frontpage to reflect these outcomes. Like everybody else I also have a day job so please be patient whilst I make these changes.

In the longer term we hope to publish highlights in BAMS and as a longer WMO Tech Doc but this will take some time and require consultation with organisers and rapporteurs and vetting by participants.

Thanks for input - how this will be considered

2010-09-01T12:51:00.003-04:00

Thanks on behalf of the entire organising committee to all those who have taken the time and effort to comment constructively upon the white papers. I am currently in the process of parsing out all comments to the most relevant white papers. In some cases this will involve your post being considered by multiple breakout groups. Within each breakout group we will start the discussion by reading the pertinent comments received before debating amongst those present. Several copies of your remarks will also be available in hard copy for breakout groups to refer to. Breakout chairs have been tasked with ensuring that your comments are adequately considered. There will then be plenary discussions before final decisions are taken. Workshop participants have been urged to read the blog postings that are relevant to their assigned breakout groups in advance of the meeting.

I will be closing comments on all remaining posts at 12Z on 9/2. I will retain comments open on this thread until further notice (comments still to follow the house rules to be considered) but will have limited connectivity and therefore ability to moderate comments until the week after the workshop so any comments received after 9/2 may not be processed until then.

After the meeting next steps will be much clearer and will start to be articulated in a journal article and a technical document although clearly no concrete dates can be given for these appearing. We also hope to emerge from the meeting with some concrete next steps which will be articulated either here or more likely on the surfacetemperatures.org site as soon as details are confirmed.

White paper 14 - solicitation of input

2010-08-10T09:14:00.004-04:00

The solicitation of input from the community at large including non-climate fields and discussion of web presence white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

There are no specific recommendations associated with this white paper. It is more a discussion document of options.

8/25 Please associate comments with the appropriate white paper

White paper 16 - Interactions with other activities

2010-07-30T10:58:00.002-04:00

The interactions with other activities white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

The recommendations are reproduced below:
1) The development of a land surface temperature databank should allow for the extension to multivariate datasets and compatibility with global databanks from the outset.

2) Digitization activities should take a multivariate approach and always incorporate all metadata.

3) Emerging links between land and ocean dataset developers and researchers should be fostered and facilitated by the development of compatible databanks, data products and joint research projects.

4) Appropriate linkage to the activities supported by space agencies aimed at the generation of long-term climate data records addressing the GCOS Essential Climate Variables should be established when developing the surface temperature databank. The experience of GHRSST in the combined management and use of satellite and in situ data should be exploited.

5) Research is needed to improve multivariate analysis methods and to develop techniques to produce consistent global data products.

6) The important role of reanalysis in providing global multivariate analyses with wide application including quality assessment should be recognized. These products will form an essential part of a successful surface temperatures project providing both a set of estimates and a wealth of metadata regarding the data quality.

7) Funding agencies should recognize that an internationally coordinated and sustained approach to the development, maintenance and improvement of climate databanks and derived data products will have wide benefits. This most logically includes a CMIP type portal for climate data records from all observing platforms with common formats and strong naming conventions to enable ease of intercomparison.

White paper 10 - Dataset algorithm performance assessment based upon all efforts

2010-07-30T10:55:00.003-04:00

The dataset algorithm performance assessment based upon all efforts white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

The recommendations are reproduced below:
• Assessment criteria should be developed entirely independently of the dataset developers and should be pre-determined and documented in advance of any tests.

• It is crucial that the purpose to which a dataset could be put be identified and that a corresponding set of assessment criteria are derived that are suitable for that purpose.

• The output of an assessment should be to determine whether a dataset is fit for a particular purpose and to enable users to determine which are most suitable datasets for their needs. Outputs should be clearly documented in such a form as to enable a clear decision tree for users.

• Validation of an algorithm should always be carried out on a different dataset from that used to develop and tune the algorithm.

• A key issue is to determine how well uncertainty estimates in datasets represent a measure of the difference between the derived value and the “true” real world value.

• It would be worthwhile to consider the future needs for the development of climate services by indentifying an appropriate set of regions or stations that any assessment should include.

• New efforts resulting from this initiative should be coordinated with on-going regional and national activities to rescue and homogenize data.

White paper 11 - Spatial and temporal interpolation of environmental data

2010-07-29T10:04:00.004-04:00

The spatial and temporal interpolation of environmental data white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

Update 8/2: A supplement has also been published for comment / consideration. Please be sure top delineate in your comments whether you are discussing the main white paper or the supplement.

The recommendations from the main white paper are reproduced below:
• The choice of interpolation technique for a particular application should be guided by a full characterization of the input observations and the field to be analyzed. No single technique can be universally applied. It is likely that different techniques will work best for different variables, and it is likely that these techniques will differ on different time scales.

• Data transformations should be used where appropriate to enhance interpolation skill. In many cases, the simple transformation of the input data by calculating anomalies from a common base period will produce improved analyses. In many climate studies, it has been found that separate interpolations of anomaly and absolute fields (for both temperature and precipitation) work best.

• With all interpolation techniques, it is imperative to derive uncertainties in the analyzed gridded fields, and it is important to realize that these should additionally take into account components from observation errors, homogeneity adjustments, biases, and variations in spatial sampling.

• Where fields on different scales are required, interpolation techniques should incorporate a hierarchy of analysis fields, where the daily interpolated fields should average or sum to monthly interpolated fields.

• Research to develop and implement improved interpolation techniques, including full spatio-temporal treatments is required to improve analyses. Developers of interpolated datasets should collaborate with statisticians to ensure that the best methods are used.

• The methods and data used to produce interpolated fields should be fully documented and guidance on the suitability of the dataset for particular applications provided.

• Interpolated fields and their associated uncertainties should be validated.

• The development, comparison and assessment of multiple estimates of environmental fields, using different input data and construction techniques, are essential to understanding and improving analyses.

White paper 15 - Governance

2010-07-28T10:47:00.003-04:00

The governance white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

The recommendations are reproduced below:
1. Efforts should be made to capitalise on the efforts of existing and ongoing projects

2. A Steering Committee should be established, to be composed of the chairs of the various working level groups established to conduct the various elements of the work, to include experts from outside the climate science community, and a number of ‘project evangelists’.

3. The steering committee should have designated authority to make reasonable choices to allow for flexibility and efficacy.

4. The steering committee should annually report to stakeholder bodies on the basis of a single annual report.

5. A small number (3-5) of ‘project evangelists’ should be identified and mandated to proactively engage with external stakeholders and facilitate effective two-way communications

6. Findings and conclusions should be reported in the scientific literature and at relevant conferences

White paper 6 - Data provenance, version control, configuration management

2010-07-27T15:44:00.003-04:00

The data provenance, version control and configuration management white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

The recommendations are reproduced below:
• As an outcome of the workshop, there should be a clear definition of primary (Level 0) and secondary (Level 1) source database across the spectrum of observing systems which may contribute data to the land surface temperature database.

• We should establish a coordinated international search and rescue of Level 0, primary-source climate data and metadata both documentary and electronic (see wp3.) This effort would recognize and support similar on-going national projects. Once located, the project should (a) provide, if necessary, a secure storage facility for these documents or hard-copies of same, (b) create, where appropriate, digital images of the documents for the archive for traceability and authenticity requirements, (c) key documentary information into digital files (native format in Level 1 and uniform format in Level 2), (d) archive, test and quality-assure raw data files, technical manuals and conversion algorithms which are necessary to understand how the geophysical variable may be unpacked and generated from electronic instrumentation, and (e) securely archive the files for public access and use.

• A certification panel will be selected to rate the authenticity of source material as to its relation to the “primary-source”, i.e. to certify a level of confidence that the Level 1 data, as archived, represents the original values from the Level 0 primary source. The process will often be dynamic, since we anticipate that new information will always become available to confirm or cast doubt on the current authenticity rating.

• Given the extent of this project and the unpredictable nature of the evolution of the archive, the reliance on an active panel to address version-control issues as they arise will be necessary. The panel will investigate the possibility of utilizing commercial off-the-shelf or open-source version control software for electronic files and software code (e.g. Subversion (http://subversion.apache.org/).

• Since one requirement of this project is to preserve older versions of the archive, and that a considerable amount of tedious research will be performed on any one version, it is generally assumed that up-versioning will be performed of the basic, Level 2 digital archive as sparingly as possible.

• The algorithms that produce the datasets used for testing and the datasets themselves must be documented and version-controlled.

• A configuration management board will be selected to initially define the necessary infrastructure, formats and other aspects of archive practices. A permanent board will then be selected to oversee the operation. This board and the version-control panel may be coincident or at least overlapping in membership.

White paper 4 - Near real-time updates

2010-07-27T15:40:00.003-04:00

The near real-time updates white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

The recommendations are reproduced below:

Recognizing that (i) there is no analog to CLIMAT bulletins on the daily timescale, (ii) daily data are not routinely shared, (iii) no central repository exists because of the lack of a formal process for sharing data with daily resolution, and (iv) the “climatological data” code group in synoptic bulletins does not support the climate communities need for daily summary observations, establish a formal mechanism for dissemination of daily climate messages or requirement for transmission of daily climate observations with synoptic reports.

Recognizing more pronounced issues of the same sort at the basic instantaneous data level, consider as a secondary priority the feasibility (including data exchange policy issues) of adoption of international mechanisms to standardize the exchange of the highest resolution data.

Recognizing that a limited number of bilateral arrangements (e.g. US-Australia, US-Canada) have proven effective at improving access and near real-time data sharing of daily and sub-daily data, establish efforts at the WMO regional level to expand bilateral arrangements for sharing of daily and sub-daily data to increase data holdings and foster regular updates of global and regional daily data sets.

Recognizing that training programs such as those of the JMA have proven effective at improving NMHS capabilities to provide CLIMAT data, expand training opportunities at the regional and national level to improve the routine and regular dissemination of CLIMAT bulletins from developing nations.

Recognizing the success that GCOS Monitoring Centres and CBS Lead Centres for GCOS have had on improving the quality and quantity of CLIMAT reports, continue to support Monitoring of quality and completeness of CLIMAT transmissions and feedback to data providers. Work to garner commitments for enhanced monitoring and feedback related to synoptic bulletins and daily climate summaries.

Recognizing the efficiencies and flexibility of Table-Driven Code Forms for transferring large amounts of data, their design for ease and efficiency of processing, as well as their cost-effectiveness, encourage and support NMCs conversion of data transmissions to TDCFs, while at the same time ensuring adequate attention to issues of long-term data homogeneity in support of climate research.

Recognizing that the GTS is not structured to meet newly evolving requirements for the exchange of data in near real-time, and recognizing the 2003 agreement to move to a WMO Information System (WIS) to meet all of the WMO’s information needs, support adoption of WIS technologies and encourage establishment of GISCs and DCPCs.

White paper 3 - Retrieval of historical data

2010-07-27T12:49:00.004-04:00

The Retrieval of historical data white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

The recommendations are reproduced below:

1. A formal governance is required for the databank construction and management effort that will also extend to cover other white paper areas on the databank. This requires a mix of management and people with direct experience wrestling with the thorny issues of data recovery and reconciliation along with expertise in database management and configuration management.

2. We should look to create a version 1 of the databank from current holdings at NCDC augmented by other easily accessible digital data to enable some research in other aspects of the surface temperature challenge to start early. We should then seek other easier targets for augmentation to build momentum before tackling more tricky cases.

3. Significant efforts are required to source and digitise additional data. This may be most easily achieved through a workshop or series of workshops. More important is to bring the ongoing and planned regional activities under the same international umbrella, in order to guarantee that the planned databank can benefit from these activities. The issue is two-fold: first the releasing of withheld data, and secondly the digitising of data in hard copy that is otherwise freely available.

4. The databank should be a truly international and ongoing effort not owned by any single institution or entity. It should be mirrored in at least two geographically distinct locations for robustness.

5. The databank should consist of four fundamental levels of data: level 0 (digital image of hard copy); level 1 (keyed data in original format); level 2 (keyed data in common format) and level 3 (integrated databank/DataSpace) with traceability between steps. For some data not all levels will be applicable (digital instruments) or possible (digital records for which the hard copy has been lost/destroyed), in which case the databank needs to provide suitable ancillary provenance information to users.

6. Reconciling data from multiple sources is non-trivial requiring substantial expertise. Substantial resource needs to be made available to support this if the databank is to be effective.

7. There is more data to be digitised than there is dedicated resource to digitise. Crowd-sourcing of digitisation should be pursued as a means to maximise data recovery efficiency. This would very likely be most efficiently achieved through a technological rather than academic or institutional host. It should be double keyed and an acceptable sample check procedure undertaken.

8. A parallel effort as an integral part of establishing the databank is required to create an adjunct metadata databank that as comprehensively as feasible describes known changes in instrumentation, observing practices and siting at each site over time. This may include photographic evidence, digital images and archive materials but the essential elements should be in machine-readable form.

9. Development may be needed of formalized by new WMO arrangements, similar to those used in the marine community, to facilitate more efficient exchanges of historical and contemporary land station data and metadata (including possibilities for further standardization).

10. In all aspects these efforts must build upon existing programs and activities to maximise efficiency and capture of current knowledge base. This effort should be an enabling and coordination mechanism and not a replacement for valuable work already underway.

White paper 8 - Creation of quality controlled homogenised datasets from the databank

2010-07-26T14:09:00.003-04:00

The creation of quality controlled homogenised datasets from the databank white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

The recommendations are reproduced below:
• To use daily maximum/minimum temperature as the ‘base’ data set to which adjustments are made, with data at monthly and longer timescales derived from the daily data (adjusted where appropriate) rather than adjusted separately.

• To ensure that all detection and adjustment of inhomogeneities is fully documented, allowing reassessments to be made in the future (e.g. if new techniques are developed or previously unknown data or metadata become available).

• To carry out an objective evaluation of known methods for homogenisation/adjustment, in collaboration with the COST action;

• To establish a testbed of data for this purpose (see white paper 9);

• To seek to ensure that all sources of uncertainty are well quantified and defined.

White paper 13 - Publication, collation of results, presentation of audit trails

2010-07-26T12:54:00.003-04:00

The publication, collation of results, presentation of audit trails white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

The recommendations are reproduced below:
1. When releasing data products, we would recommend that the following information must be provided about process of product generation, apart from data itself for the product to be considered an output of the project:
(1) A listing of the source data (databank version, stations, period) along with methodological rationale
(2) A file describing the quality control method and quality-control metadata flags;
(3) Homogeneous and / or gridded version of the data;
(4) Quality assessment report produced by running against at least a minimum set of the common test cases described in previous white papers;
(5) A published paper based on the data construction method and related products in the peer reviewed press in a journal recognized by ISI;
(6) Publication of an audit trail describing all intermediate processing steps and with a strong preference to inclusion of the source code used..

2. Datasets should be served or at the very least mirrored in a common format through a common portal akin to the CMIP portal to improve their utility.

3. Utility tools should be considered that manipulate these data in ways that end users wish.

White paper 9 - Benchmarking homogenisation algorithm performance against test cases

2010-07-26T12:32:00.004-04:00

The benchmarking homogenisation algorithm performance against test cases white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

The recommendations are reproduced below:
• Global pseudo-data with real world characteristics

• GCM or Reanalyses data should be used as source base with real spatial, temporal and climatological characteristics applied to recreate to a reasonable approximation the observational record statistics

• Review of inhomogeneity across the globe finalised via a session at an international conference (link with White Paper 8) to ensure plausibility of discontinuity test cases

• Suite of ~10 discontinuity test cases that are physically based on real world inhomogeneities and orthogonally designed to maximise the number of objective science questions that can be answered

• Benchmarking to rank homogenisation algorithm skill in terms of performance using climatology, variance and trends calculated from homogenous pseudo-data and inhomogenous data (discontinuity test cases applied) (skill assessment to be synchronised with broader efforts discussed in White Paper 10)

• Independent (of any single group of dataset creators) pseudo-data creation, test case creation and benchmarking

• Peer-reviewed publication of benchmarking methodology and pseudo-data with discontinuity test cases but ‘solutions’ (original homogenous pseudo-data) to be withheld

White paper 5 - Data policy

2010-07-26T12:16:00.004-04:00

The data policy white paper is now available for discussion. When making posts please remember to follow the house rules. Please also take time to read the full pdf before commenting and where possible refer to one or more of section titles, pages and line numbers to make it easy to cross-reference your comment with the document.

The recommendations are reproduced below:
1. Enhance data availability
a. Build a central databank in which both the original temperature observations as well as multiple versions of the value-added datasets, i.e., quality controlled, homogenized and gridded products, are stored and documented together (including version control). The opportunity to repeat any enhanced analysis should exist. Not only will the methods used for adding value change over time and between scientists, but the data policy will change as well.
b. Provide support for digitization of paper archives wherever they may exist with the proviso that any data (and metadata) digitized under this program be made available to the central databank.
c. Enhance the international exchange of climate data by linking this activity to joint projects of global and regional climate system monitoring and by promoting the free and open access of existing databanks in accordance with set principles, e.g., those of the GEO.

2. Enhance derived product availability
a. Accept that there is a trade off between transparency and data quantity used for derived products. Transparency and openness, which scientists (including the authors) advocate, are hampered by the data policies of national governments and their respective NMHSs. Data policy issues are persistent and unlikely to change in the near future.
b. Hold a series of workshops to homogenize data and produce a gridded dataset. The original and adjusted data might not be able to be released but the gridded dataset and information on the stations that contributed to each grid box value would be released. These gridded datasets could be used by NMHSs to monitor their climate and fit together seamlessly into a global gridded dataset.
c. Ensure that the datasets are correctly credited to their creators and that related rights issues on the original data and the value-added products are observed and made clear to potential users. The conditions will be different for bona fide research and commercial use of data.

3. Involve NMHSs from all countries
a. Acknowledge that involvement of data providers (mainly NMHSs) from countries throughout the world is essential for success, and involves more than simply sending the data to an international data centre. For all nations contributing station records to benefit from this exercise, the scientific community needs to also deliver derived climate change information which can be used to support local climate services by the NMHSs. This return of investment is of particular importance for developing countries.
b. Adopt an end-to-end approach in which data providers are engaged in the construction and use of value-added products, not only because it is at the local level where the necessary knowledge resides on the procedures and circumstances under which the observations have been made, but also because this will make it easier to overcome access restrictions to the original data.
c. Increase the pressure on those countries not inclined to follow a more open data policy by engaging with institutions widely beyond the community of research scientists, including funding bodies, the general public, policy makers and international organisations.

Welcome and house rules

2010-07-14T13:50:00.005-04:00

This blog will host the series of white papers for discussion at the workshop to be held at the UK Met Office in September 2010. More details are available from http://www.surfacetemperatures.org. The white papers will be posted on or around July 27th and be open for public comment for approximately 4 weeks. Update 8/19: Comments will remain open until 9/1.

All posts will be moderated and will be processed at 8am EDT each weekday and at no other time. Posts deemed by the blog owner to breach any of these guidelines, even in part, will be deleted. Posts that will be accepted will be scientifically relevant, on topic, concise and constructive in tone. Any of the following will lead to post rejection:

Off topic posts or posts with too much off-topic content
Defamatory language or insinuations against others
Overt linkage to non-relevant resources (very limited and necessary links to strictly relevant scientific papers or scientific web pages allowed but not any links to blogs)
Comments on the reality or otherwise of global warming and political ramifications - there are plenty of other blogs out there to make these comments on.
Comments judged to be too long.

Decisions are final but you are welcome to resubmit a new version that you consider may be accepted having considered the reason why your original submission was rejected.

The purpose of the blog is to get relevant input from everybody as clearly we can only invite so many people to the meeting itself and there are many more scientifically relevant viewpoints out there. We welcome comments from anyone regardless of qualifications who believes they have something positive, constructive and relevant to provide. We will ensure the comments are considered within the September workshop.