Friday 21 December 2012

CLIDB-SOS: A collaborative summer internship with NIWA

The National Institute of Water and Atmospheric Research (NIWA) operates a network of weather and climate and other measurement stations, equipped with different sensors, observing a wide range of environmental properties – from temperature over rainfall to wind speed and directions. These high frequency measurements are processed and stored in the NIWA climate database (CLIDB), which is also listed as a national significant database. NIWA has already a web interface in place, where data can be queried and downloaded. Yet this is a manual process.

The Open Geospatial Consortium (OGC) is an international consortium comprising of organisations from industry and research that develops geospatial data transfer and encoding standards and specifications in an open and consensus based process. The OGC Sensor Observation Service (SOS) is a web service interface specification describing access to sensor and time series data – the observations – measured or observed primarily by sensors.


NIWA intends that SOS services become the primary delivery mechanism of its climate, hydrometric and other time-series data to internal research staff, and also central, regional and local government, businesses, NGO's, utilities and the general public via appropriate SOS web clients. Such services provide a database independent, standards compliant approach to data discovery and delivery. Furthermore the NIWA National Climate Database (CliDB) contains a Nationally Significant Database.
With a Sensor Observation Service in place, the CLIDB could actually be queried like a database, but through the web, automagically, from within an application – based on an international standard. Plugins for e.g. R Statistics (sos4r), OpenLayers (OL sos demo) or ArcGIS (ArcHydro and 52°North ArcGIS SOS Extension) are already available to demonstrate data access from SOS servers.

In this Summer of eReseach project I will build a custom connector to source CLIDB directly, based on the sophisticated 52°North SOS server. But the devil is in the details. Although I have already worked with this software I will have to learn more about the database structure of the CLIDB and immerse in the latest development version of the 52°North SOS server, which will support Hibernate, the quite new OGC SOS 2.0 specification and exchangeable output encodings (e.g. CSV, O&MWaterML2.0 and maybe JSON). Additionally we will discuss and demonstrate ways to include SOS data services in a client application.

A lot of work, but worth the effort. I am looking forward to create something new and useful. Besides the apparent advantages for New Zealand’s researchers, also consultancies, commercial organisations and governmental agencies will profit from such a dynamic, standards-driven and web-based geospatial data delivery service that could serve as another good example for environmental data providers.

The benefits of the project include well managed freshwater use & potential impacts of climate change are critical for any country to plan & manage its natural resources. One of the foundations of any effort in these domains is ready access to climate (including rainfall) information.
NIWA is collaborating with a wide range of New Zealand and international agencies to determine strategies and appropriate standards to provide interoperable discovery and delivery facilities for data managed by NIWA, as well as to develop systems to implement those strategies. These include: CSIRO, BOM (Bureau of Meteorology), AODC (Australian Ocean Data Centre), IMOS(Integrated Marine Observing System) in Australia. Iquest and Kisters (New Zealand and internationally), 52o North.
Furthermore collaboration with GNS Science in the NZ-EU cooperative groundwater project SMART (www.smart-project.infohttp://www.gns.cri.nz/) is intended, to support those who seek to incorporate climate data for their research towards characterizing New Zealand’s aquifers.

Monday 10 December 2012

A Sensor Observation Service for the NIWA climate database

The National Institute of Water and Atmospheric Research (NIWA) operates a network of weather and climate and other measurement stations, equipped with different sensors, observing a wide range of environmental properties – from temperature over rainfall to wind speed and directions. These high frequency measurements are processed and stored in the NIWA climate database (CLIDB), which is also listed as a national significant database. NIWA has already a web interface in place, where data can be queried and downloaded. Yet this is a manual process.

The Open Geospatial Consortium (OGC) is an international consortium comprising of organisations from industry and research that develops geospatial data transfer and encoding standards and specifications in an open and consensus based process. The OGC Sensor Observation Service (SOS) is a web service interface specification describing access to sensor and time series data – the observations – measured or observed primarily by sensors.

With a Sensor Observation Service in place, the CLIDB could actually be queried like a database, but through the web, automagically, from within an application – based on an international standard. Plugins for e.g. R Statistics (sos4r), OpenLayers (OL sos demo) or ArcGIS (ArcHydro and 52°North ArcGIS SOS Extension) are already available to demonstrate data access from SOS servers.

In this Summer of eReseach project I will build a custom connector to source CLIDB directly, based on the sophisticated 52°North SOS server. But the devil is in the details. Although I have already worked with this software I will have to learn more about the database structure of the CLIDB and immerse in the latest development version of the 52°North SOS server, which will support Hibernate, the quite new OGC SOS 2.0 specification and exchangeable output encodings (e.g. CSV, O&M, WaterML2.0 and maybe JSON). Additionally we will discuss and demonstrate ways to include SOS data services in a client application.

A lot of work, but worth the effort. I am looking forward to create something new and useful. Besides the apparent advantages for New Zealand’s researchers, also consultancies, commercial organisations and governmental agencies will profit from such a dynamic, standards-driven and web-based geospatial data delivery service that could serve as another good example for environmental data providers.

Link to the eResearch blog post