Showing posts with label CLIDB. Show all posts
Showing posts with label CLIDB. Show all posts

Sunday, 25 October 2015

A Spatial Data Infrastructure Approach for the Characterization of New Zealand's Groundwater Systems


I was very happy when I got informed that our latest research article was published in "Transactions in GIS". While being embedded in our New Zealand SMART Aquifer Characterisation programme, it was a great joint effort which results went beyond SMART.

Kmoch, A., Klug, H., Ritchie, A. B. H., Schmidt, J. and White, P. A. (2015), A Spatial Data Infrastructure Approach for the Characterization of New Zealand's Groundwater Systems. Transactions in GIS. doi: 10.1111/tgis.12171

It explains technical and methodological approaches of web groundwater data infrastructure and uses NIWA's CLIDB and GNS's NGMP as examples and describes the long way from stakeholder interaction, workshops and meetings towards an NZ data standard for the National Environmental Monitoring Standards (NEMS) framework - the Environmental Observation Data Profile (EODP) as data access and transfer blueprint for NEMS.

Wiley Online Library "Transactions in GIS" Journal

The technical work with colleagues, collaborators and partners from New Zealand research institutes (CRIs) like GNS, NIWA, Landcare Research, New Zealand regional councils like Horizons and Waikato (WRC), Hawke's Bay (HBRC) and and Bay of Plenty (BOPRC) regional councils which culminated in a draft OGC profile and is being incorporated into national standards in New Zealand, GitHub EODP and are based on the work over the last two years including a great review paper on OGC standards for groundwater,  customising the 52°North SOS server to interchangeably encode WaterML2 time-series data along O&M2 through a Google Summer of Code (GSoC) programme, and then customising the SOS server database access to demonstrate it as an adaptor on legacy databases of national significance, in specific an "NGMP/GGW-SOS" and a "CLIDB-SOS" demo through a NZ eResearch programme.


Tuesday, 10 February 2015

EODP - An OGC-based Environmental Observation Data Profile

A group around Landcare, NIWA, GNS, HillTop Software and Horizons Regional council worked the last 2 years to develop an interoperable web-based data access and sharing standard. Working group formed as results from Hydrological Society Annual Conference Data Access Workshops. Subsequent meetings and workshops at Horizons regional Council office in Palmerston North even featured visits and input from Dr David Maidment as well as from delegates from BOM and Kisters, Australia. Further workshops were held at GNS in Wairakei and NIWA offices in Auckland and Wellington.

The NEMS-endorsed Environmental Observation Data Profile (EODP) is based on proven open standards of the Open Geospatial Consortium (OGC) and comprises a profile of WFS that allows the discovery of timeseries data associated with monitoring stations. This forms part of a larger project to enhance the interoperability of Environmental Observations throughout New Zealand.

Organisations such as NIWA, Landcare, GNS and Regional councils maintain a range of monitoring stations making environmental observations. These include Climate, Hydrology, Air Quality, Soils, Water Quality, and Marine data.

We wish to make this data available for research, analysis and reporting through open data protocols such as the Sensor Observation Service (SOS), Web Feature Service (WFS), Web Coverage Service (WCS) and Web Mapping Service (WMS). We also want to publish the existance of such data sets and services through online metadata catalogues (CSW).

In order to make use such data services users must first be able to discover their existence and then accurately select the subset of data that is relevant to their needs. The EOPD is focussed on this selection process in essence addressing the Where, What and When questions.

WFS provides a rich filtering capability, but typical WFS sources use a flat property/value schema that makes it difficult to express the many to one relationship of the timeseries measurements taken at a station. The EODP is an Application Schema for WFS that encodes the station metadata as a nesting of SF_SpatialSamplingFeature and OM_Observations using the language of Observations and Measurements to describe the characteristics of the timeseries.

The profile is extended by the use of an external Vocabulary service used to dereference measurement identifiers into familiar timeseries names such as rainfall or temperature.

Alistair Ritchie from Landcare Research did the main formal write-up and published the documentation on GitHub: https://github.com/EODP-NZ/eodp-dev

A implementation testbed is currently underway in joint collaboration with NIWA on the NGMP database at GNS and the Climate Database (CLIDB) at NIWA.

http://portal.smart-project.info/sos-smart/service?service=SOS&version=2.0.0&request=GetDataAvailability&observedProperty=http://vocab.smart-project.info/ngmp/phenomenon/1679

http://portal.smart-project.info/gs-smart/wfs?request=GetFeature&service=WFS&typename=sams:SF_SpatialSamplingFeature

Tuesday, 16 April 2013

Geospatial web-enablement for environmental data in New Zealand

This blog post can be seen as a sequel to a former blog post on the introduction on geospatial data sharing and spatial data infrastructures (SDI), where I explained the basics of OGC standards and web services. Quite some research organisations and governmental agencies already employ OGC standards to make data available online, often even free of charge for the public. I would like to present some really good examples of interoperable data sharing in New Zealand.
Through the standardised and web-based access to so many data sources, not only traditional geographical processing and analysis (GIS) based research is made easier, but also complete new technical and methodological research possibilities arise.

LINZ - Land Information New Zealand

I would like to start with Land Information New Zealand (LINZ). LINZ, as a governmental body, has issued and maintains New Zealand’s geospatial strategy. LINZ runs the LINZ Data Service, which provide tons of NZ-related data sets, topography, maps, place names and much more, almost all of it is available under a NZ Creative Commons license. You can register for free, get an API key and use data directly through web, basically as long as you tell that it is LINZ data. LINZ provides standard OGC CSW, WMS and WFS web service interfaces.
More news about the NZ geospatial strategy can be found on here.

DOC – Department of Conservation

Also the New Zealand Department of Conservation is going towards geospatial web services. It looks like they use ESRI software, which supports OGC standards to certain bit already, although ESRI (producer of the ArcGIS software) as a commercial closed-source software provider has been known to notoriously neglect open standards. However, the Shapefile format is open and besides ESRI REST services, the DOC Geoportal also allows for OGC-based access (CSW/ISO 19139 metadata for search and discovery and WMS/WFS for map/feature data access)

GNS Science

The Institute of Geological and Nuclear Sciences is one of the 9 New Zealand Crown Research Institutes (CRI), which conduct about one half publicly/governmentally funded and the other half commercial research projects and, together with the universities of course, can be seen as New Zealand’s main science and research providers, each claiming a particular scientific domains. GNS Science is New Zealand’s leading provider of Earth, geoscience and isotope research and the geological survey of New Zealand. GNS’s research topics also include volcanoes, earthquakes, geothermal features and groundwater.
GNS has published the 1:250 000 Geological Map of New Zealand (QMAP. It is also digitally accessible – GNS exports the QMAP as OGC WMS and WFS in the OGC format GeoSciML. An easy way and very interesting example for interoperability is to explore New Zealand’s geology is through the OneGeology project, which sources and displays such services from geological surveys from all over the world.
The GNS-EU collaborative SMART Acquifer Characterisation programme (SAC) also aims to connect OGC based data sources. Within the research aim “Data Synthesis and Visualisation” the SMART Data Portal aims to develop an integrated OGC framework for discovery, access, processing and visualisation of hydrogeological data.

NIWA – National Institute of Water and Atmospheric Research

NIWA NIWA, another CRI, has a strong reputation in climate, marine and marine ecosystem and biodiversity sciences. Whereas a lot of organisations and agencies make data available first and then (if at all) add more sophisticated search technology, NIWA started the other way round. They established a discovery portal - the Environmental Information Browser, which is basically a catalogue, where one can search by keywords, places, data and time. All the data NIWA has, will eventually be listed and can be queried and also harvested through the OGC CSW interface. Furthermore NIWA is also moving towards providing OGC web services to their data sets. One particular example has been a “Summer of eResearch” project and its progress documented on the eResearch website.

Landcare Research

Landcare Research is also a New Zealand Crown Research Institute and focuses on the management of terrestrial biodiversity and land resources in order to both protect and enhance the terrestrial environment. I have come across several Landcare projects on soils and land use data, where Landcare not only uses OGC standards, but also participates in the development and maturing of some of those standards. Like the former parties, Landcare runs a data or geoportal (LRIS), too, which can be accessed and queried through CSW, WMS and WFS web interfaces,
Landcare also hosts a dedicated soil map portal (S-map), which sources the digital soil information layers based on WMS. Furthermore they drive the development of a global soil map portal (http://www.globalsoilmap.net/), which under the hood, of course, uses OGC standards again. To enable international, comparable, interoperable soil data exchange Landcare participates in the development of a soil information standard.

Outlook

Regional councils are on the way, too. Many regional councils already make data accessible on their web sites. A quick investigation shows for example Environment Waikato, Horizons, HBRC, BOP or Environment Canterbury. However most of these data sources need to be accessed manually and/or do not provide a standardised interface. Not to speak of a generalised way to actually find them. In conjunction with the open data initiatives (Open and Transparent Government , Open New Zealand) and catalogues available ( government datasets onlineOpen Data Catalogue), there is massive potential to link diverse datasets, relate and analyse seemingly unrelated datasets and gain new insights, find and (re-use) data by type, time and location or just enable ubiquitous mobile access to the data you need. However, we might end up needing a catalogue for the catalogues, and of course a lot of existing data needs to be geo-located/geo-referenced, so that they could be found by location. There is still a way to go and definitely some more research necessary in that space.

Wednesday, 13 February 2013

SOS for CLIDB Mid-term - we got data

Based on the "vanilla" code from the 52North SVN repository I basically added a CLIDBHelper class to the "52n-sos-hibernate-core" Maven module that encapsulates the CLIDB database connection and re-directed most of the original Hibernate calls from the core DAOs to the helper.

Especially the Cache Feeder DAO needed a lot of attention, as it would fill the internal "capabilities" cache, which is the basic thing that needs to happen before the SOS instance is actually usable.

Then Describe Sensor DAO and GetObservation DAO are pretty straightforward, as they would "just" do specific queries, like give me the sensor description for sensor station XY or give rainfall data for station XY in the time period ...

Link to the full eResearch blog post


Final NZ Summer of eResearch 2012 presentation slides


The final student project presentations were held in Wellington at Victoria University.

Link to the slides of the final student project presentation




Monday, 4 February 2013

Random Reflections

Kia Ora :-)

The last weeks, two months were packed with experiences, developments, progress. I would like to shortly summarise the main three happenings. It is great to see progress and experience interest in that progress, so I'd like to share that, too :-)

New Zealand Hydrological Society Conference 2012, Nelson and the Data Access Workshop





New Zealand Summer of eResearch (Dec 2012 - Feb 2013)


SCANZ 2013 3rd nature hui and wananga symposium



This is sooo much stuff, I will have to write out details later :-) So far enjoy the links.

Monday, 14 January 2013

A Climate Database Web Service - 1st Review

The first week before the break I spent some efforts to integrate my software development project into NIWA’s software development process and environment. So it is ensured that once I have finished the eResearch project, the software can be maintained by NIWA staff afterwards. They also provide an
elegant and robust development environment with the Eclipse IDE, source code revision management (Subversion), continuous integration (Jenkins) and testing. This shall ensure software quality and functionality being checked automatically with every code commit for everything that is developed within the research institute.

Link to the full eResearch blog post

Friday, 21 December 2012

CLIDB-SOS: A collaborative summer internship with NIWA

The National Institute of Water and Atmospheric Research (NIWA) operates a network of weather and climate and other measurement stations, equipped with different sensors, observing a wide range of environmental properties – from temperature over rainfall to wind speed and directions. These high frequency measurements are processed and stored in the NIWA climate database (CLIDB), which is also listed as a national significant database. NIWA has already a web interface in place, where data can be queried and downloaded. Yet this is a manual process.

The Open Geospatial Consortium (OGC) is an international consortium comprising of organisations from industry and research that develops geospatial data transfer and encoding standards and specifications in an open and consensus based process. The OGC Sensor Observation Service (SOS) is a web service interface specification describing access to sensor and time series data – the observations – measured or observed primarily by sensors.


NIWA intends that SOS services become the primary delivery mechanism of its climate, hydrometric and other time-series data to internal research staff, and also central, regional and local government, businesses, NGO's, utilities and the general public via appropriate SOS web clients. Such services provide a database independent, standards compliant approach to data discovery and delivery. Furthermore the NIWA National Climate Database (CliDB) contains a Nationally Significant Database.
With a Sensor Observation Service in place, the CLIDB could actually be queried like a database, but through the web, automagically, from within an application – based on an international standard. Plugins for e.g. R Statistics (sos4r), OpenLayers (OL sos demo) or ArcGIS (ArcHydro and 52°North ArcGIS SOS Extension) are already available to demonstrate data access from SOS servers.

In this Summer of eReseach project I will build a custom connector to source CLIDB directly, based on the sophisticated 52°North SOS server. But the devil is in the details. Although I have already worked with this software I will have to learn more about the database structure of the CLIDB and immerse in the latest development version of the 52°North SOS server, which will support Hibernate, the quite new OGC SOS 2.0 specification and exchangeable output encodings (e.g. CSV, O&MWaterML2.0 and maybe JSON). Additionally we will discuss and demonstrate ways to include SOS data services in a client application.

A lot of work, but worth the effort. I am looking forward to create something new and useful. Besides the apparent advantages for New Zealand’s researchers, also consultancies, commercial organisations and governmental agencies will profit from such a dynamic, standards-driven and web-based geospatial data delivery service that could serve as another good example for environmental data providers.

The benefits of the project include well managed freshwater use & potential impacts of climate change are critical for any country to plan & manage its natural resources. One of the foundations of any effort in these domains is ready access to climate (including rainfall) information.
NIWA is collaborating with a wide range of New Zealand and international agencies to determine strategies and appropriate standards to provide interoperable discovery and delivery facilities for data managed by NIWA, as well as to develop systems to implement those strategies. These include: CSIRO, BOM (Bureau of Meteorology), AODC (Australian Ocean Data Centre), IMOS(Integrated Marine Observing System) in Australia. Iquest and Kisters (New Zealand and internationally), 52o North.
Furthermore collaboration with GNS Science in the NZ-EU cooperative groundwater project SMART (www.smart-project.infohttp://www.gns.cri.nz/) is intended, to support those who seek to incorporate climate data for their research towards characterizing New Zealand’s aquifers.

Monday, 10 December 2012

A Sensor Observation Service for the NIWA climate database

The National Institute of Water and Atmospheric Research (NIWA) operates a network of weather and climate and other measurement stations, equipped with different sensors, observing a wide range of environmental properties – from temperature over rainfall to wind speed and directions. These high frequency measurements are processed and stored in the NIWA climate database (CLIDB), which is also listed as a national significant database. NIWA has already a web interface in place, where data can be queried and downloaded. Yet this is a manual process.

The Open Geospatial Consortium (OGC) is an international consortium comprising of organisations from industry and research that develops geospatial data transfer and encoding standards and specifications in an open and consensus based process. The OGC Sensor Observation Service (SOS) is a web service interface specification describing access to sensor and time series data – the observations – measured or observed primarily by sensors.

With a Sensor Observation Service in place, the CLIDB could actually be queried like a database, but through the web, automagically, from within an application – based on an international standard. Plugins for e.g. R Statistics (sos4r), OpenLayers (OL sos demo) or ArcGIS (ArcHydro and 52°North ArcGIS SOS Extension) are already available to demonstrate data access from SOS servers.

In this Summer of eReseach project I will build a custom connector to source CLIDB directly, based on the sophisticated 52°North SOS server. But the devil is in the details. Although I have already worked with this software I will have to learn more about the database structure of the CLIDB and immerse in the latest development version of the 52°North SOS server, which will support Hibernate, the quite new OGC SOS 2.0 specification and exchangeable output encodings (e.g. CSV, O&M, WaterML2.0 and maybe JSON). Additionally we will discuss and demonstrate ways to include SOS data services in a client application.

A lot of work, but worth the effort. I am looking forward to create something new and useful. Besides the apparent advantages for New Zealand’s researchers, also consultancies, commercial organisations and governmental agencies will profit from such a dynamic, standards-driven and web-based geospatial data delivery service that could serve as another good example for environmental data providers.

Link to the eResearch blog post