A Spatial Data Infrastructure Approach for the Characterization of New Zealand's Groundwater Systems

I was very happy when I got informed that our latest research article was published in "Transactions in GIS". While being embedded in our New Zealand SMART Aquifer Characterisation programme, it was a great joint effort which results went beyond SMART.

Kmoch, A., Klug, H., Ritchie, A. B. H., Schmidt, J. and White, P. A. (2015), A Spatial Data Infrastructure Approach for the Characterization of New Zealand's Groundwater Systems. Transactions in GIS. doi: 10.1111/tgis.12171

It explains technical and methodological approaches of web groundwater data infrastructure and uses NIWA's CLIDB and GNS's NGMP as examples and describes the long way from stakeholder interaction, workshops and meetings towards an NZ data standard for the National Environmental Monitoring Standards (NEMS) framework - the Environmental Observation Data Profile (EODP) as data access and transfer blueprint for NEMS.

Wiley Online Library "Transactions in GIS" Journal

Alexander Kmoch on ResearchGate.

The technical work with colleagues, collaborators and partners from New Zealand research institutes (CRIs) like GNS, NIWA, Landcare Research, New Zealand regional councils like Horizons and Waikato (WRC), Hawke's Bay (HBRC) and and Bay of Plenty (BOPRC) regional councils which culminated in a draft OGC profile and is being incorporated into national standards in New Zealand, GitHub EODP and are based on the work over the last two years including a great review paper on OGC standards for groundwater,  customising the 52°North SOS server to interchangeably encode WaterML2 time-series data along O&M2 through a Google Summer of Code (GSoC) programme, and then customising the SOS server database access to demonstrate it as an adaptor on legacy databases of national significance, in specific an "NGMP/GGW-SOS" and a "CLIDB-SOS" demo through a NZ eResearch programme.

Geospatial web-enablement for environmental data in New Zealand

This blog post can be seen as a sequel to a former blog post on the introduction on geospatial data sharing and spatial data infrastructures (SDI), where I explained the basics of OGC standards and web services. Quite some research organisations and governmental agencies already employ OGC standards to make data available online, often even free of charge for the public. I would like to present some really good examples of interoperable data sharing in New Zealand.

Through the standardised and web-based access to so many data sources, not only traditional geographical processing and analysis (GIS) based research is made easier, but also complete new technical and methodological research possibilities arise.

LINZ - Land Information New Zealand

I would like to start with Land Information New Zealand (LINZ). LINZ, as a governmental body, has issued and maintains New Zealand’s geospatial strategy. LINZ runs the LINZ Data Service, which provide tons of NZ-related data sets, topography, maps, place names and much more, almost all of it is available under a NZ Creative Commons license. You can register for free, get an API key and use data directly through web, basically as long as you tell that it is LINZ data. LINZ provides standard OGC CSW, WMS and WFS web service interfaces.

http://data.linz.govt.nz

More news about the NZ geospatial strategy can be found on here.

DOC – Department of Conservation

Also the New Zealand Department of Conservation is going towards geospatial web services. It looks like they use ESRI software, which supports OGC standards to certain bit already, although ESRI (producer of the ArcGIS software) as a commercial closed-source software provider has been known to notoriously neglect open standards. However, the Shapefile format is open and besides ESRI REST services, the DOC Geoportal also allows for OGC-based access (CSW/ISO 19139 metadata for search and discovery and WMS/WFS for map/feature data access)

http://geoportal.doc.govt.nz

GNS Science

The Institute of Geological and Nuclear Sciences is one of the 9 New Zealand Crown Research Institutes (CRI), which conduct about one half publicly/governmentally funded and the other half commercial research projects and, together with the universities of course, can be seen as New Zealand’s main science and research providers, each claiming a particular scientific domains. GNS Science is New Zealand’s leading provider of Earth, geoscience and isotope research and the geological survey of New Zealand. GNS’s research topics also include volcanoes, earthquakes, geothermal features and groundwater.

GNS has published the 1:250 000 Geological Map of New Zealand (QMAP. It is also digitally accessible – GNS exports the QMAP as OGC WMS and WFS in the OGC format GeoSciML. An easy way and very interesting example for interoperability is to explore New Zealand’s geology is through the OneGeology project, which sources and displays such services from geological surveys from all over the world.

The GNS-EU collaborative SMART Acquifer Characterisation programme (SAC) also aims to connect OGC based data sources. Within the research aim “Data Synthesis and Visualisation” the SMART Data Portal aims to develop an integrated OGC framework for discovery, access, processing and visualisation of hydrogeological data.

http://data.gns.cri.nz/metadata/

NIWA – National Institute of Water and Atmospheric Research

NIWA NIWA, another CRI, has a strong reputation in climate, marine and marine ecosystem and biodiversity sciences. Whereas a lot of organisations and agencies make data available first and then (if at all) add more sophisticated search technology, NIWA started the other way round. They established a discovery portal - the Environmental Information Browser, which is basically a catalogue, where one can search by keywords, places, data and time. All the data NIWA has, will eventually be listed and can be queried and also harvested through the OGC CSW interface. Furthermore NIWA is also moving towards providing OGC web services to their data sets. One particular example has been a “Summer of eResearch” project and its progress documented on the eResearch website.

http://ei.niwa.co.nz

Landcare Research

Landcare Research is also a New Zealand Crown Research Institute and focuses on the management of terrestrial biodiversity and land resources in order to both protect and enhance the terrestrial environment. I have come across several Landcare projects on soils and land use data, where Landcare not only uses OGC standards, but also participates in the development and maturing of some of those standards. Like the former parties, Landcare runs a data or geoportal (LRIS), too, which can be accessed and queried through CSW, WMS and WFS web interfaces,

https://lris.scinfo.org.nz/

Landcare also hosts a dedicated soil map portal (S-map), which sources the digital soil information layers based on WMS. Furthermore they drive the development of a global soil map portal (http://www.globalsoilmap.net/), which under the hood, of course, uses OGC standards again. To enable international, comparable, interoperable soil data exchange Landcare participates in the development of a soil information standard.

Outlook

Regional councils are on the way, too. Many regional councils already make data accessible on their web sites. A quick investigation shows for example Environment Waikato, Horizons, HBRC, BOP or Environment Canterbury. However most of these data sources need to be accessed manually and/or do not provide a standardised interface. Not to speak of a generalised way to actually find them. In conjunction with the open data initiatives (Open and Transparent Government , Open New Zealand) and catalogues available ( government datasets online, Open Data Catalogue), there is massive potential to link diverse datasets, relate and analyse seemingly unrelated datasets and gain new insights, find and (re-use) data by type, time and location or just enable ubiquitous mobile access to the data you need. However, we might end up needing a catalogue for the catalogues, and of course a lot of existing data needs to be geo-located/geo-referenced, so that they could be found by location. There is still a way to go and definitely some more research necessary in that space.

Originally posted on the New Zealand eResearch website - 16/04/2013

The Devil is in the Details

As one of the 52°North Google Summer of Code projects, “Exchangeable Encodings for SOS” aims to implement a customisable result set encoding mechanism.   If that mechanism works,  users could add an own pre-compiled .jar-file to the SOS that contains all the necessary code to encode the requested data in simpler formats like CSV or GeoJSON – but for now, where to start?

The architecture of the SOS server is generally well structured. Based on the request – GetCapabilities, GetObservation – and given parameters, like Offering, FeatureOfInterest and/or time the RequestOperator calls the respective Listener. The Listener calls the DAO, which fills an internal response object and gives it back to the Listener. The response object, most likely a SosObservationCollection is then sent to the respective encoding module actually based on the SOS version from the request. Then the Listener decides, which Encoder will be used - to date the decision is mainly based on the requested SOS protocol version. The Encoder now creates an XML object based on the OGC O&M schema and gives it back to the Listener. Finally the Listener puts the textual representation of that XML object into the response document that will be returned via http. This is good, because other encodings are likely to be plain/text, too.

There are different entry points to dynamacally add encodings, either as additional full-fledged SOS server modules or as another responseFormat internally be evaluated and the respective encoder selected through the SosConfigurator instance. But following constraints are to be regarded:

• The SOS protocol versions 1.0 and 2.0 must be honoured
• responseFormat O&M 2.0 ships as standard
• Listener and Encoder are coupled through the SOS protocol version, the response encoding needs to be decoupled and dynamically selectable
• Other formats / encoders need to be “dynamically” registered in the SoSConfigurator and to be identified and allocated based on the requested ResponseFormat value from the request

Simple sequence diagram of a GetObservation request to the SOS server

Finally one of the challenges I suppose, is the generation of a non-XML-response in this XML dominated workflow. As the SOS internal response object is merely of the String datatype, a simple encoder would need to implement the ISosResponse for the return value and should accept the SOS internal data collection representation as delivered from the DAO backend. Here in between we will put the lever to enhance the versatility of the 52 North SOS server.

52North SOS Server for NGMP

52North SOS Server for NGMP

As I recently wrote about the three major work packages (Boiling it down), I mentoined the publishing of groundwater measurements from a legacy DB. Actually it is the GGW database (GNS Science Geothermal and Groundwater Database) which besides a lot of other stuff (geothermal data for Geothermal and volcanic monitoring, GeoNET), holds the data of the National Groundwater Monitoring Programme (NGMP). This is a governmental sponsored, freely accessible data set that holds groundwater measurements and analysis results (mostly quarterly or half-year basis). But this data can only be accessed via a web-frontend, where registered users can download pdf or excel reports including the meausred values over time.

My ambitious first pillar of the SMART data portal prototype is to consume and display data from the NGMP. Therefore I need to access the NGMP data (technically the GGW DB). And as we want to demonstrate useful SOS data usage, I need to put a SOS server on top of it!

In search for a proper way to serve NGMP data via SOS, I came across two major Open Source players, the 52North initiative, the ecosystem around OOSThethys and istSOS, from the Geomatics department is part of the Earth Science Institute (IST) at the University of Applied Sciences of the South Switzerlad (SUPSI).

http://52north.org/communities/sensorweb/sos/
https://wiki.52north.org/bin/view/Sensornet/SensorObservationService

http://www.oostethys.org/
http://code.google.com/p/oostethys/

http://istgeo.ist.supsi.ch/software/istsos/
https://sites.google.com/site/istssosproject/

The 52North SOS server is written in Java and has a roburst architecture, which especially separates the data access from the main procesing modules, a so called Data Access Object design pattern. This seems very handy, as I needed to access a completely different database and data model. The OOSTethys project has several different packages in different programming languages with different levels of sophistication and the istSOS server is written python.

[todo: reasons for decision should be more documented :-) ]

I decided for the 52North software, it has a vibrant community that strives for a bigger integration pattern. They  are providing a broad OGC eb services solution portfolio and often tend have recent standards already (at least prototypically or in beta status) implemented. OGC recently approved SOS 2.0 specification and 52North already has began work to incorporate that (52North Blog).

"52n-sos-dao-oracle"

As the NGMP / GGW database unfortunately has no spatial extensions, the planned custom backend needs to be able to abstract / fake this functionality in the software itself, instead of using typical spatial DB extensions (e.g. Postgis geometry datatype and geometric functions, BBOX search, etc).

52North SOS data model (Source: 52North Wiki)

I started by copying the existing "52n-sos-dao-postgis" to "52n-sos-dao-oracle", included the ojdbc14.jar driver and adjusted the dssos.config and main pom.xml to build the new DAO, load the oracle db driver and connect to the oracle db.
That was "almost" easy, the server builds, connects to the db and dies, as the queries from the original Postgis DAO obviously don't work.

Now the heavy-duty data part, actually the really interesting part, begins...

As the GGW / NGMP data model ist based on the Australian National Groundwater Data Transfer Standard from July 1999 and has been modified to suite the demands of GNS Science's geothermal and groundwater data collection, the schema is huge and the model quite complex. After consultion with the DBAs on the side and the scientists on the other side, I got a feeling for the whole thing and started to find things via joining the right views and tables to respect necessary constraints.

Australian National Groundwater Data Transfer Standard (Release 1.0 July 1999)

Then I began to adjust the queries in the ConfigDAO and the DAOFactory to select the proper fields to make the server initialisation work. But to keep it simple for the beginning I concentrated on only one property out of several dozens with available data - water level.

The next big challenge is coming - geometry. The coordinates are in two columns related to the groundwater The spatial reference system is well-known (EPSG 27200, NZGD49 / New Zealand Map Grid) and the coordinates are in two columns. So I intend to use Geotools to build point features for the "groundwater features" that are supposed to be wells, spings and bores and make them availabe throughout the SOS internal data structure, just as the other (frequently updated items in CapabiltiesCache) list.

Good Luck :-)