A side note on geospatial data sharing and spatial data infrastructures (SDI)

This blog post is dedicated to provide a general overview over the field of geospatial and environmental data sharing. The term geospatial is actually tautologous: The prefix “geo” implies geography, which always relates things to each other based on their location, where nearer things are stronger related than things further away from each other (Tobler’s 1st law of geography). And the word “spatial” also means having an extent and location in a space. However “geospatial” nowadays is almost exclusively used in the field of digital data with geographical context. Therefore software that delivers, analyses, presents, processes, stores and retrieves is also often called geospatial software (having its origins in the good old GIS – Geographical Information Systems).

Part 1: GIS and GI Science intro

Research areas that work on the science behind GIS and spatial data, on the analytical methods, processes techniques, on ways of (standardised) spatial data exchange, effective and efficient storage and retrieval, are called GI Science, Geoinformatics, Geomatics, Geocomputation, Spatial Computation or Spatial Science … There is actually quite some discussion, if the necessity, even an entitlement for such a dedicated mixed branch of computer science and geography exists, a similar discussion when geography emerged as an accepted field of research (ref). On the other hand you’ll often find the quote that “80% of data has a spatial component” or something like that. Apparently I can get the source right anymore, therefore you might handle this with care. It might go back to the 1990s when GIS software for PCs wanted to get their feet into the market (gis lounge). However a lot of data in the geosciences have a spatial context and - a lot - of those data are needed for governmental agencies to manage land and water resources properly :smile: Agreed?

Part 2: The Opengeospatial Consortium, aka OpenGIS, aka OGC

“The Open Geospatial Consortium (OGC) is an international industry consortium of 479 companies, government agencies and universities participating in a consensus process to develop publicly available interface standards. OGC® Standards support interoperable solutions that ‘geo-enable’ the Web, wireless and location-based services and mainstream IT.”

The OGC standards framework provides means to build a spatial Data infrastructure (LINZ - Land Information New Zealand) – which is “the technology, policies, standards, and human resources necessary to acquire, process, store, distribute and improve the usability of geospatial data. (SDI) facilitates the connections between these important sources of information, and allows people to find and access them.” Quite some of the OGC standards and web services are also ISO international standards. There are interface and service descriptions on the one hand and data encodings/formats and conceptual data models on the other side. I will provide a short summary:

WMS – Web Mapping Service (ISO 19128 WMS v1.3.0)

Essentially provides (web) maps (as images like png or jpg) output data Geographiclly correct images, png, jpg, view or portrayal service

Major methods: GetCapabilties and GetMap

WFS – Web Feature Service (ISO 19142 WFS v2.0)

Provides an interface to access, query, store and retrieve vector “features”, aka discrete data – like in ESRI shapefiles. Data is accessible by their data schema, which can be soft-typed and values in schema fields and location queries are utilized. Output GML (ISO 19136), which is in a particular “domain-specific” XML schema. With WFS-T – transactional – there is also support to write back to the WFS server.

GetCapabilities GetFeature – get the data DescribeFeatureType – get schema

WCS – Web Coverage Service

Provides an interface to access, query and retrieve raster imagery and coverages, grids (aka “fields”) as in continuous data. eg NetCDF-CF, GeoTIFF, ArcGRID

GetCapabilities DescribeCoverage GetCoverage

CSW – Catalogue Service for Web (ISO 19115 CSW 2.0.2)

Provides an interface to access, query, store and retrieve metadata, aka data about the geospatial data, which is accessible through other geospatial webservices. Output is usually XML ISO 19139 metadata or Dublin core.

GetCapabilities GetRecords – find metadata record by search criteria GetRecordById – get one record by its unique id DescribeRecord – metadata type GetDomain – get range of values and/or keywords

SOS – Sensor Observation Service

Provides an interface to access, query, store and retrieve time-series based data that has been measured at locations, eg through sensors or field surveying/sampling. But the focus is to query on temporal and then on spatial or value comparison basis. Standard output formats are O&M, WaterML2.0 time-series and SensorML sensor/procedure metadata.

SOS is part of the sensor web enablement initiative (SWE) which advances to its version 2, where a lot of things become more flexible, but also moe complicated. I will write about that later ☺ SOS explicitly also describes a group of methods to insert sensor and observation data. This is handled through different profiles.

GetCapabilities GetObservation DescribeSensor

There are quite some commercial and Open Source software packages and tool kits available, for the desktop and server-based for the web that support or where explicitly written for OGC webservices and splendid Open Source resources in the web:

OSGeo Foundation: http://www.osgeo.org

(Spatially enabled) Databases:

• Postgresql/Postgis
• MySQL
• Oracle Database
• Microsoft SQL Server
• ESRI Geodatabase / ArcSDE
• SpatiaLite
• Rasdaman
• GeoCouch

Data Servers (store data in databases):

• Geoserver (WMS, WFS, WCS)
• Mapserver (WMS, WFS, SOS)
• 52°North SOS server
• Geonetwork (CSW)
• ESRI ArcIMS / ArcServer
• Thredds (WCS, netCDF)

Web mapping tool kits / frameworks (take data from data servers):

• openlayers
• Mapbender
• MapFish
• Geomajas
• Flash
• Silverlight

Desktop clients supporting (at least partially) OGC standards:

• QuantumGIS
• uDig
• ESRI ArcGIS
• Intergraph Geomedia

And on it goes … in the next weeks, I will write about New Zealand and international examples of OGC webservices implementations and SDIs for geospatial data publishing (where the data is basically public domain and needs to made accessible) and provide some closer insights to OGC SWE and the next generation sensor networks initiative SWE 2.0 et al.

A Climate Database Web Service - 1st Review

The first week before the break I spent some efforts to integrate my software development project into NIWA’s software development process and environment. So it is ensured that once I have finished the eResearch project, the software can be maintained by NIWA staff afterwards. They also provide an
elegant and robust development environment with the Eclipse IDE, source code revision management (Subversion), continuous integration (Jenkins) and testing. This shall ensure software quality and functionality being checked automatically with every code commit for everything that is developed within the research institute.

Link to the full eResearch blog post

CLIDB-SOS: A collaborative summer internship with NIWA

The National Institute of Water and Atmospheric Research (NIWA) operates a network of weather and climate and other measurement stations, equipped with different sensors, observing a wide range of environmental properties – from temperature over rainfall to wind speed and directions. These high frequency measurements are processed and stored in the NIWA climate database (CLIDB), which is also listed as a national significant database. NIWA has already a web interface in place, where data can be queried and downloaded. Yet this is a manual process.

The Open Geospatial Consortium (OGC) is an international consortium comprising of organisations from industry and research that develops geospatial data transfer and encoding standards and specifications in an open and consensus based process. The OGC Sensor Observation Service (SOS) is a web service interface specification describing access to sensor and time series data – the observations – measured or observed primarily by sensors.

NIWA intends that SOS services become the primary delivery mechanism of its climate, hydrometric and other time-series data to internal research staff, and also central, regional and local government, businesses, NGO's, utilities and the general public via appropriate SOS web clients. Such services provide a database independent, standards compliant approach to data discovery and delivery. Furthermore the NIWA National Climate Database (CliDB) contains a Nationally Significant Database.

With a Sensor Observation Service in place, the CLIDB could actually be queried like a database, but through the web, automagically, from within an application – based on an international standard. Plugins for e.g. R Statistics (sos4r), OpenLayers (OL sos demo) or ArcGIS (ArcHydro and 52°North ArcGIS SOS Extension) are already available to demonstrate data access from SOS servers.

In this Summer of eReseach project I will build a custom connector to source CLIDB directly, based on the sophisticated 52°North SOS server. But the devil is in the details. Although I have already worked with this software I will have to learn more about the database structure of the CLIDB and immerse in the latest development version of the 52°North SOS server, which will support Hibernate, the quite new OGC SOS 2.0 specification and exchangeable output encodings (e.g. CSV, O&M, WaterML2.0 and maybe JSON). Additionally we will discuss and demonstrate ways to include SOS data services in a client application.

A lot of work, but worth the effort. I am looking forward to create something new and useful. Besides the apparent advantages for New Zealand’s researchers, also consultancies, commercial organisations and governmental agencies will profit from such a dynamic, standards-driven and web-based geospatial data delivery service that could serve as another good example for environmental data providers.

The benefits of the project include well managed freshwater use & potential impacts of climate change are critical for any country to plan & manage its natural resources. One of the foundations of any effort in these domains is ready access to climate (including rainfall) information.

NIWA is collaborating with a wide range of New Zealand and international agencies to determine strategies and appropriate standards to provide interoperable discovery and delivery facilities for data managed by NIWA, as well as to develop systems to implement those strategies. These include: CSIRO, BOM (Bureau of Meteorology), AODC (Australian Ocean Data Centre), IMOS(Integrated Marine Observing System) in Australia. Iquest and Kisters (New Zealand and internationally), 52o North.

Furthermore collaboration with GNS Science in the NZ-EU cooperative groundwater project SMART (www.smart-project.info, http://www.gns.cri.nz/) is intended, to support those who seek to incorporate climate data for their research towards characterizing New Zealand’s aquifers.

A Sensor Observation Service for the NIWA climate database

The National Institute of Water and Atmospheric Research (NIWA) operates a network of weather and climate and other measurement stations, equipped with different sensors, observing a wide range of environmental properties – from temperature over rainfall to wind speed and directions. These high frequency measurements are processed and stored in the NIWA climate database (CLIDB), which is also listed as a national significant database. NIWA has already a web interface in place, where data can be queried and downloaded. Yet this is a manual process.

The Open Geospatial Consortium (OGC) is an international consortium comprising of organisations from industry and research that develops geospatial data transfer and encoding standards and specifications in an open and consensus based process. The OGC Sensor Observation Service (SOS) is a web service interface specification describing access to sensor and time series data – the observations – measured or observed primarily by sensors.

With a Sensor Observation Service in place, the CLIDB could actually be queried like a database, but through the web, automagically, from within an application – based on an international standard. Plugins for e.g. R Statistics (sos4r), OpenLayers (OL sos demo) or ArcGIS (ArcHydro and 52°North ArcGIS SOS Extension) are already available to demonstrate data access from SOS servers.

In this Summer of eReseach project I will build a custom connector to source CLIDB directly, based on the sophisticated 52°North SOS server. But the devil is in the details. Although I have already worked with this software I will have to learn more about the database structure of the CLIDB and immerse in the latest development version of the 52°North SOS server, which will support Hibernate, the quite new OGC SOS 2.0 specification and exchangeable output encodings (e.g. CSV, O&M, WaterML2.0 and maybe JSON). Additionally we will discuss and demonstrate ways to include SOS data services in a client application.

A lot of work, but worth the effort. I am looking forward to create something new and useful. Besides the apparent advantages for New Zealand’s researchers, also consultancies, commercial organisations and governmental agencies will profit from such a dynamic, standards-driven and web-based geospatial data delivery service that could serve as another good example for environmental data providers.

Link to the eResearch blog post

Starting to play with spatial-temporal data

Thinking  the big picture is a different thing to actually implement it :-) Well, who doesn't know that. Having some hydrological time-series (groundwater levels) in place in a sensor observation service (SOS), the hydrogeological all-in-one-wonder-portal is going to get a glance of the next level :-p
The last weeks I started to play around with the R environment for statistical computing and visualization (The R Project). 52°North developed a neat R toolkit to access and digest SOS time-series - sos4R.

It is well documented and pretty easy to connect to a SOS server, and query observations. So for the fun of it and to demonstrate the general feasibilty, I quickly queried the groundwater levels of the Horowhenua area in New Zealand, where I got some sample data (courtesy by the regional council).

With the R sos4R, fields and akima packages from the CRAN R packages archive I (quite coarsely) interpolated the groundwater surfaces for the years 1991-2009 and put the images together as an animated gif (meters above mean sea level over time).

Discussion

I am aware of the total uselessness of this particualr way presenting :-) No years, the scale changes slightly, and the exact spatial extent and north orientiation are not reliable :-p

Nevertheless, for just playing around, this was a motivating simple first shot to easily visualise changes over time.

I would like to play around with the gstat and spacetime R package, integrate a more sophisticated script as a 52°North WPS process and have those things happening automagically in the interwebz.

GSoC and Master Thesis completed

Busy summer, well or winter, depends on the hemisphere and the full time extent. Last 2-3 months have been packed with work and included about 25.000 air miles (which means concatenated spending several days in airplanes).
I finally managed to finish my Master Thesis "Towards a 4D WebGIS using harmonised datasets: Examined on a New Zealand Example" which examines in the first part existing groundwater-related geoportal projects and scientific applications around the globe, existing and useful, web-based technologies and standards (e.g. OGC) to implement such a portal, a draft architecture. In the second part actual prototypical implementations of a selected subset of the examined standards and and technologies demonstrate their feasibility for the SMART project. I also hold a talk at the GI_Forum conference in Salzburg (proceedings/conference paper).
Furthermore I successfully completed my Google Summer of Code (GSoC) project with the 52°North Initiative, where I implemented an exchangeable encodings mechanism for their Sensor Observation Service (SOS). I additionally implemented a plugin that provides SOS time-series output in the WaterML2.0 format. I could acquire the latest schema, which will be published soon.. we were supposed to write a series of blog articles to document or progress in the project:

• Dynamic output formats for the Sensor Observation Service
• Exchangeable Encodings getting in shape
• Demonstrating Exchangeable Encodings: A Quantum Leap for me

After flying back to New Zealand I was lucky to visit the GeoSciML Face-to-Face-Meeting at GNS Science in Wellington. It was really exciting to meet these people, working on the international standard of a Geoscience Markup Language. also issues around the OneGeology portal have been disussed, as it uses GeoSciML to build up a world geology map. Additionally I could get in touch with one of the inspiring driving persons behind the Canadian Groundwater Information Network and their its sematic foundation, the Groundwater Markup Language (GWML), which itself is derived from GeoSciML.
Now I will dedicate my research to the development of a New Zealand groundwater geoportal within the SMART project, using OGC webservices, GeoSciML and/or GWML, WaterML2.0 and X3D as its foundations to transform and consume all available data relevant to aquifer characterisation and create a completely new, visually appealing 3D/4D view on New Zealand's groundwater resources.

Exciting ;-)

Demonstrating Exchangeable Encodings for SOS - A Quantum Leap

The final sprint towards the end of the Google Summer of Code (GSoC) is over now. After fixing bugs, working on documentation and actually producing time-series in different formats, the “Exchangeable Encodings for SOS”
project met most of its ambitious goals – and it is demonstrated on the 52°North Google summer of Code Demo Server. Users and developers documenation on the encoding mechanism has also found a place in the 52°North wiki.

We have integrated a plugin mechanism into an SOS server (source code branch), which is based on the main 52°North SOS development line (source code trunk).

Link to the article on the 52°North blog