AGILE 2016 - SensorWeb Semantics on MQTT for responsive Rainfall Recharge Modelling

Integrating Wireless Sensor Networks (WSNs) and spatial data web services is becoming common in ecological applications. However, WSNs were developed in application domains with different sensor and user types, and often with their own low-level metadata semantics, data format and communication protocols. The sensor web enablement initiative (SWE) within the Open Geospatial Consortium (OGC) has released a set of open standards for interoperable interface specifications and (meta) data encodings for the real time integration of sensors and sensor networks into a web services architecture.
Such XML-based web services exhibit disadvantages in terms of payload and connectivity in low-bandwidth low energy unreliable networks, such as remote 3G uplinks. Monitoring stations deliver frequent measurements in real-time, but dynamic implementation of measurement frequencies, adapted to certain environmental conditions, are rarely implemented. We describe a responsive integrated hydrological monitoring prototype to calculate rainfall recharge for water management purposes.
When rainfall is observed, a threshold event triggers a reconfiguration task for the soil moisture sensors, using asynchronous, push-based communication implemented with an MQTT queue. A Sensor Planning Service commits that request via MQTT into the wireless sensor network, and updates the measurement frequency of the target sensors to gain higher resolution for the vertical soil water infiltration.
The system integrates a Sensor Observation Service (SOS) including field observations and internet-based environmental data with a rainfall recharge model that allows near-real time calculation of rainfall recharge in the Upper Rangitaiki catchment, Bay of Plenty region in New Zealand.

Figure 1: Setup and location of the sensor field site, central North Island, New Zealand

The prototype site comprises a main station conducting comprehensive measurements of meteorological, hydrological and pedological parameters. For the wireless data transmission within the local site installation XBee-PRO modules from the Digi Company  ZigBee IEEE 802.15.4 protocol are used. The main station receives continuous sensor measurements from the attached sensor units, and acts as the gateway to the online SOS and SPS services by providing the communication channel from the local sensor network to the web-enabled data management infrastructure.

The field site has been established in the Upper Rangitaiki catchment (Figure 1) and comprises a field computer (Raspberry Pi) with a direct internet link (GPRS/3G) and a sensor board (Waspmote) that has 12 typical meteorological, hydrological and pedological sensors attached (i.e., wind speed, wind direction, rainfall, 1x groundwater probe, 5x temperature and 3x soil moisture). The Raspberry Pi and Waspmote can be monitored and reprogrammed from an online server.

Figure 2: Raw sensor series visualized in a website from a SOS query.

The site setup allows scaling up to a multitude of low cost, low energy sensor stations throughout the catchment, with only one field computer that serves as data logger for backup. The observations were available in a standardized open format. The website accessed the raw data from the SOS server and plotted data points within 5-10 minutes of field measurement. This website was easily accessible via browsers and smartphones (Figure 2).

The paper was was presented at the 19th AGILE International Conference on Geographic Information Science, 15th of June, in  Helsinki, Finlkand.

Kmoch, A., Klug, H., White, P., & Reichel, S. (2016). SensorWeb Semantics on MQTT for responsive Rainfall Recharge Modelling. In 19th AGILE International Conference on Geographic Information Science. Helsinki.

Amazon IoT and OGC SensorThings vs SOS/SPS on MQTT

Today I stumbled over a blog post on the OGC Blog about "Amazon IoT and the candidate OGC SensorThings API Standard"

It was great to learn that Amazon IoT uses MQTT as network / transport layer. The more interesting is the move towards REST (which is HTTP) to address semantic issues with IoT systems:

AWS IoT, MQTT and many other network interoperability standards (LWM2M, CoAP, etc.) enable message interchange.  However, IoT network interoperability doesn’t enable the systems that are exchanging messages to interpret those messages. To realize the many-to-many system-of-systems vision, IoT applications need to implement standard ways of communicating sensor locations, sensor and data parameters, and sensor instruction sets. This is what the OGC SensorThings API provides.

It is understandable from the point of view, that OGC OWS is in general intrinsically interwoven with HTTP semantics and the application of SWE in the IoT context is desirable, well, even crucial.

However, MQTT stems from the constraints of low bandwidth, low energy consumption and unreliable connectivity, and implements a robust, small footprint asynchronous messaging platform. Earlier this we have developed a prototype system, that aims to add SWE semantics into MQTT messaging. We keep the specialisation of MQTT, where for example connections can be resumed which under SSL/TLS security can be a significant power saver. MQTT would also better address the "addressability" of distributed "things" which might be isolated behind IP NAT or run with dynamically assigned addresses, because in MQTT context the "things" would subscribe to SPS management topics and therefore higher order SDI services don't need to direct data queries or task requests to an address but place it on the queue with respective topics.

For an agricultural research project we distributed monitoring stations as a wireless sensor network throughout the catchment to deliver frequent measurements in near-realtime, which mimics OGC SOS.

A dynamic implementation of measurement frequencies, adapted to certain environmental conditions like heavy rainfall events require re-tasking of the nodes, which mimics OGC SPS. Within this paper we provide a framework where a threshold event triggers a reconfiguration task for a phosphorus measurement device, using asynchronous, push-based communication, on an MQTT queue, which links the ground stations with a cloud-based control system. OGC WPS algorithms continuously analyse incoming SOS measurements commits such a request into the wireless sensor network, and updates the measurement frequency of the target nodes to enable nutrient peak flow estimation during storm events.

This on-going work is based on original thoughts on the  interlinking  of  OGC  SWE  semantics  for  sensor  descriptions  and  observations (OGC Sensor Observation Service, SOS), sensor configuration and planning (OGC  Sensor Planning Service, SPS)  with  the  open  Message  Queue  Telemetry  Transport  (MQTT)  protocol.

MQTT  is  a  simple,  yet  very performing  and  robust  publish/subscribe  message  passing  system,  which  can  also  serve  for  event  brokering as  described  in  the  SWE  Service  Model.  Actual  addressing  and  data  transmission  is  done  by  a  topic  string and an arbitrary payload. The topic string and the payload need to reflect the necessary standard semantics to be  mapped  back  and  forth  to  allow  for  a  seamless,  and  preferably  lossless  bi-directional  multi-lateral communication  between  the  WSN  as  data  provider  (SOS),  web  clients  as  data  consumers  (SOS)  and  the management system, which interacts with the WSN in a standardised way, too (SOS and SPS). 

A Spatial Data Infrastructure Approach for the Characterization of New Zealand's Groundwater Systems

I was very happy when I got informed that our latest research article was published in "Transactions in GIS". While being embedded in our New Zealand SMART Aquifer Characterisation programme, it was a great joint effort which results went beyond SMART.

Kmoch, A., Klug, H., Ritchie, A. B. H., Schmidt, J. and White, P. A. (2015), A Spatial Data Infrastructure Approach for the Characterization of New Zealand's Groundwater Systems. Transactions in GIS. doi: 10.1111/tgis.12171

It explains technical and methodological approaches of web groundwater data infrastructure and uses NIWA's CLIDB and GNS's NGMP as examples and describes the long way from stakeholder interaction, workshops and meetings towards an NZ data standard for the National Environmental Monitoring Standards (NEMS) framework - the Environmental Observation Data Profile (EODP) as data access and transfer blueprint for NEMS.

Wiley Online Library "Transactions in GIS" Journal

Alexander Kmoch on ResearchGate.

The technical work with colleagues, collaborators and partners from New Zealand research institutes (CRIs) like GNS, NIWA, Landcare Research, New Zealand regional councils like Horizons and Waikato (WRC), Hawke's Bay (HBRC) and and Bay of Plenty (BOPRC) regional councils which culminated in a draft OGC profile and is being incorporated into national standards in New Zealand, GitHub EODP and are based on the work over the last two years including a great review paper on OGC standards for groundwater,  customising the 52°North SOS server to interchangeably encode WaterML2 time-series data along O&M2 through a Google Summer of Code (GSoC) programme, and then customising the SOS server database access to demonstrate it as an adaptor on legacy databases of national significance, in specific an "NGMP/GGW-SOS" and a "CLIDB-SOS" demo through a NZ eResearch programme.

Archived posts from my Github pages

• July 16, 2015: Google Cloud Platform, Cassandra and Mesos
• July 14, 2015: Insights into Raster Imagery Tiling with GeoTrellis
• June 30, 2015: A GeoTrellis Spark Cassandra Experiment on Google Compute Engine
• June 05, 2015: Setting up your IDE for development with GeoTrellis (or any other complex Scala project)
• March 27, 2015: GeoTrellis Cassandra Backend to GeoTrellis-Spark
• March 26, 2015: ZOO-Project Java-API and JGrasstools
• April 13, 2014: Space Apps Challenge 2014 - TETRIS Space Suit HUD demo with a Raspberry
• April 16, 2013: Geospatial web-enablement for environmental data in New Zealand
• February 04, 2013: SOS for CLIDB Mid-term - we got data
• January 21, 2013: A side note on geospatial data sharing and spatial data infrastructures (SDI)
• January 14, 2013: A Climate Database Web Service - 1st Review
• December 10, 2012: A Sensor Observation Service for the NIWA climate database
• August 23, 2012: Demonstrating Exchangeable Encodings for SOS - A Quantum Leap for me
• July 04, 2012: Exchangeable Encodings Getting in Shape
• May 30, 2012: Dynamic Output Formats for the Sensor Observation Service
• February 29, 2012: Exchangeable Encodings for SOS - extension support WaterML2

Disclaimer: There might be some duplication with older posts here, too :-)

Random Reflections

Kia Ora :-)

The last weeks, two months were packed with experiences, developments, progress. I would like to shortly summarise the main three happenings. It is great to see progress and experience interest in that progress, so I'd like to share that, too :-)

New Zealand Hydrological Society Conference 2012, Nelson and the Data Access Workshop

• Water: know your limits (linked from NZHS website)
• Data Access and Visualisation Workshop at HydSoc (resources)

New Zealand Summer of eResearch (Dec 2012 - Feb 2013)

• NZ Summer of eResearch (Centre for eResearch, Auckland University)
• Sensor Observation Service (OGC SOS) for NIWA CLIDB
• Blog posts within the Summer of eResearch project context
• Introduction
• A first review
• A side note on geospatial data sharing
• Mid-Term review

SCANZ 2013 3rd nature hui and wananga symposium

• Intercreate.org and SCANZ 2013: 3rd nature
• Wananga-symposium (Friday to Sunday)
• Our abstract - a groundwater portal for New Zealand, towards a Te Reo Maori and Matauranga Maori integration with "Western Science"

This is sooo much stuff, I will have to write out details later :-) So far enjoy the links.

A side note on geospatial data sharing and spatial data infrastructures (SDI)

This blog post is dedicated to provide a general overview over the field of geospatial and environmental data sharing. The term geospatial is actually tautologous: The prefix “geo” implies geography, which always relates things to each other based on their location, where nearer things are stronger related than things further away from each other (Tobler’s 1st law of geography). And the word “spatial” also means having an extent and location in a space. However “geospatial” nowadays is almost exclusively used in the field of digital data with geographical context. Therefore software that delivers, analyses, presents, processes, stores and retrieves is also often called geospatial software (having its origins in the good old GIS – Geographical Information Systems).

Part 1: GIS and GI Science intro

Research areas that work on the science behind GIS and spatial data, on the analytical methods, processes techniques, on ways of (standardised) spatial data exchange, effective and efficient storage and retrieval, are called GI Science, Geoinformatics, Geomatics, Geocomputation, Spatial Computation or Spatial Science … There is actually quite some discussion, if the necessity, even an entitlement for such a dedicated mixed branch of computer science and geography exists, a similar discussion when geography emerged as an accepted field of research (ref). On the other hand you’ll often find the quote that “80% of data has a spatial component” or something like that. Apparently I can get the source right anymore, therefore you might handle this with care. It might go back to the 1990s when GIS software for PCs wanted to get their feet into the market (gis lounge). However a lot of data in the geosciences have a spatial context and - a lot - of those data are needed for governmental agencies to manage land and water resources properly :smile: Agreed?

Part 2: The Opengeospatial Consortium, aka OpenGIS, aka OGC

“The Open Geospatial Consortium (OGC) is an international industry consortium of 479 companies, government agencies and universities participating in a consensus process to develop publicly available interface standards. OGC® Standards support interoperable solutions that ‘geo-enable’ the Web, wireless and location-based services and mainstream IT.”

The OGC standards framework provides means to build a spatial Data infrastructure (LINZ - Land Information New Zealand) – which is “the technology, policies, standards, and human resources necessary to acquire, process, store, distribute and improve the usability of geospatial data. (SDI) facilitates the connections between these important sources of information, and allows people to find and access them.” Quite some of the OGC standards and web services are also ISO international standards. There are interface and service descriptions on the one hand and data encodings/formats and conceptual data models on the other side. I will provide a short summary:

WMS – Web Mapping Service (ISO 19128 WMS v1.3.0)

Essentially provides (web) maps (as images like png or jpg) output data Geographiclly correct images, png, jpg, view or portrayal service

Major methods: GetCapabilties and GetMap

WFS – Web Feature Service (ISO 19142 WFS v2.0)

Provides an interface to access, query, store and retrieve vector “features”, aka discrete data – like in ESRI shapefiles. Data is accessible by their data schema, which can be soft-typed and values in schema fields and location queries are utilized. Output GML (ISO 19136), which is in a particular “domain-specific” XML schema. With WFS-T – transactional – there is also support to write back to the WFS server.

GetCapabilities GetFeature – get the data DescribeFeatureType – get schema

WCS – Web Coverage Service

Provides an interface to access, query and retrieve raster imagery and coverages, grids (aka “fields”) as in continuous data. eg NetCDF-CF, GeoTIFF, ArcGRID

GetCapabilities DescribeCoverage GetCoverage

CSW – Catalogue Service for Web (ISO 19115 CSW 2.0.2)

Provides an interface to access, query, store and retrieve metadata, aka data about the geospatial data, which is accessible through other geospatial webservices. Output is usually XML ISO 19139 metadata or Dublin core.

GetCapabilities GetRecords – find metadata record by search criteria GetRecordById – get one record by its unique id DescribeRecord – metadata type GetDomain – get range of values and/or keywords

SOS – Sensor Observation Service

Provides an interface to access, query, store and retrieve time-series based data that has been measured at locations, eg through sensors or field surveying/sampling. But the focus is to query on temporal and then on spatial or value comparison basis. Standard output formats are O&M, WaterML2.0 time-series and SensorML sensor/procedure metadata.

SOS is part of the sensor web enablement initiative (SWE) which advances to its version 2, where a lot of things become more flexible, but also moe complicated. I will write about that later ☺ SOS explicitly also describes a group of methods to insert sensor and observation data. This is handled through different profiles.

GetCapabilities GetObservation DescribeSensor

There are quite some commercial and Open Source software packages and tool kits available, for the desktop and server-based for the web that support or where explicitly written for OGC webservices and splendid Open Source resources in the web:

OSGeo Foundation: http://www.osgeo.org

(Spatially enabled) Databases:

• Postgresql/Postgis
• MySQL
• Oracle Database
• Microsoft SQL Server
• ESRI Geodatabase / ArcSDE
• SpatiaLite
• Rasdaman
• GeoCouch

Data Servers (store data in databases):

• Geoserver (WMS, WFS, WCS)
• Mapserver (WMS, WFS, SOS)
• 52°North SOS server
• Geonetwork (CSW)
• ESRI ArcIMS / ArcServer
• Thredds (WCS, netCDF)

Web mapping tool kits / frameworks (take data from data servers):

• openlayers
• Mapbender
• MapFish
• Geomajas
• Flash
• Silverlight

Desktop clients supporting (at least partially) OGC standards:

• QuantumGIS
• uDig
• ESRI ArcGIS
• Intergraph Geomedia

And on it goes … in the next weeks, I will write about New Zealand and international examples of OGC webservices implementations and SDIs for geospatial data publishing (where the data is basically public domain and needs to made accessible) and provide some closer insights to OGC SWE and the next generation sensor networks initiative SWE 2.0 et al.

A Climate Database Web Service - 1st Review

The first week before the break I spent some efforts to integrate my software development project into NIWA’s software development process and environment. So it is ensured that once I have finished the eResearch project, the software can be maintained by NIWA staff afterwards. They also provide an
elegant and robust development environment with the Eclipse IDE, source code revision management (Subversion), continuous integration (Jenkins) and testing. This shall ensure software quality and functionality being checked automatically with every code commit for everything that is developed within the research institute.

Link to the full eResearch blog post

CLIDB-SOS: A collaborative summer internship with NIWA

The National Institute of Water and Atmospheric Research (NIWA) operates a network of weather and climate and other measurement stations, equipped with different sensors, observing a wide range of environmental properties – from temperature over rainfall to wind speed and directions. These high frequency measurements are processed and stored in the NIWA climate database (CLIDB), which is also listed as a national significant database. NIWA has already a web interface in place, where data can be queried and downloaded. Yet this is a manual process.

The Open Geospatial Consortium (OGC) is an international consortium comprising of organisations from industry and research that develops geospatial data transfer and encoding standards and specifications in an open and consensus based process. The OGC Sensor Observation Service (SOS) is a web service interface specification describing access to sensor and time series data – the observations – measured or observed primarily by sensors.

NIWA intends that SOS services become the primary delivery mechanism of its climate, hydrometric and other time-series data to internal research staff, and also central, regional and local government, businesses, NGO's, utilities and the general public via appropriate SOS web clients. Such services provide a database independent, standards compliant approach to data discovery and delivery. Furthermore the NIWA National Climate Database (CliDB) contains a Nationally Significant Database.

With a Sensor Observation Service in place, the CLIDB could actually be queried like a database, but through the web, automagically, from within an application – based on an international standard. Plugins for e.g. R Statistics (sos4r), OpenLayers (OL sos demo) or ArcGIS (ArcHydro and 52°North ArcGIS SOS Extension) are already available to demonstrate data access from SOS servers.

In this Summer of eReseach project I will build a custom connector to source CLIDB directly, based on the sophisticated 52°North SOS server. But the devil is in the details. Although I have already worked with this software I will have to learn more about the database structure of the CLIDB and immerse in the latest development version of the 52°North SOS server, which will support Hibernate, the quite new OGC SOS 2.0 specification and exchangeable output encodings (e.g. CSV, O&M, WaterML2.0 and maybe JSON). Additionally we will discuss and demonstrate ways to include SOS data services in a client application.

A lot of work, but worth the effort. I am looking forward to create something new and useful. Besides the apparent advantages for New Zealand’s researchers, also consultancies, commercial organisations and governmental agencies will profit from such a dynamic, standards-driven and web-based geospatial data delivery service that could serve as another good example for environmental data providers.

The benefits of the project include well managed freshwater use & potential impacts of climate change are critical for any country to plan & manage its natural resources. One of the foundations of any effort in these domains is ready access to climate (including rainfall) information.

NIWA is collaborating with a wide range of New Zealand and international agencies to determine strategies and appropriate standards to provide interoperable discovery and delivery facilities for data managed by NIWA, as well as to develop systems to implement those strategies. These include: CSIRO, BOM (Bureau of Meteorology), AODC (Australian Ocean Data Centre), IMOS(Integrated Marine Observing System) in Australia. Iquest and Kisters (New Zealand and internationally), 52o North.

Furthermore collaboration with GNS Science in the NZ-EU cooperative groundwater project SMART (www.smart-project.info, http://www.gns.cri.nz/) is intended, to support those who seek to incorporate climate data for their research towards characterizing New Zealand’s aquifers.

Starting to play with spatial-temporal data

Thinking  the big picture is a different thing to actually implement it :-) Well, who doesn't know that. Having some hydrological time-series (groundwater levels) in place in a sensor observation service (SOS), the hydrogeological all-in-one-wonder-portal is going to get a glance of the next level :-p
The last weeks I started to play around with the R environment for statistical computing and visualization (The R Project). 52°North developed a neat R toolkit to access and digest SOS time-series - sos4R.

It is well documented and pretty easy to connect to a SOS server, and query observations. So for the fun of it and to demonstrate the general feasibilty, I quickly queried the groundwater levels of the Horowhenua area in New Zealand, where I got some sample data (courtesy by the regional council).

With the R sos4R, fields and akima packages from the CRAN R packages archive I (quite coarsely) interpolated the groundwater surfaces for the years 1991-2009 and put the images together as an animated gif (meters above mean sea level over time).

Discussion

I am aware of the total uselessness of this particualr way presenting :-) No years, the scale changes slightly, and the exact spatial extent and north orientiation are not reliable :-p

Nevertheless, for just playing around, this was a motivating simple first shot to easily visualise changes over time.

I would like to play around with the gstat and spacetime R package, integrate a more sophisticated script as a 52°North WPS process and have those things happening automagically in the interwebz.

GSoC and Master Thesis completed

Busy summer, well or winter, depends on the hemisphere and the full time extent. Last 2-3 months have been packed with work and included about 25.000 air miles (which means concatenated spending several days in airplanes).
I finally managed to finish my Master Thesis "Towards a 4D WebGIS using harmonised datasets: Examined on a New Zealand Example" which examines in the first part existing groundwater-related geoportal projects and scientific applications around the globe, existing and useful, web-based technologies and standards (e.g. OGC) to implement such a portal, a draft architecture. In the second part actual prototypical implementations of a selected subset of the examined standards and and technologies demonstrate their feasibility for the SMART project. I also hold a talk at the GI_Forum conference in Salzburg (proceedings/conference paper).
Furthermore I successfully completed my Google Summer of Code (GSoC) project with the 52°North Initiative, where I implemented an exchangeable encodings mechanism for their Sensor Observation Service (SOS). I additionally implemented a plugin that provides SOS time-series output in the WaterML2.0 format. I could acquire the latest schema, which will be published soon.. we were supposed to write a series of blog articles to document or progress in the project:

• Dynamic output formats for the Sensor Observation Service
• Exchangeable Encodings getting in shape
• Demonstrating Exchangeable Encodings: A Quantum Leap for me

After flying back to New Zealand I was lucky to visit the GeoSciML Face-to-Face-Meeting at GNS Science in Wellington. It was really exciting to meet these people, working on the international standard of a Geoscience Markup Language. also issues around the OneGeology portal have been disussed, as it uses GeoSciML to build up a world geology map. Additionally I could get in touch with one of the inspiring driving persons behind the Canadian Groundwater Information Network and their its sematic foundation, the Groundwater Markup Language (GWML), which itself is derived from GeoSciML.
Now I will dedicate my research to the development of a New Zealand groundwater geoportal within the SMART project, using OGC webservices, GeoSciML and/or GWML, WaterML2.0 and X3D as its foundations to transform and consume all available data relevant to aquifer characterisation and create a completely new, visually appealing 3D/4D view on New Zealand's groundwater resources.

Exciting ;-)

Demonstrating Exchangeable Encodings for SOS - A Quantum Leap

The final sprint towards the end of the Google Summer of Code (GSoC) is over now. After fixing bugs, working on documentation and actually producing time-series in different formats, the “Exchangeable Encodings for SOS”
project met most of its ambitious goals – and it is demonstrated on the 52°North Google summer of Code Demo Server. Users and developers documenation on the encoding mechanism has also found a place in the 52°North wiki.

We have integrated a plugin mechanism into an SOS server (source code branch), which is based on the main 52°North SOS development line (source code trunk).

Link to the article on the 52°North blog

Exchangeable Encodings Getting in Shape

The time is flying by and soon we reach midway of the Google Summer of Code (GSoC) with 52°North. It is the typical time in between – I already managed to meet the first milestones, but there is no proper product yet. Catching up with the changes from the SOS main development branch is a constant challenge, but the infrastructural changes within the “Exchangeable Encodings for SOS” project are slowly but continuously leading towards a robust and standardized, yet limited, encoding plugin API (application programming interface).

The main use case is the request for observations (GetObservation request) that could be delivered in CSV (comma separated values) format for legacy applications, WaterML2.0, an upcoming OGC (Open Geospatial Consortium)
data transfer standard for the hydrological domain, possibly netCDF (Network Common Data Form), also an OGC standard for meteorological applications, and in O&M (Observations & Measurements) for the generic use within sensor networks and spatial data infrastructures.

Link to the article on the 52°North blog

Dynamic Output Formats for the Sensor Observation Service

As one of the 52°North Google Summer of Code projects, “Exchangeable Encodings for the 52°North Sensor Observation Service (SOS)” aims to implement a customisable result set encoding mechanism. The users would then be able to add their own compiled libraries (like plugins), which contain all the necessary code to provide the requested data in different formats, such as CSV or GeoJSON, to the SOS. When that mechanism works, we can easily add more encodings, e.g. the planned WaterML2.0 – but for now, where to start?

The internal architecture of the 52°North SOS server is generally well structured. Based on the user request, the respective Listener will get the data from the data backend and send it to the corresponding Encoder. The response document, very likely a collection of observations, is then returned to the user in the requested encoding, also referred to as the response format. The typical application flow of such a request is depicted in the comic strip below. Available response formats can be queried from SOS server and are listed in the capabilities document. To date the two integrated encoders return XML documents based on the OGC© O&M schema and the OGC© SOS versions.

Link to the article on the 52°North blog

The Devil is in the Details

As one of the 52°North Google Summer of Code projects, “Exchangeable Encodings for SOS” aims to implement a customisable result set encoding mechanism.   If that mechanism works,  users could add an own pre-compiled .jar-file to the SOS that contains all the necessary code to encode the requested data in simpler formats like CSV or GeoJSON – but for now, where to start?

The architecture of the SOS server is generally well structured. Based on the request – GetCapabilities, GetObservation – and given parameters, like Offering, FeatureOfInterest and/or time the RequestOperator calls the respective Listener. The Listener calls the DAO, which fills an internal response object and gives it back to the Listener. The response object, most likely a SosObservationCollection is then sent to the respective encoding module actually based on the SOS version from the request. Then the Listener decides, which Encoder will be used - to date the decision is mainly based on the requested SOS protocol version. The Encoder now creates an XML object based on the OGC O&M schema and gives it back to the Listener. Finally the Listener puts the textual representation of that XML object into the response document that will be returned via http. This is good, because other encodings are likely to be plain/text, too.

There are different entry points to dynamacally add encodings, either as additional full-fledged SOS server modules or as another responseFormat internally be evaluated and the respective encoder selected through the SosConfigurator instance. But following constraints are to be regarded:

• The SOS protocol versions 1.0 and 2.0 must be honoured
• responseFormat O&M 2.0 ships as standard
• Listener and Encoder are coupled through the SOS protocol version, the response encoding needs to be decoupled and dynamically selectable
• Other formats / encoders need to be “dynamically” registered in the SoSConfigurator and to be identified and allocated based on the requested ResponseFormat value from the request

Simple sequence diagram of a GetObservation request to the SOS server

Finally one of the challenges I suppose, is the generation of a non-XML-response in this XML dominated workflow. As the SOS internal response object is merely of the String datatype, a simple encoder would need to implement the ISosResponse for the return value and should accept the SOS internal data collection representation as delivered from the DAO backend. Here in between we will put the lever to enhance the versatility of the 52 North SOS server.

52North SOS Server for NGMP

52North SOS Server for NGMP

As I recently wrote about the three major work packages (Boiling it down), I mentoined the publishing of groundwater measurements from a legacy DB. Actually it is the GGW database (GNS Science Geothermal and Groundwater Database) which besides a lot of other stuff (geothermal data for Geothermal and volcanic monitoring, GeoNET), holds the data of the National Groundwater Monitoring Programme (NGMP). This is a governmental sponsored, freely accessible data set that holds groundwater measurements and analysis results (mostly quarterly or half-year basis). But this data can only be accessed via a web-frontend, where registered users can download pdf or excel reports including the meausred values over time.

My ambitious first pillar of the SMART data portal prototype is to consume and display data from the NGMP. Therefore I need to access the NGMP data (technically the GGW DB). And as we want to demonstrate useful SOS data usage, I need to put a SOS server on top of it!

In search for a proper way to serve NGMP data via SOS, I came across two major Open Source players, the 52North initiative, the ecosystem around OOSThethys and istSOS, from the Geomatics department is part of the Earth Science Institute (IST) at the University of Applied Sciences of the South Switzerlad (SUPSI).

http://52north.org/communities/sensorweb/sos/
https://wiki.52north.org/bin/view/Sensornet/SensorObservationService

http://www.oostethys.org/
http://code.google.com/p/oostethys/

http://istgeo.ist.supsi.ch/software/istsos/
https://sites.google.com/site/istssosproject/

The 52North SOS server is written in Java and has a roburst architecture, which especially separates the data access from the main procesing modules, a so called Data Access Object design pattern. This seems very handy, as I needed to access a completely different database and data model. The OOSTethys project has several different packages in different programming languages with different levels of sophistication and the istSOS server is written python.

[todo: reasons for decision should be more documented :-) ]

I decided for the 52North software, it has a vibrant community that strives for a bigger integration pattern. They  are providing a broad OGC eb services solution portfolio and often tend have recent standards already (at least prototypically or in beta status) implemented. OGC recently approved SOS 2.0 specification and 52North already has began work to incorporate that (52North Blog).

"52n-sos-dao-oracle"

As the NGMP / GGW database unfortunately has no spatial extensions, the planned custom backend needs to be able to abstract / fake this functionality in the software itself, instead of using typical spatial DB extensions (e.g. Postgis geometry datatype and geometric functions, BBOX search, etc).

52North SOS data model (Source: 52North Wiki)

I started by copying the existing "52n-sos-dao-postgis" to "52n-sos-dao-oracle", included the ojdbc14.jar driver and adjusted the dssos.config and main pom.xml to build the new DAO, load the oracle db driver and connect to the oracle db.
That was "almost" easy, the server builds, connects to the db and dies, as the queries from the original Postgis DAO obviously don't work.

Now the heavy-duty data part, actually the really interesting part, begins...

As the GGW / NGMP data model ist based on the Australian National Groundwater Data Transfer Standard from July 1999 and has been modified to suite the demands of GNS Science's geothermal and groundwater data collection, the schema is huge and the model quite complex. After consultion with the DBAs on the side and the scientists on the other side, I got a feeling for the whole thing and started to find things via joining the right views and tables to respect necessary constraints.

Australian National Groundwater Data Transfer Standard (Release 1.0 July 1999)

Then I began to adjust the queries in the ConfigDAO and the DAOFactory to select the proper fields to make the server initialisation work. But to keep it simple for the beginning I concentrated on only one property out of several dozens with available data - water level.

The next big challenge is coming - geometry. The coordinates are in two columns related to the groundwater The spatial reference system is well-known (EPSG 27200, NZGD49 / New Zealand Map Grid) and the coordinates are in two columns. So I intend to use Geotools to build point features for the "groundwater features" that are supposed to be wells, spings and bores and make them availabe throughout the SOS internal data structure, just as the other (frequently updated items in CapabiltiesCache) list.

Good Luck :-)

Designing the big picture

Designing the big picture

After literature review and further research and gathering a lot of information about state-of-the-art technology and paradigms, I designed a bigger concept of a possible, full-fledged WebGIS that is more than capable to support the data-related and computational needs of the research of the SMART aquifer characterisation program. after writing it down, we already got a paper to submit to the GI_Forum Geoinformatics conference.

.

SMART Portal archtitecture concept

As shown in the picture, the whole portal / WebGIS is based on standardised interfaces, mainly the OGC geo-spatial web services. as a requirement from the New Zealand Ministry of Science and Innovation all software produced (and used) should be Open Source, the main building blocks for the portal will be tested and evaluated in the prototype, I am going to develop:

• PostGres/PostGis - data storage, spatial database
• Geoserver - for WMS and WFS (subsequently WCS, too)
• OpenLayers as web mapping client
• 52North SOS Server, customise DAO backend to serve data from an existing legacy DB
• Thredds server to serve netCDF coverage data
• X3D as 3D file format to display 3D geological model in the browser
• Geonetwork - CSW, metadata, to connect to upstream NZ spatial data infrastructure

Exchangeable Encodings for SOS - extension support WaterML2

The 52°North SOS implements only a subset of the O&M data format, because O&M is too flexible to be supported completely. It would be better if users could just implement a simple encoding (for example to support their own CSV or JSON data format) and upload a compiled jar file to the SOS with a browser-based user interface. Then a client selects the newly available response format to retrieve the data in a new format. As an extension it would be very useful to researchers and data maintainers in the hydrological and meteorological domain to have the SOS server deliver WaterML2.0 encoded time-series. Based on the SOS adminstrator project, the desired encodings could be prepared as .jar-files and added via a web frontend.

Link to the 2012 GSoC proposal

Hello World

"Hello World"

Obligatory, awkward first post :-)

I am going to post infos about stuff I am actually doing. That mainly includes my UNIGIS studies and the Master Thesis in particular. I am officially matriculated at the University of Salzburg, Austria. I am contributing to their research in the SMART project as a working student in New Zealand. My main interests and recently my main activities lie in the convergence zone of web technologies and GIS. Pretty cool stuff ^^

Cheers,
Alex