Hydrological Society Conference - Web-based real-time processing of environmental measurements

This case study was presented as poster abstract.

Kmoch, A., White, P. A., & Klug, H. (2015). Sensor Observation Service and web-based real-time Processing of environmental Measurements in the Upper Rangitaiki Catchment (Poster). In The NZ Hydrological Society Conference 2015, 26th November, in Hamilton, New Zealand

Abstract:

Environmental assessments naturally depend on field observations and technological advancements. , such as tTelemetry, allow the automated collection, transmission and processing of these measurements. However, modelling of natural processes is typically a complex challenge and involves applying expertise of scientists as well as a host of data preparation steps (White, 2006, White et al., 2003).
In addition, automation of model execution with the most recent observation data is dependent on the integration of the data collection, storage and processing elements (Klug and Kmoch, 2014). This paper demonstrates a system that integrates a Sensor Observation Service (SOS) that includinges field observations and internet-based environmental data with a rainfall recharge model that allows near-real time calculation of rainfall recharge in the Upper Rangitaiki catchment, Bay of Plenty region.
The SOS specification is an Open Geospatial Consortium (OGC) standard for the open and standardised integration of environmental sensors into an internet-based environmental data infrastructure (Klug and Kmoch, 2015, Klug, Kmoch, Reichel, 2015).

Figure 1. Process of data flow from field site sensors, to SOS data service to a simulation model process

Results:

• We showed that it is possible to link the collected data directly to a simple rainfall recharge model (Figure 1)
• The low cost sensor and circuit board instrumentation collects data and forwards them to the field computer in 10-minute intervals via robust, low power, ZigBee wireless protocol
• The field computer running a standard Linux operating system, transfers observation data in 10 minute intervals via a 3G mobile data connection to an online SOS server.
• From the service the observations are available in a standardised open format.
• A website can access the raw data from the SOS server and plotted data points within 5-10 minutes of field measurement
• A rainfall recharge model runs with the latest data points from the online SOS server.

References:

Klug, H., & Kmoch, A. (2014). A SMART groundwater portal: An OGC web services orchestration framework for hydrology to improve data access and visualisation in New Zealand. Computers & Geosciences, 69(0), 78–86. http://dx.doi.org/10.1016/j.cageo.2014.04.016

Klug, H., Kmoch, A. (2015). Operationalizing environmental indicators for real time multi-purpose decision making and action support. Ecological Modelling, 295, 66-74. http://dx.doi.org/10.1016/j.ecolmodel.2014.04.009.

White, P. A. (2006). Some Future Directions in Hydrology. Journal of Hydrology (NZ), 45(2), 63–68.

White, P. A., Hong, Y.-S., Murray, D. L., Scott, D. M., & Thorpe, H. R. (2003). Evaluation of regional models of rainfall recharge to groundwater by comparison with lysimeter measurements, Canterbury, New Zealand. Journal of Hydrology (NZ), 42(1), 39–64.

Klug, H., Kmoch, A., & Reichel, S. (2015). Adjusting the Frequency of Automated Phosphorus Measurements to Environmental Conditions. GI_Forum 2015 - Journal for Geographic Information Science - Geospatial Minds for Society, 1, 590–599. http://doi.org/10.1553/giscience2015s590

AGILE 2016 - SensorWeb Semantics on MQTT for responsive Rainfall Recharge Modelling

Integrating Wireless Sensor Networks (WSNs) and spatial data web services is becoming common in ecological applications. However, WSNs were developed in application domains with different sensor and user types, and often with their own low-level metadata semantics, data format and communication protocols. The sensor web enablement initiative (SWE) within the Open Geospatial Consortium (OGC) has released a set of open standards for interoperable interface specifications and (meta) data encodings for the real time integration of sensors and sensor networks into a web services architecture.
Such XML-based web services exhibit disadvantages in terms of payload and connectivity in low-bandwidth low energy unreliable networks, such as remote 3G uplinks. Monitoring stations deliver frequent measurements in real-time, but dynamic implementation of measurement frequencies, adapted to certain environmental conditions, are rarely implemented. We describe a responsive integrated hydrological monitoring prototype to calculate rainfall recharge for water management purposes.
When rainfall is observed, a threshold event triggers a reconfiguration task for the soil moisture sensors, using asynchronous, push-based communication implemented with an MQTT queue. A Sensor Planning Service commits that request via MQTT into the wireless sensor network, and updates the measurement frequency of the target sensors to gain higher resolution for the vertical soil water infiltration.
The system integrates a Sensor Observation Service (SOS) including field observations and internet-based environmental data with a rainfall recharge model that allows near-real time calculation of rainfall recharge in the Upper Rangitaiki catchment, Bay of Plenty region in New Zealand.

Figure 1: Setup and location of the sensor field site, central North Island, New Zealand

The prototype site comprises a main station conducting comprehensive measurements of meteorological, hydrological and pedological parameters. For the wireless data transmission within the local site installation XBee-PRO modules from the Digi Company  ZigBee IEEE 802.15.4 protocol are used. The main station receives continuous sensor measurements from the attached sensor units, and acts as the gateway to the online SOS and SPS services by providing the communication channel from the local sensor network to the web-enabled data management infrastructure.

The field site has been established in the Upper Rangitaiki catchment (Figure 1) and comprises a field computer (Raspberry Pi) with a direct internet link (GPRS/3G) and a sensor board (Waspmote) that has 12 typical meteorological, hydrological and pedological sensors attached (i.e., wind speed, wind direction, rainfall, 1x groundwater probe, 5x temperature and 3x soil moisture). The Raspberry Pi and Waspmote can be monitored and reprogrammed from an online server.

Figure 2: Raw sensor series visualized in a website from a SOS query.

The site setup allows scaling up to a multitude of low cost, low energy sensor stations throughout the catchment, with only one field computer that serves as data logger for backup. The observations were available in a standardized open format. The website accessed the raw data from the SOS server and plotted data points within 5-10 minutes of field measurement. This website was easily accessible via browsers and smartphones (Figure 2).

The paper was was presented at the 19th AGILE International Conference on Geographic Information Science, 15th of June, in  Helsinki, Finlkand.

Kmoch, A., Klug, H., White, P., & Reichel, S. (2016). SensorWeb Semantics on MQTT for responsive Rainfall Recharge Modelling. In 19th AGILE International Conference on Geographic Information Science. Helsinki.

Amazon IoT and OGC SensorThings vs SOS/SPS on MQTT

Today I stumbled over a blog post on the OGC Blog about "Amazon IoT and the candidate OGC SensorThings API Standard"

It was great to learn that Amazon IoT uses MQTT as network / transport layer. The more interesting is the move towards REST (which is HTTP) to address semantic issues with IoT systems:

AWS IoT, MQTT and many other network interoperability standards (LWM2M, CoAP, etc.) enable message interchange.  However, IoT network interoperability doesn’t enable the systems that are exchanging messages to interpret those messages. To realize the many-to-many system-of-systems vision, IoT applications need to implement standard ways of communicating sensor locations, sensor and data parameters, and sensor instruction sets. This is what the OGC SensorThings API provides.

It is understandable from the point of view, that OGC OWS is in general intrinsically interwoven with HTTP semantics and the application of SWE in the IoT context is desirable, well, even crucial.

However, MQTT stems from the constraints of low bandwidth, low energy consumption and unreliable connectivity, and implements a robust, small footprint asynchronous messaging platform. Earlier this we have developed a prototype system, that aims to add SWE semantics into MQTT messaging. We keep the specialisation of MQTT, where for example connections can be resumed which under SSL/TLS security can be a significant power saver. MQTT would also better address the "addressability" of distributed "things" which might be isolated behind IP NAT or run with dynamically assigned addresses, because in MQTT context the "things" would subscribe to SPS management topics and therefore higher order SDI services don't need to direct data queries or task requests to an address but place it on the queue with respective topics.

For an agricultural research project we distributed monitoring stations as a wireless sensor network throughout the catchment to deliver frequent measurements in near-realtime, which mimics OGC SOS.

A dynamic implementation of measurement frequencies, adapted to certain environmental conditions like heavy rainfall events require re-tasking of the nodes, which mimics OGC SPS. Within this paper we provide a framework where a threshold event triggers a reconfiguration task for a phosphorus measurement device, using asynchronous, push-based communication, on an MQTT queue, which links the ground stations with a cloud-based control system. OGC WPS algorithms continuously analyse incoming SOS measurements commits such a request into the wireless sensor network, and updates the measurement frequency of the target nodes to enable nutrient peak flow estimation during storm events.

This on-going work is based on original thoughts on the  interlinking  of  OGC  SWE  semantics  for  sensor  descriptions  and  observations (OGC Sensor Observation Service, SOS), sensor configuration and planning (OGC  Sensor Planning Service, SPS)  with  the  open  Message  Queue  Telemetry  Transport  (MQTT)  protocol.

MQTT  is  a  simple,  yet  very performing  and  robust  publish/subscribe  message  passing  system,  which  can  also  serve  for  event  brokering as  described  in  the  SWE  Service  Model.  Actual  addressing  and  data  transmission  is  done  by  a  topic  string and an arbitrary payload. The topic string and the payload need to reflect the necessary standard semantics to be  mapped  back  and  forth  to  allow  for  a  seamless,  and  preferably  lossless  bi-directional  multi-lateral communication  between  the  WSN  as  data  provider  (SOS),  web  clients  as  data  consumers  (SOS)  and  the management system, which interacts with the WSN in a standardised way, too (SOS and SPS). 

Digital Earth Conference 2015, Canada

"Towards a One-World Vision for the Blue Planet" was the slogan for the 9th Symposium of the International Society for Digital Earth (ISDE) from 5th - 9th of October, 2015, in Halifax, Nova Scotia, Canada.

At Digital Earth 2015 scientists, engineers, technologists, and environmental managers from around the world will meet to share concepts, research findings, technologies, and practical applications relating to the Digital Earth vision. *()

I presented an experimental approach towards integrating scientific legacy codes into the OGC web services framework, on the example of exposing USGS MODFLOW through a vanilla and open source WPS (without using proprietary tools like ESRI Arc Server and Arc Hydro Tools etc).

I was very honoured and greatly appreciated the "Best Student Poster" award. It seems I really had the right audience at ISDE.

Furthermore it was extremely informative, and in the session about "Discrete Global Grid Systems" I learned a lot about this new approach of unifying traditional raster/coverage data with an equal area per pixel advantage ... one of the great shortfalls of todays popular Web Mercator projections. In fact spinning this further it could be a  great new approach and how groundwater, geology, ocean modelling and atmospheric sciences, which would benefit from an equal volume grid, particularly when re-mashing resources from different environmental/scientific/governmental/industrial domains.

There is even an DGGS OGC Standards Working Group on the way.

ZOO-Project WPS Java-API and JGrasstools Java Hydrological Toolbox

The ZOO-Project (http://zoo-project.org/) is a solid Open Geospatial Consortium (OGC) Web Processing Service (WPS - http://www.opengeospatial.org/standards/wps) standard server implementation with an open flexible API that works well with many different programming languages. The Java bindings have never been tested in advanced configurations and complex data types, and to date only implement the minimum necessary interfaces. The JGrasstools project is a modular processing library and its highly annotated nature makes it possible to adapt quite easily to other toolboxes. JGrasstools contains a wide variety of powerful and efficient GIS, hydrology and geomorphological tools and processes, that can be exposed to and used by other libraries and toolkits. One example has been the adaptation to the Geotools Process API. The JGrasstools project, as well as other java based projects (as JTS, Sextante or even Geotools) would benefit greatly from the possibility to be used within a web-enabled WPS execution environment, as well as being integrated with the open standards suite of the OGC.  Some time ago Moovida tried integrating the JGrasstools libraries with the ZOO-Project Java binding to expose them as native WPS processes. This would allow them to work inside the ZOO-Project and serve its modules under the WPS standard.

Some Background

The ZOO-Project WPS implementation is a flexible, modular high performance HTTP CGI implementation. ZOO-Kernel is a powerful server-side C Kernel which makes it possible to manage and chain Web services, by loading dynamic libraries and handling them as on-demand Web services. The ZOO Kernel is written in C language, and supports several common programming languages in order to connect to numerous libraries and models (http://zoo-project.org/trac/wiki/ZooWebSite/ZooKernel). The generic ZOO API is basically accessible for every possible programming and web scripting language that can be run under the CGI interface. Main API implementations, the ZOO services, are available for C/C++, Python, JavaScript, PHP, Fortran and Java. Some API bindings are more advanced and complete and make the full ZOO-API (http://zoo-project.org/trac/wiki/ZooWebSite/ZOOAPI/Classes#ZOOAPIClasses) accessible to the ZOO service in the particular programming language (e.g. C, Python or JavaScript). In comparison the the Java API binding only exposes the minimum functionality to run from the ZOO Kernel.

JGrasstools (http://moovida.github.io/jgrasstools/) is a powerful GIS toolkits with functionality reaching from standard geoprocessing algorithms to advanced processing features used in hydrology and geomorphology. JGrasstools is based on a Maven (http://maven.apache.org/) build process, which takes care of dependency resolution and creating the succinct jar packages with the compiled classes. Maven is a defacto standard for managing (source and dependencies) building and deploying (jar packaging, resources, copying, publishing, archiving and installing) Java-based software projects. JGrasstools is also used as a toolbox in the uDig desktop GIS software (http://udig.refractions.net/). If JGrasstools could be exposed via a open standards and interfaces, web-based processing and execution environment (like ZOO-Project provides) it can be widely used in WebGIS deployments and large scale cloud based processing chains.

The next level

Andrea from Moovida said he didn't have enough time to continue developing this idea. He developed a generator which would programmatically scan through the annotated JGrasstools modules and generate respective ZooJavaWps classes per JGrasstools module/method and the corresponding ZOO-Project .zcfg config file. The only struggle I had was getting the CLASSPATH properly set up, as the ZOO-Project is basically an HHTP CGI application which will start a JVM per request. When I picked up on this in preparation for a GSoC proposal, I found that there were a few little botches with the parameter mapping from ZOO Java API into the very nicely annotated JGrasstools methods. So I took one example generated (WPS-ified) JGrasstool process and adjusted the parameter mapping and got it running with ZOO-Project. Additionally I adjusted the JGrasstools Maven config files to download the necessary dependencies in the target folders to copy them collectively in the ZOO-Project Java CLASSPATH.

Unfortunately I also didn't have time to drive this further still. However, it is just soooo close really :-) Alternatively a 52North WPS implementation based on the super practical JGrasstools annotations and Andrea's generator would also be relatively straightforward.

Web-based 3D Data Visualisation for Hydrogeology

When I was in Salzburg/Austria last year at the GI_Forum 2014 conference, I had the chance to present some of my recent experiments with the mapping of OGC interoperable geo-data to X3D interoperable open web 3D scenes and visualisation in the browser. Back in New Zealand later that year I had to present this work to the New Zealand Hydrological Society, too, of course :-)

Characterisation of a hydrogeological setting is a multi-faceted complex task. The assessment of usefulness and quality of relevant data is a major challenge. Statistical analysis and visual exploration of the datasets demand practical support by computer applications. Although a variety of software for this purpose is freely available nowadays, they require a good understanding of the technology or programming language for application in complex hydrogeological settings. Thus, integrated proprietary software products are often used to analyse and particularly provide high-quality visualisation of the system. However, these software tools are typically desktop programs with a strict licensing scheme and a limited extensibility and lack of interoperability with other applications.
We present an open and free to use web-based (platform independent) framework to enable retrieval, exploration and visualisation of hydro-climate time series data as well as three-dimensional geological information via a web browser. How distributed data and processing services can be linked to prepare an on-demand 3D visualisation of geological and hydrological data is demonstrated. A flexible toolbox design enables extensibility via open standards.

The method developed is applied to a case study area presented (s. figure), which is the Horowhenua district in the Manawatu-Wanganui region. Available datasets of 3D geology, hydrology and hydrogeology are combined and serve as example data for demonstrating the framework.

The slides of the presentations are now made accessible here (click here).

The full ISI-indexed conference paper is available here (click here).

A side note on geospatial data sharing and spatial data infrastructures (SDI)

This blog post is dedicated to provide a general overview over the field of geospatial and environmental data sharing. The term geospatial is actually tautologous: The prefix “geo” implies geography, which always relates things to each other based on their location, where nearer things are stronger related than things further away from each other (Tobler’s 1st law of geography). And the word “spatial” also means having an extent and location in a space. However “geospatial” nowadays is almost exclusively used in the field of digital data with geographical context. Therefore software that delivers, analyses, presents, processes, stores and retrieves is also often called geospatial software (having its origins in the good old GIS – Geographical Information Systems).

Part 1: GIS and GI Science intro

Research areas that work on the science behind GIS and spatial data, on the analytical methods, processes techniques, on ways of (standardised) spatial data exchange, effective and efficient storage and retrieval, are called GI Science, Geoinformatics, Geomatics, Geocomputation, Spatial Computation or Spatial Science … There is actually quite some discussion, if the necessity, even an entitlement for such a dedicated mixed branch of computer science and geography exists, a similar discussion when geography emerged as an accepted field of research (ref). On the other hand you’ll often find the quote that “80% of data has a spatial component” or something like that. Apparently I can get the source right anymore, therefore you might handle this with care. It might go back to the 1990s when GIS software for PCs wanted to get their feet into the market (gis lounge). However a lot of data in the geosciences have a spatial context and - a lot - of those data are needed for governmental agencies to manage land and water resources properly :smile: Agreed?

Part 2: The Opengeospatial Consortium, aka OpenGIS, aka OGC

“The Open Geospatial Consortium (OGC) is an international industry consortium of 479 companies, government agencies and universities participating in a consensus process to develop publicly available interface standards. OGC® Standards support interoperable solutions that ‘geo-enable’ the Web, wireless and location-based services and mainstream IT.”

The OGC standards framework provides means to build a spatial Data infrastructure (LINZ - Land Information New Zealand) – which is “the technology, policies, standards, and human resources necessary to acquire, process, store, distribute and improve the usability of geospatial data. (SDI) facilitates the connections between these important sources of information, and allows people to find and access them.” Quite some of the OGC standards and web services are also ISO international standards. There are interface and service descriptions on the one hand and data encodings/formats and conceptual data models on the other side. I will provide a short summary:

WMS – Web Mapping Service (ISO 19128 WMS v1.3.0)

Essentially provides (web) maps (as images like png or jpg) output data Geographiclly correct images, png, jpg, view or portrayal service

Major methods: GetCapabilties and GetMap

WFS – Web Feature Service (ISO 19142 WFS v2.0)

Provides an interface to access, query, store and retrieve vector “features”, aka discrete data – like in ESRI shapefiles. Data is accessible by their data schema, which can be soft-typed and values in schema fields and location queries are utilized. Output GML (ISO 19136), which is in a particular “domain-specific” XML schema. With WFS-T – transactional – there is also support to write back to the WFS server.

GetCapabilities GetFeature – get the data DescribeFeatureType – get schema

WCS – Web Coverage Service

Provides an interface to access, query and retrieve raster imagery and coverages, grids (aka “fields”) as in continuous data. eg NetCDF-CF, GeoTIFF, ArcGRID

GetCapabilities DescribeCoverage GetCoverage

CSW – Catalogue Service for Web (ISO 19115 CSW 2.0.2)

Provides an interface to access, query, store and retrieve metadata, aka data about the geospatial data, which is accessible through other geospatial webservices. Output is usually XML ISO 19139 metadata or Dublin core.

GetCapabilities GetRecords – find metadata record by search criteria GetRecordById – get one record by its unique id DescribeRecord – metadata type GetDomain – get range of values and/or keywords

SOS – Sensor Observation Service

Provides an interface to access, query, store and retrieve time-series based data that has been measured at locations, eg through sensors or field surveying/sampling. But the focus is to query on temporal and then on spatial or value comparison basis. Standard output formats are O&M, WaterML2.0 time-series and SensorML sensor/procedure metadata.

SOS is part of the sensor web enablement initiative (SWE) which advances to its version 2, where a lot of things become more flexible, but also moe complicated. I will write about that later ☺ SOS explicitly also describes a group of methods to insert sensor and observation data. This is handled through different profiles.

GetCapabilities GetObservation DescribeSensor

There are quite some commercial and Open Source software packages and tool kits available, for the desktop and server-based for the web that support or where explicitly written for OGC webservices and splendid Open Source resources in the web:

OSGeo Foundation: http://www.osgeo.org

(Spatially enabled) Databases:

• Postgresql/Postgis
• MySQL
• Oracle Database
• Microsoft SQL Server
• ESRI Geodatabase / ArcSDE
• SpatiaLite
• Rasdaman
• GeoCouch

Data Servers (store data in databases):

• Geoserver (WMS, WFS, WCS)
• Mapserver (WMS, WFS, SOS)
• 52°North SOS server
• Geonetwork (CSW)
• ESRI ArcIMS / ArcServer
• Thredds (WCS, netCDF)

Web mapping tool kits / frameworks (take data from data servers):

• openlayers
• Mapbender
• MapFish
• Geomajas
• Flash
• Silverlight

Desktop clients supporting (at least partially) OGC standards:

• QuantumGIS
• uDig
• ESRI ArcGIS
• Intergraph Geomedia

And on it goes … in the next weeks, I will write about New Zealand and international examples of OGC webservices implementations and SDIs for geospatial data publishing (where the data is basically public domain and needs to made accessible) and provide some closer insights to OGC SWE and the next generation sensor networks initiative SWE 2.0 et al.