ZOO-Project WPS Java-API and JGrasstools Java Hydrological Toolbox

The ZOO-Project (http://zoo-project.org/) is a solid Open Geospatial Consortium (OGC) Web Processing Service (WPS - http://www.opengeospatial.org/standards/wps) standard server implementation with an open flexible API that works well with many different programming languages. The Java bindings have never been tested in advanced configurations and complex data types, and to date only implement the minimum necessary interfaces. The JGrasstools project is a modular processing library and its highly annotated nature makes it possible to adapt quite easily to other toolboxes. JGrasstools contains a wide variety of powerful and efficient GIS, hydrology and geomorphological tools and processes, that can be exposed to and used by other libraries and toolkits. One example has been the adaptation to the Geotools Process API. The JGrasstools project, as well as other java based projects (as JTS, Sextante or even Geotools) would benefit greatly from the possibility to be used within a web-enabled WPS execution environment, as well as being integrated with the open standards suite of the OGC.  Some time ago Moovida tried integrating the JGrasstools libraries with the ZOO-Project Java binding to expose them as native WPS processes. This would allow them to work inside the ZOO-Project and serve its modules under the WPS standard.

Some Background

The ZOO-Project WPS implementation is a flexible, modular high performance HTTP CGI implementation. ZOO-Kernel is a powerful server-side C Kernel which makes it possible to manage and chain Web services, by loading dynamic libraries and handling them as on-demand Web services. The ZOO Kernel is written in C language, and supports several common programming languages in order to connect to numerous libraries and models (http://zoo-project.org/trac/wiki/ZooWebSite/ZooKernel). The generic ZOO API is basically accessible for every possible programming and web scripting language that can be run under the CGI interface. Main API implementations, the ZOO services, are available for C/C++, Python, JavaScript, PHP, Fortran and Java. Some API bindings are more advanced and complete and make the full ZOO-API (http://zoo-project.org/trac/wiki/ZooWebSite/ZOOAPI/Classes#ZOOAPIClasses) accessible to the ZOO service in the particular programming language (e.g. C, Python or JavaScript). In comparison the the Java API binding only exposes the minimum functionality to run from the ZOO Kernel.

JGrasstools (http://moovida.github.io/jgrasstools/) is a powerful GIS toolkits with functionality reaching from standard geoprocessing algorithms to advanced processing features used in hydrology and geomorphology. JGrasstools is based on a Maven (http://maven.apache.org/) build process, which takes care of dependency resolution and creating the succinct jar packages with the compiled classes. Maven is a defacto standard for managing (source and dependencies) building and deploying (jar packaging, resources, copying, publishing, archiving and installing) Java-based software projects. JGrasstools is also used as a toolbox in the uDig desktop GIS software (http://udig.refractions.net/). If JGrasstools could be exposed via a open standards and interfaces, web-based processing and execution environment (like ZOO-Project provides) it can be widely used in WebGIS deployments and large scale cloud based processing chains.

The next level

Andrea from Moovida said he didn't have enough time to continue developing this idea. He developed a generator which would programmatically scan through the annotated JGrasstools modules and generate respective ZooJavaWps classes per JGrasstools module/method and the corresponding ZOO-Project .zcfg config file. The only struggle I had was getting the CLASSPATH properly set up, as the ZOO-Project is basically an HHTP CGI application which will start a JVM per request. When I picked up on this in preparation for a GSoC proposal, I found that there were a few little botches with the parameter mapping from ZOO Java API into the very nicely annotated JGrasstools methods. So I took one example generated (WPS-ified) JGrasstool process and adjusted the parameter mapping and got it running with ZOO-Project. Additionally I adjusted the JGrasstools Maven config files to download the necessary dependencies in the target folders to copy them collectively in the ZOO-Project Java CLASSPATH.

Unfortunately I also didn't have time to drive this further still. However, it is just soooo close really :-) Alternatively a 52North WPS implementation based on the super practical JGrasstools annotations and Andrea's generator would also be relatively straightforward.

EODP - An OGC-based Environmental Observation Data Profile

A group around Landcare, NIWA, GNS, HillTop Software and Horizons Regional council worked the last 2 years to develop an interoperable web-based data access and sharing standard. Working group formed as results from Hydrological Society Annual Conference Data Access Workshops. Subsequent meetings and workshops at Horizons regional Council office in Palmerston North even featured visits and input from Dr David Maidment as well as from delegates from BOM and Kisters, Australia. Further workshops were held at GNS in Wairakei and NIWA offices in Auckland and Wellington.

The NEMS-endorsed Environmental Observation Data Profile (EODP) is based on proven open standards of the Open Geospatial Consortium (OGC) and comprises a profile of WFS that allows the discovery of timeseries data associated with monitoring stations. This forms part of a larger project to enhance the interoperability of Environmental Observations throughout New Zealand.

Organisations such as NIWA, Landcare, GNS and Regional councils maintain a range of monitoring stations making environmental observations. These include Climate, Hydrology, Air Quality, Soils, Water Quality, and Marine data.

We wish to make this data available for research, analysis and reporting through open data protocols such as the Sensor Observation Service (SOS), Web Feature Service (WFS), Web Coverage Service (WCS) and Web Mapping Service (WMS). We also want to publish the existance of such data sets and services through online metadata catalogues (CSW).

In order to make use such data services users must first be able to discover their existence and then accurately select the subset of data that is relevant to their needs. The EOPD is focussed on this selection process in essence addressing the Where, What and When questions.

WFS provides a rich filtering capability, but typical WFS sources use a flat property/value schema that makes it difficult to express the many to one relationship of the timeseries measurements taken at a station. The EODP is an Application Schema for WFS that encodes the station metadata as a nesting of SF_SpatialSamplingFeature and OM_Observations using the language of Observations and Measurements to describe the characteristics of the timeseries.

The profile is extended by the use of an external Vocabulary service used to dereference measurement identifiers into familiar timeseries names such as rainfall or temperature.

Alistair Ritchie from Landcare Research did the main formal write-up and published the documentation on GitHub: https://github.com/EODP-NZ/eodp-dev

A implementation testbed is currently underway in joint collaboration with NIWA on the NGMP database at GNS and the Climate Database (CLIDB) at NIWA.

http://portal.smart-project.info/sos-smart/service?service=SOS&version=2.0.0&request=GetDataAvailability&observedProperty=http://vocab.smart-project.info/ngmp/phenomenon/1679

http://portal.smart-project.info/gs-smart/wfs?request=GetFeature&service=WFS&typename=sams:SF_SpatialSamplingFeature

Web-based 3D Data Visualisation for Hydrogeology

When I was in Salzburg/Austria last year at the GI_Forum 2014 conference, I had the chance to present some of my recent experiments with the mapping of OGC interoperable geo-data to X3D interoperable open web 3D scenes and visualisation in the browser. Back in New Zealand later that year I had to present this work to the New Zealand Hydrological Society, too, of course :-)

Characterisation of a hydrogeological setting is a multi-faceted complex task. The assessment of usefulness and quality of relevant data is a major challenge. Statistical analysis and visual exploration of the datasets demand practical support by computer applications. Although a variety of software for this purpose is freely available nowadays, they require a good understanding of the technology or programming language for application in complex hydrogeological settings. Thus, integrated proprietary software products are often used to analyse and particularly provide high-quality visualisation of the system. However, these software tools are typically desktop programs with a strict licensing scheme and a limited extensibility and lack of interoperability with other applications.
We present an open and free to use web-based (platform independent) framework to enable retrieval, exploration and visualisation of hydro-climate time series data as well as three-dimensional geological information via a web browser. How distributed data and processing services can be linked to prepare an on-demand 3D visualisation of geological and hydrological data is demonstrated. A flexible toolbox design enables extensibility via open standards.

The method developed is applied to a case study area presented (s. figure), which is the Horowhenua district in the Manawatu-Wanganui region. Available datasets of 3D geology, hydrology and hydrogeology are combined and serve as example data for demonstrating the framework.

The slides of the presentations are now made accessible here (click here).

The full ISI-indexed conference paper is available here (click here).

2014 NASA International Space Apps Challenge - TETRIS GroupSpace Apps Challenge 2014 - TETRIS Space Suit HUD demo with a Raspberry

As part of the 2014 NASA International Space Apps Challenge I joined an Auckland AUT University Textile Lab group to support their space suit concept idea with some cool wireless video and sensor streaming based on Raspberry Pi.

TETRIS (Terran Expeditionary Technologically Radical Instrumental Suit) is designing a suit, aimed for use by astronauts in space, which will make the astronaut’s work easier and safer to perform. It achieves this by embedding many of the new technologies being released in the recent years into the suit itself in order to: - monitor the astronaut’s life signs and condition via biometric sensors, such as a galvanic sensor for emotions, pulse sensor for the heart rate - allow simple integrated communication via a shoulder mounted camera and microphone - display biometric information to user via projected HUD - take simple electrical readings (when testing/diagnosing equipment) via embedded measurement tools (threaded conductive fabric in gloves) These hardware devices all interface with a RaspberryPi, being used as a central control unit in our design, as well the transceiver for all wireless communications between the astronaut and their remote operations control room.

Link to the 2014 Space Apps project website

The Raspberry Pi video camera live capture was streamed into the remote browser with an overlay of the HUD mockup.

Sources here https://github.com/TetrisGroup/spaceapps

IAH 2013, Perth, Australia

This presentation provides a great overview of my PhD research.
(presented at IAH 2013, Perth, Australia)

PLACE 2013 - Maori GIS Conference

I had the great opportunity to present my PhD work at the 2013 Maori GIS conference in Auckland. In order to provide a seamless spatial and multi-purpose view of collected groundwater related datasets, the SMART project joins forces to establish a valuable basis for groundwater analysis and decision support tools (Kmoch et al., 2012, Klug et al., 2011). One of the project’s objectives is to build a web-based data and knowledge portal and attached three-dimensional web visualisation tool according to OGC and ISO compliant standards (OGC, 2012).

Iwi/Māori have a well-recognised relationship with the natural environment which spans many centuries and is the result of interaction and adaptation with native flora and fauna of Aotearoa/New Zealand. Integral to this relationship is water which sustains life and is a taonga (treasure) with significant cultural and physical dimension. This is reflected through the on-going desire of many iwi/Māori groups to have a role in the way water is managed in New Zealand to ensure its sustainable utilisation moving forward (Kawharu, 2002). The development of scientific research tools and models that incorporate mātauranga Māori (Māori knowledge) and te reo Māori (the Māori language) are also beneficial to iwi/Māori resource policymakers, planners and decision makers.

GNS has been collecting and compiling Māori terms on hydrology, geology and geothermal phenomena to, amongst other things, explore the contribution that traditional indigenous knowledge can make to the research outcomes and model development by identifying the cultural significance of groundwater and the associated cultural links with surface water (Tipa and Tierney, 2003, Boast, 1991). The potential benefits of creating research tools that utilise te reo Maori and mātauranga Maori (within government institutional settings) includes generating increased uptake in and familiarity with te reo Māori and exposure of te reo Māori as a minority language to broader audiences. Further research could identify gaps in the dual knowledge systems (either the western scientific knowledge paradigm, or mātauranga Māori) that could be explored as an outcome of this combined research (De Bres, 2008).

To support te reo Māori and mātauranga Māori within the SMART portal web mapping and catalogue application, we evaluate a multi-language concept to incorporate semantic web methodologies to map and connect English and Māori terms and descriptions of presented natural phenomena as well as metadata and descriptive text within the application (Lutz et al., 2009). Beside a language template system for in-application-navigation use, a vocabulary web service is demonstrated to access content the thesauri, classification schemes, taxonomies, metadata and other types of controlled vocabulary and to document, link and merge concepts/terms to be with other spatial and non-spatial data (Antoine Isaac and Ed Summers, 2008). Further intentions include also a possibility to upload and geolocate orally passed on knowledge in the regard of ancient place information.

Evaluation of the lexicon’s effectiveness will be measured in part by its ability to be applied successfully to the SMART portal web mapping and cataloguing application.

References:

  ANTOINE ISAAC, V. U. A. & ED SUMMERS, L. O. C. 2008. SKOS Simple Knowledge Organization System Primer [Online]. Available: http://www.w3.org/TR/2008/WD-skos-primer-20080221/.
  BOAST, R. P. 1991. The legal framework for geothermal resources : a historical study : a report to the Waitangi Tribunal. Wai 153. Wellington: Waitangi Tribunal, 1991.
  DE BRES, J. 2008. Planning for tolerability: promoting positive attitudes and behaviours towards the Māori Language among non-Māori New Zealanders. Ph. D (Linguistics). Wellington: Victoria University of Wellington.
  KAWHARU, M. 2002. Whenua : managing our resources, Auckland: Reed, 2002.
  KLUG, H., DAUGHNEY, C., VERHAGEN, F., WESTERHOFF, R. & WARD, N. D. 2011. Freshwater resources management: Starting SMART characterization of New Zealand’s aquifers. Earthzine. http://www.earthzine.org/2011/12/13/freshwater-resources-management-starting-smart-characterization-of-new-zealands-aquifers/.
  KMOCH, A., KLUG, H. & WHITE, P. 2012. Freshwater resources management: first steps towards the characterisation of New Zealand's aquifers. GI_Forum 2012: Geovisualization, Society and Learning, 2012 2012 Salzburg. Car, A., Griesebner, G., Strobl, J., 376-385.
  LUTZ, M., J.SPRADO, E.KLIEN, C.SCHUBERT & CHRIST, I. 2009. Overcoming semantic heterogeneity in spatial data infrastructures. Computers &Geosciences, 35, 739-752.
  OGC 2012. OGC Standards. http://www.opengeospatial.org/standards: Open Geospatial Consortium.
  TIPA, G. & TIERNEY, L. 2003. A Cultural health index for streams and waterways : indicators for recognising and expressing Māori values. New Zealand. Ministry for the Environment.
  WHITE, P. A. 2006. Some Future Directions in Hydrology. Journal of Hydrology (NZ), 45, 63-68

Geospatial web-enablement for environmental data in New Zealand

This blog post can be seen as a sequel to a former blog post on the introduction on geospatial data sharing and spatial data infrastructures (SDI), where I explained the basics of OGC standards and web services. Quite some research organisations and governmental agencies already employ OGC standards to make data available online, often even free of charge for the public. I would like to present some really good examples of interoperable data sharing in New Zealand.

Through the standardised and web-based access to so many data sources, not only traditional geographical processing and analysis (GIS) based research is made easier, but also complete new technical and methodological research possibilities arise.

LINZ - Land Information New Zealand

I would like to start with Land Information New Zealand (LINZ). LINZ, as a governmental body, has issued and maintains New Zealand’s geospatial strategy. LINZ runs the LINZ Data Service, which provide tons of NZ-related data sets, topography, maps, place names and much more, almost all of it is available under a NZ Creative Commons license. You can register for free, get an API key and use data directly through web, basically as long as you tell that it is LINZ data. LINZ provides standard OGC CSW, WMS and WFS web service interfaces.

http://data.linz.govt.nz

More news about the NZ geospatial strategy can be found on here.

DOC – Department of Conservation

Also the New Zealand Department of Conservation is going towards geospatial web services. It looks like they use ESRI software, which supports OGC standards to certain bit already, although ESRI (producer of the ArcGIS software) as a commercial closed-source software provider has been known to notoriously neglect open standards. However, the Shapefile format is open and besides ESRI REST services, the DOC Geoportal also allows for OGC-based access (CSW/ISO 19139 metadata for search and discovery and WMS/WFS for map/feature data access)

http://geoportal.doc.govt.nz

GNS Science

The Institute of Geological and Nuclear Sciences is one of the 9 New Zealand Crown Research Institutes (CRI), which conduct about one half publicly/governmentally funded and the other half commercial research projects and, together with the universities of course, can be seen as New Zealand’s main science and research providers, each claiming a particular scientific domains. GNS Science is New Zealand’s leading provider of Earth, geoscience and isotope research and the geological survey of New Zealand. GNS’s research topics also include volcanoes, earthquakes, geothermal features and groundwater.

GNS has published the 1:250 000 Geological Map of New Zealand (QMAP. It is also digitally accessible – GNS exports the QMAP as OGC WMS and WFS in the OGC format GeoSciML. An easy way and very interesting example for interoperability is to explore New Zealand’s geology is through the OneGeology project, which sources and displays such services from geological surveys from all over the world.

The GNS-EU collaborative SMART Acquifer Characterisation programme (SAC) also aims to connect OGC based data sources. Within the research aim “Data Synthesis and Visualisation” the SMART Data Portal aims to develop an integrated OGC framework for discovery, access, processing and visualisation of hydrogeological data.

http://data.gns.cri.nz/metadata/

NIWA – National Institute of Water and Atmospheric Research

NIWA NIWA, another CRI, has a strong reputation in climate, marine and marine ecosystem and biodiversity sciences. Whereas a lot of organisations and agencies make data available first and then (if at all) add more sophisticated search technology, NIWA started the other way round. They established a discovery portal - the Environmental Information Browser, which is basically a catalogue, where one can search by keywords, places, data and time. All the data NIWA has, will eventually be listed and can be queried and also harvested through the OGC CSW interface. Furthermore NIWA is also moving towards providing OGC web services to their data sets. One particular example has been a “Summer of eResearch” project and its progress documented on the eResearch website.

http://ei.niwa.co.nz

Landcare Research

Landcare Research is also a New Zealand Crown Research Institute and focuses on the management of terrestrial biodiversity and land resources in order to both protect and enhance the terrestrial environment. I have come across several Landcare projects on soils and land use data, where Landcare not only uses OGC standards, but also participates in the development and maturing of some of those standards. Like the former parties, Landcare runs a data or geoportal (LRIS), too, which can be accessed and queried through CSW, WMS and WFS web interfaces,

https://lris.scinfo.org.nz/

Landcare also hosts a dedicated soil map portal (S-map), which sources the digital soil information layers based on WMS. Furthermore they drive the development of a global soil map portal (http://www.globalsoilmap.net/), which under the hood, of course, uses OGC standards again. To enable international, comparable, interoperable soil data exchange Landcare participates in the development of a soil information standard.

Outlook

Regional councils are on the way, too. Many regional councils already make data accessible on their web sites. A quick investigation shows for example Environment Waikato, Horizons, HBRC, BOP or Environment Canterbury. However most of these data sources need to be accessed manually and/or do not provide a standardised interface. Not to speak of a generalised way to actually find them. In conjunction with the open data initiatives (Open and Transparent Government , Open New Zealand) and catalogues available ( government datasets online, Open Data Catalogue), there is massive potential to link diverse datasets, relate and analyse seemingly unrelated datasets and gain new insights, find and (re-use) data by type, time and location or just enable ubiquitous mobile access to the data you need. However, we might end up needing a catalogue for the catalogues, and of course a lot of existing data needs to be geo-located/geo-referenced, so that they could be found by location. There is still a way to go and definitely some more research necessary in that space.

Originally posted on the New Zealand eResearch website - 16/04/2013