WARREDOC International Winter School on Data Rich Hydrology 2019

The "Data Rich Hydrology" Winter School 2019 took place in beautiful Colombella, Perugia (Italy). It was jointly organised by the Water Resources Research and Documentation Center (WARREDOC) and UNESCO World Water Assessment Program (WWAP). The WARREDOC was established at the Università per Stranieri di Perugia (UNISTRAPG) since 1985 - developing research, advanced training and scientific communication in the field of water, environment and disaster risk management.

http://warredoc.unistrapg.it/en/events/2019-winter-school/

The days were organised into a serious of lectures and lab sessions and food and accommodation were provided all at the location of the Villa Colombella, that was an extraordinary experience. We were completely immersed in this place students and lecturers altogether.

The program encompassed mainly lectures of absolute high scientific standard and well presented by the experienced and well-known lecturers.

- The Era of Data Rich Hydrology, 1st keynote lecture, by Prof. Rafael L. Bras

Prof. Bras is one of the forefathers of hydrology (Google Scholar). He gave us a history lesson of conceptual, numerical and later computational hydrology and modelling of catchments. He concluded with the outlook of what we as young hydrologists should keep striving towards to improve understanding and modelling of the hydrological cycle.

- The WWDR and SDG 6 Synthesis Report, 2nd keynote lecture, by Prof. Stefan Uhlenbrook, also head of UNESCO WWAP

- Remote sensing and data assimilation in hydrology by Prof. Fabio Castelli

- Hydrologic modelling in a data rich world by Prof. Prof Riccardo Rigon

- Citizen science and big data in hydrology by Prof. Fernando Nardi

- Beyond traditional extreme value theory: lessons learned from rainfall and hurricane intensity by Prof. Marco Marani

More topics got covered by further renowned professors, researchers and practitioners in hydrological and hydraulic modelling:

- Groundwater hydrology and hydrological process mechanics
- The water-food-energy nexus
- Modelling scaling properties of precipitation fields
- Hydrologic measurements and novel observation technologies
- Drones in Hydrology (lecture & hands on)
- Hydrological risk assessment: Return period and probability of failure
- Advances in the space-time analysis of rainfall extremes
- Data poor vs. data rich cases for flood hazard (lecture & hands on)
- Distributed Data quality and urban flood modelling uncertainty
- Stream flow measurements: ground and satellite observations
- Remote sensing data and tools to foster inland water monitoring and flood modeling

I also had the pleasure to get interviewed by research fellow and PhD student Francisco Pena, who does a radio show on Disaster Risk Reduction. We had a great chat about our ideas and views on the topics and lectures during this Winter School on Hydrology and did some brainstorming:

http://www.radiophonica.com/podcast/13941 (link to the radio show)

If you like to check out Francisco's pages: https://www.linkedin.com/in/franciscope%C3%B1a/ (LinkedIn) and and https://twitter.com/FebronioPena (Twitter)

Interoperable exchange of groundwater data with OGC GroundWaterML2

WaterML2 has become a well-known synonym for internationally standardised hydrological data exchange, in particular for government agencies and research institutes across North America, Europe, Australia and New Zealand. Technically, WaterML2 is becoming a suite of standards actively promoted and endorsed by the World Meteorological Organisation (WMO), more details: http://www.whycos.org/wordpress/?page_id=929)

- WaterML 2.0: Part 1 - Time series of Observations

- WaterML 2.0: Part 2 - Ratings, Gaugings and Sections

- WaterML 2.0: Part 3 - Surface Hydrology Features (aka HY_Features)

- WaterML 2.0: Part 4 - aka GroundWaterML 2 (GWML2) Data Exchange for Groundwater Features (including wells, springs, borelogs and well constructions)

Now there is a scientific publication that explains the GWML2 standard, its development and application in hydrogeology in detail:

"GWML2 is an international standard for the online exchange of groundwater data that addresses the problem of data heterogeneity. This problem makes groundwater data hard to find and use because the data are diversely structured and fragmented into numerous data silos. Overcoming data heterogeneity requires a common data format; however, until the development of GWML2, an appropriate international standard has been lacking. GWML2 represents key hydrogeological entities such as aquifers and water wells, as well as related measurements and groundwater flows. It is developed and tested by an international consortium of groundwater data providers from North America, Europe, and Australasia, and facilitates many forms of data exchange, information representation, and the development of online web portals and tools."

Brodaric, B., Boisvert, E., Chery, L. et al. (2018) Enabling global exchange of groundwater data: GroundWaterML2 (GWML2). Hydrogeology Journal. https://doi.org/10.1007/s10040-018-1747-9

Related links and information:

https://link.springer.com/article/10.1007%2Fs10040-018-1747-9

https://www.researchgate.net/publication/323914313_Enabling_global_exchange_of_groundwater_data_GroundWaterML2_GWML2


WaterML2 Part 4: GroundWaterML2 (GWML2) http://www.opengeospatial.org/standards/gwml2

Groundwater SWG http://www.opengeospatial.org/projects/groups/groundwaterswg

OGC GroundWaterML 2 – GW2IE FINAL REPORT https://portal.opengeospatial.org/files/?artifact_id=64688

PhD Dissertation publicly accssible

My dissertation to achieve the degree 'Doctor of Philosophy' at the Auckland University of Technology (AUT) has the title:

A Context-based Groundwater Data Infrastructure

Online access via the AUT Online Library: http://hdl.handle.net/10292/10740

Abstract

Groundwater bodies are among the most important and valuable natural resources available, but at the same time they are also the least understood. To better understand the hydrological state of the environment and groundwater dynamics, data sets and measurements need to be made available and accessible to scientists, planners, and stakeholders to allow for proper decision making support. Information exchange via the internet has become faster, but at the same time data sets remain scattered both in location and formats. Present research in hydrogeology and freshwater resources management can be significantly supported and accelerated by relating, reusing and combining existing data sets, models and simulations in a streamlined, computer-aided and networked fashion and yield more new and reproducible insights.

In this thesis Design Science Research, Grounded Theory and Case Studies are applied in order to design a spatial data infrastructure that addresses the full data life cycle in the context of hydrogeology in New Zealand. This 'Hydrogeology Infrastructure' design was successfully implemented and evaluated via a networked and open standards-based prototype. Formerly disconnected and distributed data sets may now, for the first time, be used for hydrogeological data analysis, visualisation and modelling within one data portal.


My Supervisor(s)

Many thanks go out again to my supervisors who supported me continuously.

Assoc Prof Jacqueline Whalley (AUT, New Zealand); Assoc Prof Hermann Klug (Z_GIS, Salzburg University, Austria); Prof Philip Sallis (AUT, New Zealand)

The interested reader can find the publication also via ResearchGate:
https://www.researchgate.net/publication/319164590_A_Context-based_Groundwater_Data_Infrastructure

A Context-based #GroundwaterData Infrastructure #Hydrogeology #GIScience # SpatialDatahttps://t.co/QMpSKTpsp2

— Scholarly Commons (@AUT_SC) 17 August 2017

Environment Southland Information Management Conference

Regional councils and government agencies are increasingly under pressure to resolve data questions and discover how best to acquire, manage, collate, analyse, report and disseminate data, while managing  the associated costs. Steering organisations through these complex issues requires a solid understanding of what technologies are available and the information demands of the future *(source).

I had the great opportunity to speak at the Environment Southland Information Management Conference in Invercargill. It was a great event, well organised and very informative. I believe I could contribute my part to the line up and fill a few more gaps in the whole picture.
This was not a business as usual conference, it was obvious that the speakers took it serious to cater their presentations to the needs of the stakeholders. And with 70 attendees from regional and central government, as well as visitors from research and industry.

It was great to see the emerging patterns around NZ and similar approaches to a holistic, comprehensive and modern data strategy. If you are interested, this is a link to programme, and please see below for my slides. Watch the talk on Youtube.

A Spatial Data Infrastructure Approach for the Characterization of New Zealand's Groundwater Systems

I was very happy when I got informed that our latest research article was published in "Transactions in GIS". While being embedded in our New Zealand SMART Aquifer Characterisation programme, it was a great joint effort which results went beyond SMART.

Kmoch, A., Klug, H., Ritchie, A. B. H., Schmidt, J. and White, P. A. (2015), A Spatial Data Infrastructure Approach for the Characterization of New Zealand's Groundwater Systems. Transactions in GIS. doi: 10.1111/tgis.12171

It explains technical and methodological approaches of web groundwater data infrastructure and uses NIWA's CLIDB and GNS's NGMP as examples and describes the long way from stakeholder interaction, workshops and meetings towards an NZ data standard for the National Environmental Monitoring Standards (NEMS) framework - the Environmental Observation Data Profile (EODP) as data access and transfer blueprint for NEMS.

Wiley Online Library "Transactions in GIS" Journal

Alexander Kmoch on ResearchGate.

The technical work with colleagues, collaborators and partners from New Zealand research institutes (CRIs) like GNS, NIWA, Landcare Research, New Zealand regional councils like Horizons and Waikato (WRC), Hawke's Bay (HBRC) and and Bay of Plenty (BOPRC) regional councils which culminated in a draft OGC profile and is being incorporated into national standards in New Zealand, GitHub EODP and are based on the work over the last two years including a great review paper on OGC standards for groundwater,  customising the 52°North SOS server to interchangeably encode WaterML2 time-series data along O&M2 through a Google Summer of Code (GSoC) programme, and then customising the SOS server database access to demonstrate it as an adaptor on legacy databases of national significance, in specific an "NGMP/GGW-SOS" and a "CLIDB-SOS" demo through a NZ eResearch programme.

Digital Earth Conference 2015, Canada

"Towards a One-World Vision for the Blue Planet" was the slogan for the 9th Symposium of the International Society for Digital Earth (ISDE) from 5th - 9th of October, 2015, in Halifax, Nova Scotia, Canada.

At Digital Earth 2015 scientists, engineers, technologists, and environmental managers from around the world will meet to share concepts, research findings, technologies, and practical applications relating to the Digital Earth vision. *()

I presented an experimental approach towards integrating scientific legacy codes into the OGC web services framework, on the example of exposing USGS MODFLOW through a vanilla and open source WPS (without using proprietary tools like ESRI Arc Server and Arc Hydro Tools etc).

I was very honoured and greatly appreciated the "Best Student Poster" award. It seems I really had the right audience at ISDE.

Furthermore it was extremely informative, and in the session about "Discrete Global Grid Systems" I learned a lot about this new approach of unifying traditional raster/coverage data with an equal area per pixel advantage ... one of the great shortfalls of todays popular Web Mercator projections. In fact spinning this further it could be a  great new approach and how groundwater, geology, ocean modelling and atmospheric sciences, which would benefit from an equal volume grid, particularly when re-mashing resources from different environmental/scientific/governmental/industrial domains.

There is even an DGGS OGC Standards Working Group on the way.

ZOO-Project WPS Java-API and JGrasstools Java Hydrological Toolbox

The ZOO-Project (http://zoo-project.org/) is a solid Open Geospatial Consortium (OGC) Web Processing Service (WPS - http://www.opengeospatial.org/standards/wps) standard server implementation with an open flexible API that works well with many different programming languages. The Java bindings have never been tested in advanced configurations and complex data types, and to date only implement the minimum necessary interfaces. The JGrasstools project is a modular processing library and its highly annotated nature makes it possible to adapt quite easily to other toolboxes. JGrasstools contains a wide variety of powerful and efficient GIS, hydrology and geomorphological tools and processes, that can be exposed to and used by other libraries and toolkits. One example has been the adaptation to the Geotools Process API. The JGrasstools project, as well as other java based projects (as JTS, Sextante or even Geotools) would benefit greatly from the possibility to be used within a web-enabled WPS execution environment, as well as being integrated with the open standards suite of the OGC.  Some time ago Moovida tried integrating the JGrasstools libraries with the ZOO-Project Java binding to expose them as native WPS processes. This would allow them to work inside the ZOO-Project and serve its modules under the WPS standard.

Some Background

The ZOO-Project WPS implementation is a flexible, modular high performance HTTP CGI implementation. ZOO-Kernel is a powerful server-side C Kernel which makes it possible to manage and chain Web services, by loading dynamic libraries and handling them as on-demand Web services. The ZOO Kernel is written in C language, and supports several common programming languages in order to connect to numerous libraries and models (http://zoo-project.org/trac/wiki/ZooWebSite/ZooKernel). The generic ZOO API is basically accessible for every possible programming and web scripting language that can be run under the CGI interface. Main API implementations, the ZOO services, are available for C/C++, Python, JavaScript, PHP, Fortran and Java. Some API bindings are more advanced and complete and make the full ZOO-API (http://zoo-project.org/trac/wiki/ZooWebSite/ZOOAPI/Classes#ZOOAPIClasses) accessible to the ZOO service in the particular programming language (e.g. C, Python or JavaScript). In comparison the the Java API binding only exposes the minimum functionality to run from the ZOO Kernel.

JGrasstools (http://moovida.github.io/jgrasstools/) is a powerful GIS toolkits with functionality reaching from standard geoprocessing algorithms to advanced processing features used in hydrology and geomorphology. JGrasstools is based on a Maven (http://maven.apache.org/) build process, which takes care of dependency resolution and creating the succinct jar packages with the compiled classes. Maven is a defacto standard for managing (source and dependencies) building and deploying (jar packaging, resources, copying, publishing, archiving and installing) Java-based software projects. JGrasstools is also used as a toolbox in the uDig desktop GIS software (http://udig.refractions.net/). If JGrasstools could be exposed via a open standards and interfaces, web-based processing and execution environment (like ZOO-Project provides) it can be widely used in WebGIS deployments and large scale cloud based processing chains.

The next level

Andrea from Moovida said he didn't have enough time to continue developing this idea. He developed a generator which would programmatically scan through the annotated JGrasstools modules and generate respective ZooJavaWps classes per JGrasstools module/method and the corresponding ZOO-Project .zcfg config file. The only struggle I had was getting the CLASSPATH properly set up, as the ZOO-Project is basically an HHTP CGI application which will start a JVM per request. When I picked up on this in preparation for a GSoC proposal, I found that there were a few little botches with the parameter mapping from ZOO Java API into the very nicely annotated JGrasstools methods. So I took one example generated (WPS-ified) JGrasstool process and adjusted the parameter mapping and got it running with ZOO-Project. Additionally I adjusted the JGrasstools Maven config files to download the necessary dependencies in the target folders to copy them collectively in the ZOO-Project Java CLASSPATH.

Unfortunately I also didn't have time to drive this further still. However, it is just soooo close really :-) Alternatively a 52North WPS implementation based on the super practical JGrasstools annotations and Andrea's generator would also be relatively straightforward.

EODP - An OGC-based Environmental Observation Data Profile

A group around Landcare, NIWA, GNS, HillTop Software and Horizons Regional council worked the last 2 years to develop an interoperable web-based data access and sharing standard. Working group formed as results from Hydrological Society Annual Conference Data Access Workshops. Subsequent meetings and workshops at Horizons regional Council office in Palmerston North even featured visits and input from Dr David Maidment as well as from delegates from BOM and Kisters, Australia. Further workshops were held at GNS in Wairakei and NIWA offices in Auckland and Wellington.

The NEMS-endorsed Environmental Observation Data Profile (EODP) is based on proven open standards of the Open Geospatial Consortium (OGC) and comprises a profile of WFS that allows the discovery of timeseries data associated with monitoring stations. This forms part of a larger project to enhance the interoperability of Environmental Observations throughout New Zealand.

Organisations such as NIWA, Landcare, GNS and Regional councils maintain a range of monitoring stations making environmental observations. These include Climate, Hydrology, Air Quality, Soils, Water Quality, and Marine data.

We wish to make this data available for research, analysis and reporting through open data protocols such as the Sensor Observation Service (SOS), Web Feature Service (WFS), Web Coverage Service (WCS) and Web Mapping Service (WMS). We also want to publish the existance of such data sets and services through online metadata catalogues (CSW).

In order to make use such data services users must first be able to discover their existence and then accurately select the subset of data that is relevant to their needs. The EOPD is focussed on this selection process in essence addressing the Where, What and When questions.

WFS provides a rich filtering capability, but typical WFS sources use a flat property/value schema that makes it difficult to express the many to one relationship of the timeseries measurements taken at a station. The EODP is an Application Schema for WFS that encodes the station metadata as a nesting of SF_SpatialSamplingFeature and OM_Observations using the language of Observations and Measurements to describe the characteristics of the timeseries.

The profile is extended by the use of an external Vocabulary service used to dereference measurement identifiers into familiar timeseries names such as rainfall or temperature.

Alistair Ritchie from Landcare Research did the main formal write-up and published the documentation on GitHub: https://github.com/EODP-NZ/eodp-dev

A implementation testbed is currently underway in joint collaboration with NIWA on the NGMP database at GNS and the Climate Database (CLIDB) at NIWA.

http://portal.smart-project.info/sos-smart/service?service=SOS&version=2.0.0&request=GetDataAvailability&observedProperty=http://vocab.smart-project.info/ngmp/phenomenon/1679

http://portal.smart-project.info/gs-smart/wfs?request=GetFeature&service=WFS&typename=sams:SF_SpatialSamplingFeature

Web-based 3D Data Visualisation for Hydrogeology

When I was in Salzburg/Austria last year at the GI_Forum 2014 conference, I had the chance to present some of my recent experiments with the mapping of OGC interoperable geo-data to X3D interoperable open web 3D scenes and visualisation in the browser. Back in New Zealand later that year I had to present this work to the New Zealand Hydrological Society, too, of course :-)

Characterisation of a hydrogeological setting is a multi-faceted complex task. The assessment of usefulness and quality of relevant data is a major challenge. Statistical analysis and visual exploration of the datasets demand practical support by computer applications. Although a variety of software for this purpose is freely available nowadays, they require a good understanding of the technology or programming language for application in complex hydrogeological settings. Thus, integrated proprietary software products are often used to analyse and particularly provide high-quality visualisation of the system. However, these software tools are typically desktop programs with a strict licensing scheme and a limited extensibility and lack of interoperability with other applications.
We present an open and free to use web-based (platform independent) framework to enable retrieval, exploration and visualisation of hydro-climate time series data as well as three-dimensional geological information via a web browser. How distributed data and processing services can be linked to prepare an on-demand 3D visualisation of geological and hydrological data is demonstrated. A flexible toolbox design enables extensibility via open standards.

The method developed is applied to a case study area presented (s. figure), which is the Horowhenua district in the Manawatu-Wanganui region. Available datasets of 3D geology, hydrology and hydrogeology are combined and serve as example data for demonstrating the framework.

The slides of the presentations are now made accessible here (click here).

The full ISI-indexed conference paper is available here (click here).