WARREDOC International Winter School on Data Rich Hydrology 2019

The "Data Rich Hydrology" Winter School 2019 took place in beautiful Colombella, Perugia (Italy). It was jointly organised by the Water Resources Research and Documentation Center (WARREDOC) and UNESCO World Water Assessment Program (WWAP). The WARREDOC was established at the Università per Stranieri di Perugia (UNISTRAPG) since 1985 - developing research, advanced training and scientific communication in the field of water, environment and disaster risk management.

http://warredoc.unistrapg.it/en/events/2019-winter-school/

The days were organised into a serious of lectures and lab sessions and food and accommodation were provided all at the location of the Villa Colombella, that was an extraordinary experience. We were completely immersed in this place students and lecturers altogether.

The program encompassed mainly lectures of absolute high scientific standard and well presented by the experienced and well-known lecturers.

- The Era of Data Rich Hydrology, 1st keynote lecture, by Prof. Rafael L. Bras

Prof. Bras is one of the forefathers of hydrology (Google Scholar). He gave us a history lesson of conceptual, numerical and later computational hydrology and modelling of catchments. He concluded with the outlook of what we as young hydrologists should keep striving towards to improve understanding and modelling of the hydrological cycle.

- The WWDR and SDG 6 Synthesis Report, 2nd keynote lecture, by Prof. Stefan Uhlenbrook, also head of UNESCO WWAP

- Remote sensing and data assimilation in hydrology by Prof. Fabio Castelli

- Hydrologic modelling in a data rich world by Prof. Prof Riccardo Rigon

- Citizen science and big data in hydrology by Prof. Fernando Nardi

- Beyond traditional extreme value theory: lessons learned from rainfall and hurricane intensity by Prof. Marco Marani

More topics got covered by further renowned professors, researchers and practitioners in hydrological and hydraulic modelling:

- Groundwater hydrology and hydrological process mechanics
- The water-food-energy nexus
- Modelling scaling properties of precipitation fields
- Hydrologic measurements and novel observation technologies
- Drones in Hydrology (lecture & hands on)
- Hydrological risk assessment: Return period and probability of failure
- Advances in the space-time analysis of rainfall extremes
- Data poor vs. data rich cases for flood hazard (lecture & hands on)
- Distributed Data quality and urban flood modelling uncertainty
- Stream flow measurements: ground and satellite observations
- Remote sensing data and tools to foster inland water monitoring and flood modeling

I also had the pleasure to get interviewed by research fellow and PhD student Francisco Pena, who does a radio show on Disaster Risk Reduction. We had a great chat about our ideas and views on the topics and lectures during this Winter School on Hydrology and did some brainstorming:

http://www.radiophonica.com/podcast/13941 (link to the radio show)

If you like to check out Francisco's pages: https://www.linkedin.com/in/franciscope%C3%B1a/ (LinkedIn) and and https://twitter.com/FebronioPena (Twitter)

Interoperable exchange of groundwater data with OGC GroundWaterML2

WaterML2 has become a well-known synonym for internationally standardised hydrological data exchange, in particular for government agencies and research institutes across North America, Europe, Australia and New Zealand. Technically, WaterML2 is becoming a suite of standards actively promoted and endorsed by the World Meteorological Organisation (WMO), more details: http://www.whycos.org/wordpress/?page_id=929)

- WaterML 2.0: Part 1 - Time series of Observations

- WaterML 2.0: Part 2 - Ratings, Gaugings and Sections

- WaterML 2.0: Part 3 - Surface Hydrology Features (aka HY_Features)

- WaterML 2.0: Part 4 - aka GroundWaterML 2 (GWML2) Data Exchange for Groundwater Features (including wells, springs, borelogs and well constructions)

Now there is a scientific publication that explains the GWML2 standard, its development and application in hydrogeology in detail:

"GWML2 is an international standard for the online exchange of groundwater data that addresses the problem of data heterogeneity. This problem makes groundwater data hard to find and use because the data are diversely structured and fragmented into numerous data silos. Overcoming data heterogeneity requires a common data format; however, until the development of GWML2, an appropriate international standard has been lacking. GWML2 represents key hydrogeological entities such as aquifers and water wells, as well as related measurements and groundwater flows. It is developed and tested by an international consortium of groundwater data providers from North America, Europe, and Australasia, and facilitates many forms of data exchange, information representation, and the development of online web portals and tools."

Brodaric, B., Boisvert, E., Chery, L. et al. (2018) Enabling global exchange of groundwater data: GroundWaterML2 (GWML2). Hydrogeology Journal. https://doi.org/10.1007/s10040-018-1747-9

Related links and information:

https://link.springer.com/article/10.1007%2Fs10040-018-1747-9

https://www.researchgate.net/publication/323914313_Enabling_global_exchange_of_groundwater_data_GroundWaterML2_GWML2


WaterML2 Part 4: GroundWaterML2 (GWML2) http://www.opengeospatial.org/standards/gwml2

Groundwater SWG http://www.opengeospatial.org/projects/groups/groundwaterswg

OGC GroundWaterML 2 – GW2IE FINAL REPORT https://portal.opengeospatial.org/files/?artifact_id=64688

Archived posts from my Github pages

• July 16, 2015: Google Cloud Platform, Cassandra and Mesos
• July 14, 2015: Insights into Raster Imagery Tiling with GeoTrellis
• June 30, 2015: A GeoTrellis Spark Cassandra Experiment on Google Compute Engine
• June 05, 2015: Setting up your IDE for development with GeoTrellis (or any other complex Scala project)
• March 27, 2015: GeoTrellis Cassandra Backend to GeoTrellis-Spark
• March 26, 2015: ZOO-Project Java-API and JGrasstools
• April 13, 2014: Space Apps Challenge 2014 - TETRIS Space Suit HUD demo with a Raspberry
• April 16, 2013: Geospatial web-enablement for environmental data in New Zealand
• February 04, 2013: SOS for CLIDB Mid-term - we got data
• January 21, 2013: A side note on geospatial data sharing and spatial data infrastructures (SDI)
• January 14, 2013: A Climate Database Web Service - 1st Review
• December 10, 2012: A Sensor Observation Service for the NIWA climate database
• August 23, 2012: Demonstrating Exchangeable Encodings for SOS - A Quantum Leap for me
• July 04, 2012: Exchangeable Encodings Getting in Shape
• May 30, 2012: Dynamic Output Formats for the Sensor Observation Service
• February 29, 2012: Exchangeable Encodings for SOS - extension support WaterML2

Disclaimer: There might be some duplication with older posts here, too :-)

SOS for CLIDB Mid-term - we got data

Based on the "vanilla" code from the 52North SVN repository I basically added a CLIDBHelper class to the "52n-sos-hibernate-core" Maven module that encapsulates the CLIDB database connection and re-directed most of the original Hibernate calls from the core DAOs to the helper.

Especially the Cache Feeder DAO needed a lot of attention, as it would fill the internal "capabilities" cache, which is the basic thing that needs to happen before the SOS instance is actually usable.

Then Describe Sensor DAO and GetObservation DAO are pretty straightforward, as they would "just" do specific queries, like give me the sensor description for sensor station XY or give rainfall data for station XY in the time period ...

Link to the full eResearch blog post

Final NZ Summer of eResearch 2012 presentation slides

The final student project presentations were held in Wellington at Victoria University.

Link to the slides of the final student project presentation

A Sensor Observation Service for the NIWA climate database

The National Institute of Water and Atmospheric Research (NIWA) operates a network of weather and climate and other measurement stations, equipped with different sensors, observing a wide range of environmental properties – from temperature over rainfall to wind speed and directions. These high frequency measurements are processed and stored in the NIWA climate database (CLIDB), which is also listed as a national significant database. NIWA has already a web interface in place, where data can be queried and downloaded. Yet this is a manual process.

The Open Geospatial Consortium (OGC) is an international consortium comprising of organisations from industry and research that develops geospatial data transfer and encoding standards and specifications in an open and consensus based process. The OGC Sensor Observation Service (SOS) is a web service interface specification describing access to sensor and time series data – the observations – measured or observed primarily by sensors.

With a Sensor Observation Service in place, the CLIDB could actually be queried like a database, but through the web, automagically, from within an application – based on an international standard. Plugins for e.g. R Statistics (sos4r), OpenLayers (OL sos demo) or ArcGIS (ArcHydro and 52°North ArcGIS SOS Extension) are already available to demonstrate data access from SOS servers.

In this Summer of eReseach project I will build a custom connector to source CLIDB directly, based on the sophisticated 52°North SOS server. But the devil is in the details. Although I have already worked with this software I will have to learn more about the database structure of the CLIDB and immerse in the latest development version of the 52°North SOS server, which will support Hibernate, the quite new OGC SOS 2.0 specification and exchangeable output encodings (e.g. CSV, O&M, WaterML2.0 and maybe JSON). Additionally we will discuss and demonstrate ways to include SOS data services in a client application.

A lot of work, but worth the effort. I am looking forward to create something new and useful. Besides the apparent advantages for New Zealand’s researchers, also consultancies, commercial organisations and governmental agencies will profit from such a dynamic, standards-driven and web-based geospatial data delivery service that could serve as another good example for environmental data providers.

Link to the eResearch blog post

GSoC and Master Thesis completed

Busy summer, well or winter, depends on the hemisphere and the full time extent. Last 2-3 months have been packed with work and included about 25.000 air miles (which means concatenated spending several days in airplanes).
I finally managed to finish my Master Thesis "Towards a 4D WebGIS using harmonised datasets: Examined on a New Zealand Example" which examines in the first part existing groundwater-related geoportal projects and scientific applications around the globe, existing and useful, web-based technologies and standards (e.g. OGC) to implement such a portal, a draft architecture. In the second part actual prototypical implementations of a selected subset of the examined standards and and technologies demonstrate their feasibility for the SMART project. I also hold a talk at the GI_Forum conference in Salzburg (proceedings/conference paper).
Furthermore I successfully completed my Google Summer of Code (GSoC) project with the 52°North Initiative, where I implemented an exchangeable encodings mechanism for their Sensor Observation Service (SOS). I additionally implemented a plugin that provides SOS time-series output in the WaterML2.0 format. I could acquire the latest schema, which will be published soon.. we were supposed to write a series of blog articles to document or progress in the project:

• Dynamic output formats for the Sensor Observation Service
• Exchangeable Encodings getting in shape
• Demonstrating Exchangeable Encodings: A Quantum Leap for me

After flying back to New Zealand I was lucky to visit the GeoSciML Face-to-Face-Meeting at GNS Science in Wellington. It was really exciting to meet these people, working on the international standard of a Geoscience Markup Language. also issues around the OneGeology portal have been disussed, as it uses GeoSciML to build up a world geology map. Additionally I could get in touch with one of the inspiring driving persons behind the Canadian Groundwater Information Network and their its sematic foundation, the Groundwater Markup Language (GWML), which itself is derived from GeoSciML.
Now I will dedicate my research to the development of a New Zealand groundwater geoportal within the SMART project, using OGC webservices, GeoSciML and/or GWML, WaterML2.0 and X3D as its foundations to transform and consume all available data relevant to aquifer characterisation and create a completely new, visually appealing 3D/4D view on New Zealand's groundwater resources.

Exciting ;-)

Demonstrating Exchangeable Encodings for SOS - A Quantum Leap

The final sprint towards the end of the Google Summer of Code (GSoC) is over now. After fixing bugs, working on documentation and actually producing time-series in different formats, the “Exchangeable Encodings for SOS”
project met most of its ambitious goals – and it is demonstrated on the 52°North Google summer of Code Demo Server. Users and developers documenation on the encoding mechanism has also found a place in the 52°North wiki.

We have integrated a plugin mechanism into an SOS server (source code branch), which is based on the main 52°North SOS development line (source code trunk).

Link to the article on the 52°North blog

Exchangeable Encodings Getting in Shape

The time is flying by and soon we reach midway of the Google Summer of Code (GSoC) with 52°North. It is the typical time in between – I already managed to meet the first milestones, but there is no proper product yet. Catching up with the changes from the SOS main development branch is a constant challenge, but the infrastructural changes within the “Exchangeable Encodings for SOS” project are slowly but continuously leading towards a robust and standardized, yet limited, encoding plugin API (application programming interface).

The main use case is the request for observations (GetObservation request) that could be delivered in CSV (comma separated values) format for legacy applications, WaterML2.0, an upcoming OGC (Open Geospatial Consortium)
data transfer standard for the hydrological domain, possibly netCDF (Network Common Data Form), also an OGC standard for meteorological applications, and in O&M (Observations & Measurements) for the generic use within sensor networks and spatial data infrastructures.

Link to the article on the 52°North blog

Dynamic Output Formats for the Sensor Observation Service

As one of the 52°North Google Summer of Code projects, “Exchangeable Encodings for the 52°North Sensor Observation Service (SOS)” aims to implement a customisable result set encoding mechanism. The users would then be able to add their own compiled libraries (like plugins), which contain all the necessary code to provide the requested data in different formats, such as CSV or GeoJSON, to the SOS. When that mechanism works, we can easily add more encodings, e.g. the planned WaterML2.0 – but for now, where to start?

The internal architecture of the 52°North SOS server is generally well structured. Based on the user request, the respective Listener will get the data from the data backend and send it to the corresponding Encoder. The response document, very likely a collection of observations, is then returned to the user in the requested encoding, also referred to as the response format. The typical application flow of such a request is depicted in the comic strip below. Available response formats can be queried from SOS server and are listed in the capabilities document. To date the two integrated encoders return XML documents based on the OGC© O&M schema and the OGC© SOS versions.

Link to the article on the 52°North blog

Boiling it down

Boiling it down

After all this theoretical research about why, what, when and how I identified three major work packages to develop a functional prototype:

1. making existing groundwater-related time-series and measurements data sets available via SOS O&M (or even WaterML2.0) - March 2012
2. chosing a demonstration area and associated available sample data, analyse, describe and harmonise these different data sets, design the database scheme and migrate them to the SMART main repository, display all fine in proper standard 2D web map - April 2012
3. take exisitng 3D geological model ( x-y-z grids from EarthVision) and make them visible as X3D model in the browser (with or without plugin :-) ) - May 2012

The thesis should be submitted around mid of June, so there are some smaller buffers. But thesis itself needs to be written, too :-)

Groundwater Resources in New Zealand

Data Synthesis and visualisation for groundwater resources in New Zealand

“As a conservative estimate, better understanding of groundwater resources could lead to improvements in water management with potential to prevent social, cultural, environmental and economic losses of $1 billion or more, about 1% of New Zealand’s GDP. Understandably, central and regional government has identified improved characterisation of groundwater resources as a top research priority.”

The SMART project is about characterisation of New Zealand’s aquifers, to improve ground- and freshwater management (www.smart-project.info).

The main research aims of the SMART project

• focus on novel, passive suite of methods to measure groundwater volume and changes in it over time
• aquifer hydraulic properties
• fluxes of groundwater interchange with surface waters
• water age
• with methods like ...
• ambient noise seismic tomography
• airborne geophysical surveying
• satellite remote sensing
• fibre optic temperature sensing
• novel age tracers

Outcomes shall made available via a web portal and harmonised 3D groundwater database that will meet stakeholder needs for open access, ease of use, and interoperability with existing systems. Furthermore the integration of existing data is crucial.

My Thesis Topics

I contribute in the field of “data synthesis and visualisation” in the form of a prototype 3D/4D WebGIS, data processing and web services to provide harmonised hydro(geo)logical data from multiple sources. Within the works for my Master Thesis until June 2012, I identified following milestones:

• stakeholder consultation - groundwater workshop, visited HBRC and EW
• gather and analyse (sample) hydro(geo)logical datasets from wells, boreholes or pump tests, managed in systems/databases like GGW/NGMP (GNS), Hilltop and Wiski (very popular among regional councils) – Feb 2012
• find a common data scheme/data model, migrate and/or transform data online and provide via web services to connect to a (NZ) SDI or integrate/link in web portal for easy access – April 2012
• identify appropriate technology and standards (ISO, OGC, OASIS … ) 3D/4D visualisation in the web, as aquifers are inherently 3D structures plus time (water time series data -> 4D) – June 2012

The prototype implementations of this thesis will be in test mode, accessible by designated staff and stakeholders. The research about “data synthesis and visualisation” including the topics of this Master Thesis will be continued in more detail by a PhD student from July 2012.

We focus on hydro(geo)logical data. The first steps are about accessing multiple existing sources of data and find a way to publish them via web services (most likely OGC).

Major Challenges include

• complexity of water-related data – many parameters and measurement properties, parameter naming, units etc.
• data policy issues (ownership, restricted access ...)
• data quality and quality and assurance issues – uncertainty, hint/description in metadata
• as existing data often is not accessible outside their environments (GNS network, regional councils’ local data storage), or mostly via manual file-based email/ftp interchange or as “pdf”-reports, the publishing of existing data from multiple sources is anticipated via the web e.g. via WMS, WFS, WCS, WaterML2.0?, SOS, download (shapefile, raster coverage) to allow easy recombination, correlation and analysis at national scale (surpassing regional boundaries)
• find a common data scheme, probably “just” implement a software to access databases and deliver WaterML2.0?! Good idea might be to transform online (e.g. XSLT/WPS web service)
• provide infrastructure, for storage, access, delivery and manipulation of different geodata (vector, raster, time series …)
• Prepare catalogue and provide metadata in standard encodings (XML, OGC web services, ISO 19115/19139 metadata, CSW, CS-W ebRIM/ISO)
• Regarding exisiting methods and infrstructures, like Auscope / SISS (AU), GeoNet (NZ), GENESIS FP7 (EU), CUAHSI (North America)
• Think about GeoSciML for description of geology and geometry, as WaterML is more about time series and measurement data prototypical web 3D/4D map/aquifer visualisation (X3D earth ...)

Exchangeable Encodings for SOS - extension support WaterML2

The 52°North SOS implements only a subset of the O&M data format, because O&M is too flexible to be supported completely. It would be better if users could just implement a simple encoding (for example to support their own CSV or JSON data format) and upload a compiled jar file to the SOS with a browser-based user interface. Then a client selects the newly available response format to retrieve the data in a new format. As an extension it would be very useful to researchers and data maintainers in the hydrological and meteorological domain to have the SOS server deliver WaterML2.0 encoded time-series. Based on the SOS adminstrator project, the desired encodings could be prepared as .jar-files and added via a web frontend.

Link to the 2012 GSoC proposal