Making environmental research articles discoverable with OGC catalogue service

I am very happy to announce that our new article "Enhancing Location-related Hydrogeological Knowledge" has been published in the ISPRS International Jounral of Geo-information.

Kmoch, A.; Uuemaa, E.; Klug, H.; Cameron, S.G. (2018) Enhancing Location-Related Hydrogeological Knowledge. ISPRS Int. J. Geo-Inf. , 7, 132, http://www.mdpi.com/2220-9964/7/4/132

In a joint study by the University of Tartu (Estonia), the Institute of Geological and Nuclear Sciences (New Zealand) and the University of Salzburg (Austria) more than 5,800 scientific articles from three environmental research journals were digitised and analysed. In addition, a geographical search method was developed to identify the location of a studied area.

We use Georeferencing of scientific journal articles and text-mining in order to provide spatial search capabilities for environmental research. Journal articles are made discoverable through spatial queries. We propose that journal publishers should provide these capabilities on their platforms. This would allow everyone to search for journal articles for their desired regions of interest and it would really well complement existing search functionalities with keywords etc.

The press release from Tartu University explains really nicely how our results enable geographic search for scientific papers through text mining and geocoding; and how to better find environmental research articles via location.

http://researchinestonia.eu/2018/04/17/enabling-geographic-search-for-scientific-papers-through-text-mining-and-geocoding-how-to-better-find-environmental-research-articles-via-location/

Estonian version of the press release: https://novaator.err.ee/821096/kuidas-otsida-teadustoid-geograafilise-piirkonna-jargi


(This article belongs to the Special Issue Place-Based Research in GIScience and Geoinformatics)
MDPI OpenAccess: http://www.mdpi.com/2220-9964/7/4/132

ResearchGate: https://www.researchgate.net/publication/323990934_Enhancing_Location-Related_Hydrogeological_Knowledge

An approach towards a unified hydro-climate metadata search across New Zealand

The SMART project was presenting at the joint Water Symposium 2014 of the New Zealand Hydrological Society, New Zealand Freshwater Sciences Society and the IPENZ Rivers Group.  The Symposium was held at the Marlborough Convention Centre, 24 - 28 November 2014. The theme was “Integration: ‘The Final Frontier’ ~ Whakakotahi te amine rohenga" - integration recognises the continuity of hydrological processes in space and time.In that context we talked about our approach towards a unified hydro-climate metada search across New Zealand.

Acquisition of relevant and useful data for hydrological, meteorological or hydrogeological assessments and research projects in New Zealand’s regions is primarily based on personal communication and reliance on the knowledge of domain specialists. To support the task of identification and subsequently retrieval of available datasets for an area of interest, a new unified method for metadata and data search in New Zealand with a focus on hydro-climate and geo-scientific datasets is presented. The method is implemented within an online accessible search website which searches the many distributed New Zealand based data providers and web portals of scientific and governmental organisations.

The research work undertaken comprises of two parts. First, the publicly accessible abstracts from the website of the Journal of Hydrology New Zealand (NZHS 2014) are analysed. The title, date, authors and abstract text were indexed and where possible a spatial context was derived. Additional keywords were selected for a thesaurus based on occurrence of domain specific terms in the abstract text. Subsequently, metadata records in the New Zealand metadata standard format are created and provided online. Secondly, the web search algorithm distributes the search to a selection of pre-registered geo- and hydro-climate data portals like “LINZ Data Service” (LINZ 2014), “Geodata.govt.nz” (NZGO 2014), “NIWA DC” (NIWA 2014), “Landcare LRIS”(Landcare 2014), “Koordinates” (Koordinates 2014) and “data.govt.nz” (NZ_DIA 2014). The search query can be enriched with related terms from a thesaurus or glossary and spatial context with place names or coordinates. Results are ranked based on spatial and semantic correlations with New Zealand place names register and hydrological terms from the developed glossary.

The slides of the presentation are now publicly accessible. Please click here to view the presentation slides,

EODP - An OGC-based Environmental Observation Data Profile

A group around Landcare, NIWA, GNS, HillTop Software and Horizons Regional council worked the last 2 years to develop an interoperable web-based data access and sharing standard. Working group formed as results from Hydrological Society Annual Conference Data Access Workshops. Subsequent meetings and workshops at Horizons regional Council office in Palmerston North even featured visits and input from Dr David Maidment as well as from delegates from BOM and Kisters, Australia. Further workshops were held at GNS in Wairakei and NIWA offices in Auckland and Wellington.

The NEMS-endorsed Environmental Observation Data Profile (EODP) is based on proven open standards of the Open Geospatial Consortium (OGC) and comprises a profile of WFS that allows the discovery of timeseries data associated with monitoring stations. This forms part of a larger project to enhance the interoperability of Environmental Observations throughout New Zealand.

Organisations such as NIWA, Landcare, GNS and Regional councils maintain a range of monitoring stations making environmental observations. These include Climate, Hydrology, Air Quality, Soils, Water Quality, and Marine data.

We wish to make this data available for research, analysis and reporting through open data protocols such as the Sensor Observation Service (SOS), Web Feature Service (WFS), Web Coverage Service (WCS) and Web Mapping Service (WMS). We also want to publish the existance of such data sets and services through online metadata catalogues (CSW).

In order to make use such data services users must first be able to discover their existence and then accurately select the subset of data that is relevant to their needs. The EOPD is focussed on this selection process in essence addressing the Where, What and When questions.

WFS provides a rich filtering capability, but typical WFS sources use a flat property/value schema that makes it difficult to express the many to one relationship of the timeseries measurements taken at a station. The EODP is an Application Schema for WFS that encodes the station metadata as a nesting of SF_SpatialSamplingFeature and OM_Observations using the language of Observations and Measurements to describe the characteristics of the timeseries.

The profile is extended by the use of an external Vocabulary service used to dereference measurement identifiers into familiar timeseries names such as rainfall or temperature.

Alistair Ritchie from Landcare Research did the main formal write-up and published the documentation on GitHub: https://github.com/EODP-NZ/eodp-dev

A implementation testbed is currently underway in joint collaboration with NIWA on the NGMP database at GNS and the Climate Database (CLIDB) at NIWA.

http://portal.smart-project.info/sos-smart/service?service=SOS&version=2.0.0&request=GetDataAvailability&observedProperty=http://vocab.smart-project.info/ngmp/phenomenon/1679

http://portal.smart-project.info/gs-smart/wfs?request=GetFeature&service=WFS&typename=sams:SF_SpatialSamplingFeature

PLACE 2013 - Maori GIS Conference

I had the great opportunity to present my PhD work at the 2013 Maori GIS conference in Auckland. In order to provide a seamless spatial and multi-purpose view of collected groundwater related datasets, the SMART project joins forces to establish a valuable basis for groundwater analysis and decision support tools (Kmoch et al., 2012, Klug et al., 2011). One of the project’s objectives is to build a web-based data and knowledge portal and attached three-dimensional web visualisation tool according to OGC and ISO compliant standards (OGC, 2012).

Iwi/Māori have a well-recognised relationship with the natural environment which spans many centuries and is the result of interaction and adaptation with native flora and fauna of Aotearoa/New Zealand. Integral to this relationship is water which sustains life and is a taonga (treasure) with significant cultural and physical dimension. This is reflected through the on-going desire of many iwi/Māori groups to have a role in the way water is managed in New Zealand to ensure its sustainable utilisation moving forward (Kawharu, 2002). The development of scientific research tools and models that incorporate mātauranga Māori (Māori knowledge) and te reo Māori (the Māori language) are also beneficial to iwi/Māori resource policymakers, planners and decision makers.

GNS has been collecting and compiling Māori terms on hydrology, geology and geothermal phenomena to, amongst other things, explore the contribution that traditional indigenous knowledge can make to the research outcomes and model development by identifying the cultural significance of groundwater and the associated cultural links with surface water (Tipa and Tierney, 2003, Boast, 1991). The potential benefits of creating research tools that utilise te reo Maori and mātauranga Maori (within government institutional settings) includes generating increased uptake in and familiarity with te reo Māori and exposure of te reo Māori as a minority language to broader audiences. Further research could identify gaps in the dual knowledge systems (either the western scientific knowledge paradigm, or mātauranga Māori) that could be explored as an outcome of this combined research (De Bres, 2008).

To support te reo Māori and mātauranga Māori within the SMART portal web mapping and catalogue application, we evaluate a multi-language concept to incorporate semantic web methodologies to map and connect English and Māori terms and descriptions of presented natural phenomena as well as metadata and descriptive text within the application (Lutz et al., 2009). Beside a language template system for in-application-navigation use, a vocabulary web service is demonstrated to access content the thesauri, classification schemes, taxonomies, metadata and other types of controlled vocabulary and to document, link and merge concepts/terms to be with other spatial and non-spatial data (Antoine Isaac and Ed Summers, 2008). Further intentions include also a possibility to upload and geolocate orally passed on knowledge in the regard of ancient place information.

Evaluation of the lexicon’s effectiveness will be measured in part by its ability to be applied successfully to the SMART portal web mapping and cataloguing application.

References:

  ANTOINE ISAAC, V. U. A. & ED SUMMERS, L. O. C. 2008. SKOS Simple Knowledge Organization System Primer [Online]. Available: http://www.w3.org/TR/2008/WD-skos-primer-20080221/.
  BOAST, R. P. 1991. The legal framework for geothermal resources : a historical study : a report to the Waitangi Tribunal. Wai 153. Wellington: Waitangi Tribunal, 1991.
  DE BRES, J. 2008. Planning for tolerability: promoting positive attitudes and behaviours towards the Māori Language among non-Māori New Zealanders. Ph. D (Linguistics). Wellington: Victoria University of Wellington.
  KAWHARU, M. 2002. Whenua : managing our resources, Auckland: Reed, 2002.
  KLUG, H., DAUGHNEY, C., VERHAGEN, F., WESTERHOFF, R. & WARD, N. D. 2011. Freshwater resources management: Starting SMART characterization of New Zealand’s aquifers. Earthzine. http://www.earthzine.org/2011/12/13/freshwater-resources-management-starting-smart-characterization-of-new-zealands-aquifers/.
  KMOCH, A., KLUG, H. & WHITE, P. 2012. Freshwater resources management: first steps towards the characterisation of New Zealand's aquifers. GI_Forum 2012: Geovisualization, Society and Learning, 2012 2012 Salzburg. Car, A., Griesebner, G., Strobl, J., 376-385.
  LUTZ, M., J.SPRADO, E.KLIEN, C.SCHUBERT & CHRIST, I. 2009. Overcoming semantic heterogeneity in spatial data infrastructures. Computers &Geosciences, 35, 739-752.
  OGC 2012. OGC Standards. http://www.opengeospatial.org/standards: Open Geospatial Consortium.
  TIPA, G. & TIERNEY, L. 2003. A Cultural health index for streams and waterways : indicators for recognising and expressing Māori values. New Zealand. Ministry for the Environment.
  WHITE, P. A. 2006. Some Future Directions in Hydrology. Journal of Hydrology (NZ), 45, 63-68

Geospatial web-enablement for environmental data in New Zealand

This blog post can be seen as a sequel to a former blog post on the introduction on geospatial data sharing and spatial data infrastructures (SDI), where I explained the basics of OGC standards and web services. Quite some research organisations and governmental agencies already employ OGC standards to make data available online, often even free of charge for the public. I would like to present some really good examples of interoperable data sharing in New Zealand.

Through the standardised and web-based access to so many data sources, not only traditional geographical processing and analysis (GIS) based research is made easier, but also complete new technical and methodological research possibilities arise.

LINZ - Land Information New Zealand

I would like to start with Land Information New Zealand (LINZ). LINZ, as a governmental body, has issued and maintains New Zealand’s geospatial strategy. LINZ runs the LINZ Data Service, which provide tons of NZ-related data sets, topography, maps, place names and much more, almost all of it is available under a NZ Creative Commons license. You can register for free, get an API key and use data directly through web, basically as long as you tell that it is LINZ data. LINZ provides standard OGC CSW, WMS and WFS web service interfaces.

http://data.linz.govt.nz

More news about the NZ geospatial strategy can be found on here.

DOC – Department of Conservation

Also the New Zealand Department of Conservation is going towards geospatial web services. It looks like they use ESRI software, which supports OGC standards to certain bit already, although ESRI (producer of the ArcGIS software) as a commercial closed-source software provider has been known to notoriously neglect open standards. However, the Shapefile format is open and besides ESRI REST services, the DOC Geoportal also allows for OGC-based access (CSW/ISO 19139 metadata for search and discovery and WMS/WFS for map/feature data access)

http://geoportal.doc.govt.nz

GNS Science

The Institute of Geological and Nuclear Sciences is one of the 9 New Zealand Crown Research Institutes (CRI), which conduct about one half publicly/governmentally funded and the other half commercial research projects and, together with the universities of course, can be seen as New Zealand’s main science and research providers, each claiming a particular scientific domains. GNS Science is New Zealand’s leading provider of Earth, geoscience and isotope research and the geological survey of New Zealand. GNS’s research topics also include volcanoes, earthquakes, geothermal features and groundwater.

GNS has published the 1:250 000 Geological Map of New Zealand (QMAP. It is also digitally accessible – GNS exports the QMAP as OGC WMS and WFS in the OGC format GeoSciML. An easy way and very interesting example for interoperability is to explore New Zealand’s geology is through the OneGeology project, which sources and displays such services from geological surveys from all over the world.

The GNS-EU collaborative SMART Acquifer Characterisation programme (SAC) also aims to connect OGC based data sources. Within the research aim “Data Synthesis and Visualisation” the SMART Data Portal aims to develop an integrated OGC framework for discovery, access, processing and visualisation of hydrogeological data.

http://data.gns.cri.nz/metadata/

NIWA – National Institute of Water and Atmospheric Research

NIWA NIWA, another CRI, has a strong reputation in climate, marine and marine ecosystem and biodiversity sciences. Whereas a lot of organisations and agencies make data available first and then (if at all) add more sophisticated search technology, NIWA started the other way round. They established a discovery portal - the Environmental Information Browser, which is basically a catalogue, where one can search by keywords, places, data and time. All the data NIWA has, will eventually be listed and can be queried and also harvested through the OGC CSW interface. Furthermore NIWA is also moving towards providing OGC web services to their data sets. One particular example has been a “Summer of eResearch” project and its progress documented on the eResearch website.

http://ei.niwa.co.nz

Landcare Research

Landcare Research is also a New Zealand Crown Research Institute and focuses on the management of terrestrial biodiversity and land resources in order to both protect and enhance the terrestrial environment. I have come across several Landcare projects on soils and land use data, where Landcare not only uses OGC standards, but also participates in the development and maturing of some of those standards. Like the former parties, Landcare runs a data or geoportal (LRIS), too, which can be accessed and queried through CSW, WMS and WFS web interfaces,

https://lris.scinfo.org.nz/

Landcare also hosts a dedicated soil map portal (S-map), which sources the digital soil information layers based on WMS. Furthermore they drive the development of a global soil map portal (http://www.globalsoilmap.net/), which under the hood, of course, uses OGC standards again. To enable international, comparable, interoperable soil data exchange Landcare participates in the development of a soil information standard.

Outlook

Regional councils are on the way, too. Many regional councils already make data accessible on their web sites. A quick investigation shows for example Environment Waikato, Horizons, HBRC, BOP or Environment Canterbury. However most of these data sources need to be accessed manually and/or do not provide a standardised interface. Not to speak of a generalised way to actually find them. In conjunction with the open data initiatives (Open and Transparent Government , Open New Zealand) and catalogues available ( government datasets online, Open Data Catalogue), there is massive potential to link diverse datasets, relate and analyse seemingly unrelated datasets and gain new insights, find and (re-use) data by type, time and location or just enable ubiquitous mobile access to the data you need. However, we might end up needing a catalogue for the catalogues, and of course a lot of existing data needs to be geo-located/geo-referenced, so that they could be found by location. There is still a way to go and definitely some more research necessary in that space.

Originally posted on the New Zealand eResearch website - 16/04/2013

A side note on geospatial data sharing and spatial data infrastructures (SDI)

This blog post is dedicated to provide a general overview over the field of geospatial and environmental data sharing. The term geospatial is actually tautologous: The prefix “geo” implies geography, which always relates things to each other based on their location, where nearer things are stronger related than things further away from each other (Tobler’s 1st law of geography). And the word “spatial” also means having an extent and location in a space. However “geospatial” nowadays is almost exclusively used in the field of digital data with geographical context. Therefore software that delivers, analyses, presents, processes, stores and retrieves is also often called geospatial software (having its origins in the good old GIS – Geographical Information Systems).

Part 1: GIS and GI Science intro

Research areas that work on the science behind GIS and spatial data, on the analytical methods, processes techniques, on ways of (standardised) spatial data exchange, effective and efficient storage and retrieval, are called GI Science, Geoinformatics, Geomatics, Geocomputation, Spatial Computation or Spatial Science … There is actually quite some discussion, if the necessity, even an entitlement for such a dedicated mixed branch of computer science and geography exists, a similar discussion when geography emerged as an accepted field of research (ref). On the other hand you’ll often find the quote that “80% of data has a spatial component” or something like that. Apparently I can get the source right anymore, therefore you might handle this with care. It might go back to the 1990s when GIS software for PCs wanted to get their feet into the market (gis lounge). However a lot of data in the geosciences have a spatial context and - a lot - of those data are needed for governmental agencies to manage land and water resources properly :smile: Agreed?

Part 2: The Opengeospatial Consortium, aka OpenGIS, aka OGC

“The Open Geospatial Consortium (OGC) is an international industry consortium of 479 companies, government agencies and universities participating in a consensus process to develop publicly available interface standards. OGC® Standards support interoperable solutions that ‘geo-enable’ the Web, wireless and location-based services and mainstream IT.”

The OGC standards framework provides means to build a spatial Data infrastructure (LINZ - Land Information New Zealand) – which is “the technology, policies, standards, and human resources necessary to acquire, process, store, distribute and improve the usability of geospatial data. (SDI) facilitates the connections between these important sources of information, and allows people to find and access them.” Quite some of the OGC standards and web services are also ISO international standards. There are interface and service descriptions on the one hand and data encodings/formats and conceptual data models on the other side. I will provide a short summary:

WMS – Web Mapping Service (ISO 19128 WMS v1.3.0)

Essentially provides (web) maps (as images like png or jpg) output data Geographiclly correct images, png, jpg, view or portrayal service

Major methods: GetCapabilties and GetMap

WFS – Web Feature Service (ISO 19142 WFS v2.0)

Provides an interface to access, query, store and retrieve vector “features”, aka discrete data – like in ESRI shapefiles. Data is accessible by their data schema, which can be soft-typed and values in schema fields and location queries are utilized. Output GML (ISO 19136), which is in a particular “domain-specific” XML schema. With WFS-T – transactional – there is also support to write back to the WFS server.

GetCapabilities GetFeature – get the data DescribeFeatureType – get schema

WCS – Web Coverage Service

Provides an interface to access, query and retrieve raster imagery and coverages, grids (aka “fields”) as in continuous data. eg NetCDF-CF, GeoTIFF, ArcGRID

GetCapabilities DescribeCoverage GetCoverage

CSW – Catalogue Service for Web (ISO 19115 CSW 2.0.2)

Provides an interface to access, query, store and retrieve metadata, aka data about the geospatial data, which is accessible through other geospatial webservices. Output is usually XML ISO 19139 metadata or Dublin core.

GetCapabilities GetRecords – find metadata record by search criteria GetRecordById – get one record by its unique id DescribeRecord – metadata type GetDomain – get range of values and/or keywords

SOS – Sensor Observation Service

Provides an interface to access, query, store and retrieve time-series based data that has been measured at locations, eg through sensors or field surveying/sampling. But the focus is to query on temporal and then on spatial or value comparison basis. Standard output formats are O&M, WaterML2.0 time-series and SensorML sensor/procedure metadata.

SOS is part of the sensor web enablement initiative (SWE) which advances to its version 2, where a lot of things become more flexible, but also moe complicated. I will write about that later ☺ SOS explicitly also describes a group of methods to insert sensor and observation data. This is handled through different profiles.

GetCapabilities GetObservation DescribeSensor

There are quite some commercial and Open Source software packages and tool kits available, for the desktop and server-based for the web that support or where explicitly written for OGC webservices and splendid Open Source resources in the web:

OSGeo Foundation: http://www.osgeo.org

(Spatially enabled) Databases:

• Postgresql/Postgis
• MySQL
• Oracle Database
• Microsoft SQL Server
• ESRI Geodatabase / ArcSDE
• SpatiaLite
• Rasdaman
• GeoCouch

Data Servers (store data in databases):

• Geoserver (WMS, WFS, WCS)
• Mapserver (WMS, WFS, SOS)
• 52°North SOS server
• Geonetwork (CSW)
• ESRI ArcIMS / ArcServer
• Thredds (WCS, netCDF)

Web mapping tool kits / frameworks (take data from data servers):

• openlayers
• Mapbender
• MapFish
• Geomajas
• Flash
• Silverlight

Desktop clients supporting (at least partially) OGC standards:

• QuantumGIS
• uDig
• ESRI ArcGIS
• Intergraph Geomedia

And on it goes … in the next weeks, I will write about New Zealand and international examples of OGC webservices implementations and SDIs for geospatial data publishing (where the data is basically public domain and needs to made accessible) and provide some closer insights to OGC SWE and the next generation sensor networks initiative SWE 2.0 et al.

Designing the big picture

Designing the big picture

After literature review and further research and gathering a lot of information about state-of-the-art technology and paradigms, I designed a bigger concept of a possible, full-fledged WebGIS that is more than capable to support the data-related and computational needs of the research of the SMART aquifer characterisation program. after writing it down, we already got a paper to submit to the GI_Forum Geoinformatics conference.

.

SMART Portal archtitecture concept

As shown in the picture, the whole portal / WebGIS is based on standardised interfaces, mainly the OGC geo-spatial web services. as a requirement from the New Zealand Ministry of Science and Innovation all software produced (and used) should be Open Source, the main building blocks for the portal will be tested and evaluated in the prototype, I am going to develop:

• PostGres/PostGis - data storage, spatial database
• Geoserver - for WMS and WFS (subsequently WCS, too)
• OpenLayers as web mapping client
• 52North SOS Server, customise DAO backend to serve data from an existing legacy DB
• Thredds server to serve netCDF coverage data
• X3D as 3D file format to display 3D geological model in the browser
• Geonetwork - CSW, metadata, to connect to upstream NZ spatial data infrastructure