A Python Pandas Coding Meetup

On October, 12th, I organised a local meetup event at the Department of Geography. We had around 15 attendees, both from Tartu industry and university students.

In this meetup we played with the Jupyter Notebook (http://jupyter.org/) and did some data wrangling with Python, Pandas (http://pandas.pydata.org/) and co., some data analysis and visualisation.

I presented on how we can use Miniconda, an encapsulated versatile virtual python environment installer, that works under the hood of the big Anaconda python distribution. Miniconda is basically a mini version of Anaconda that includes only the conda package manager and its dependencies. We used a Python version 3.4 because some of the libraries/tools we wanted to work with supposedly only are supported up to Python 3.4. Here it becomes obvious how practical virtual environments can be. They help you to keep various Python versions around without messing up your system.

After we had set-up our local Python working environment, I introduced the Jupyter Notebook. The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

I prepared a variety of resources that we worked through in the meetup. I shared the collated tutorials and course materials on my GitHub account, The repository contains a curated collection of Jupyter Notebooks of introductory materials about programming in Python in general, the Pandas data manipulation and visualisation toolkit, and the Geopandas geospatial library.

This repository aims at different degrees of experience with the Python programming language and Python data manipultation and visualisation libraries. Each folder contains a README.md (for the the online repository) and README.notebook.md (readme as readable Notebook) file that list the contents and some details for each of the 3 big modules here:

• 01 - Introduction to Python and the Jupyter Notebook
• 02 - Introduction to the Pandas library
• 03 - Introduction to the Geopandas library

In order to get started, download the full Meetup-Repositiry from https://github.com/allixender/meetup-notes/archive/master.zip and extract it to a LOCAL folder. Go back into the console/commandline prompt and change into that folder. There you should then start the Jupyter notebook.

A few Impressions of the ESTGIS conference of the Estonian Society for Geoinformatics

On the ‎29. ‎September ‎2017  geographers and GIS specialists of the Estonian Landboard, Estonian GIS companies, and Estonian Universities - especially from University of Tartu Department of Geography and environmental Technologies, Tartu University of Life Sciences and Tallinn Technical University - joined at the annual gathering of the Estonian Society of Geoinformatics (EstGIS). Our Head of Chair of Geoinformatics at Dept. of Geography, Tartu, Prof Tönu Oja, is also the chairman of the boards of EstGIS.

The conference took place in the Vooremaa district North of Tartu.

The 2-day conference took place in a small village north of Tartu in small family-run holiday park. Embedded in idyllic country-side the place had accommodation, sauna of course, and a large conferencing room.

"Think Spatially" says the big flyer outside, welcoming the guests. 

Beautifully embedded in the landscape.

I presented about the work on a paper which was originating from my PhD, because I just arrived here, on making environmental research articles discoverable with Open Geospatial Consortium (OGC) catalogue services. The paper would eventually be finalised and published on MDPI International Journal of Geo-Information together with a colleague from the Department of Geography where I just had started:  "Enhancing Location-Related Hydrogeological Knowledge" (https://www.mdpi.com/2220-9964/7/4/132), on geo-referencing hydrological and hydrogeological journal articles and making them discoverable spatially through  OGC Catalogue Service for Web (CSW).

The conference talks were mixed in English and Estonian. There were a few international guests and a few students, but mainly local (Estonian) GIS practitioners.

More details about the article: https://allixender.blogspot.com/2018/04/making-environmental-research-articles.html

PhD Dissertation publicly accssible

My dissertation to achieve the degree 'Doctor of Philosophy' at the Auckland University of Technology (AUT) has the title:

A Context-based Groundwater Data Infrastructure

Online access via the AUT Online Library: http://hdl.handle.net/10292/10740

Abstract

Groundwater bodies are among the most important and valuable natural resources available, but at the same time they are also the least understood. To better understand the hydrological state of the environment and groundwater dynamics, data sets and measurements need to be made available and accessible to scientists, planners, and stakeholders to allow for proper decision making support. Information exchange via the internet has become faster, but at the same time data sets remain scattered both in location and formats. Present research in hydrogeology and freshwater resources management can be significantly supported and accelerated by relating, reusing and combining existing data sets, models and simulations in a streamlined, computer-aided and networked fashion and yield more new and reproducible insights.

In this thesis Design Science Research, Grounded Theory and Case Studies are applied in order to design a spatial data infrastructure that addresses the full data life cycle in the context of hydrogeology in New Zealand. This 'Hydrogeology Infrastructure' design was successfully implemented and evaluated via a networked and open standards-based prototype. Formerly disconnected and distributed data sets may now, for the first time, be used for hydrogeological data analysis, visualisation and modelling within one data portal.


My Supervisor(s)

Many thanks go out again to my supervisors who supported me continuously.

Assoc Prof Jacqueline Whalley (AUT, New Zealand); Assoc Prof Hermann Klug (Z_GIS, Salzburg University, Austria); Prof Philip Sallis (AUT, New Zealand)

The interested reader can find the publication also via ResearchGate:
https://www.researchgate.net/publication/319164590_A_Context-based_Groundwater_Data_Infrastructure

A Context-based #GroundwaterData Infrastructure #Hydrogeology #GIScience # SpatialDatahttps://t.co/QMpSKTpsp2

— Scholarly Commons (@AUT_SC) 17 August 2017

2017 LA Conference on diffuse pollution

It was an awkward conference - not knowing anyone, but also, really experiencing America first hand, in LA. Presenting went well, with barely connecting to the audience, a comparatively anonymous dinner.

UCLA has a pompous campus, it was nice weather, dry and warm. Comparing it to other conferences like EGU, or ANZ Hydrological conferences, Hydrogeology Association and Digital Earth summits, that conference was a strange experience.

OGC OWS GeoJSON

We are implementing OWC GeoJSON in the SAC groundwater portal as part of the SAC research program. In the latest OGC OWS Context GeoJSON Encoding Standard 14-055r2 we came across a few ambiguities that we couldn't solve for ourselves immediately.

I am summarising those subsequently here. The examples in the OGC Schemas repository are mostly still based on the former OWS-10 testbed results. Hut the OWC working group announced to update the official samples.

7.1.1 Class OWC:Context:

specReference: <xz>.properties.links.profiles requires the Specification Reference (requirements class) identifying that this is an OWC Context document and its version. Table 1 defines the data type as an Array of links profiles (as defined in Table 10 - data type "links") where one element SHALL have the href value “http://www.opengis.net/spec/owc-geojson/1.0/req/core”

But the multiplicity defines: One (mandatory) and the GeoJSON example 7.1.1.1 shows another different picture, which is only an array of String.

"links" : {
 "profiles" : [
  "http://www.opengis.net/spec/owc-geojson/1.0/req/core"
 ], ...
},

We implemented links.profiles as an array of links as objects of data type links from 7.1.10, table 10 with href, type, lang, title and length. And one mandatory link object element is required that has the specified core profile url.

Is that the desired behaviour?

Secondly, we found a quirk with the "<>.properties.links.via" description.

In 7.1.2 Class OWC:Resource "contextMetadata" is an Array of links as defined in Table 10 (Zero or more), and the example shows a JSON array. all good so far. However in OWC:Resource it is "resourceMetadata" also a "<>.properties.links.via" but explicitly call "Link object" (whereas links.data, links.previews and links.alternatives are Array of Link objects). But then in the column multiplicity it states "Zero or more" as with all the other link object arrays? But the example explicitly shows a single link object under the path "<>.properties.links.via":

"via" : {
"href" : "http://www.acme.com/products/algal20090123090856.xml",
"type" : "application/xml",
"length" : "435",
"title" : "XML metadata file for the entry 2009-01-23 09:08:56"
}

This is confusing. What is the intended behaviour? We chose an array of link objects to mimic the other link object arrays general behaviour. Is that appropriate?


Finally, there seems to be an exceptionally different behaviour for "7.1.1.9 creator". In OWC:Context there is only mentioned the path: <xz>.properties.creator

with a presumably wrong and irrelevant example:

"creator" : "ACME CSW Server”,

Because the abstract 7.1.7 DataType OWC:Creator is later typed into specific:

- owc:CreatorApplication - <xz>.properties.generator
  (as defined in Table 8)

- owc:CreatorDisplay - <xz>.properties.display
  (as defined in Table 9)

But the actual issue that I mean is that in 7.1.7 DataType OWC:Creator specifies at first a generic extension (without a JSON path, because it is abstract) - then extension for <xz>.properties.display.* in Table 9 - Definitions of owc:CreatorDisplay elements is explicitly described BUT COMPLETELY ABSENT for <xz>.properties.generator.* in Table 8 - Definitions of owc:Creator/OWC:CreatorApplication elements.

That seems either accidental or the reason is not obvious. We chose the freedom to allow for an extension in OWC:CreatorApplication the same way the extension is declared for ALL other data types.

If I can be of any help to improve the standard or provide material or examples, please don't hesitate to contact me. I would be happy to join an OWC Context working group if possible.


- Our GitHub repository: https://github.com/ZGIS/smart-owc-geojson
- OpenHub Open Source statistics: https://www.openhub.net/p/smart-owc-geojson
- Bintray Maven/Ivy artefacts download: https://bintray.com/allixender/ivy2/smart-owc-geojson

Looking forward to support the use of OGC OWC Context for data exchange in the geospatial web.

Alex

Hydrological Society Conference - Web-based real-time processing of environmental measurements

This case study was presented as poster abstract.

Kmoch, A., White, P. A., & Klug, H. (2015). Sensor Observation Service and web-based real-time Processing of environmental Measurements in the Upper Rangitaiki Catchment (Poster). In The NZ Hydrological Society Conference 2015, 26th November, in Hamilton, New Zealand

Abstract:

Environmental assessments naturally depend on field observations and technological advancements. , such as tTelemetry, allow the automated collection, transmission and processing of these measurements. However, modelling of natural processes is typically a complex challenge and involves applying expertise of scientists as well as a host of data preparation steps (White, 2006, White et al., 2003).
In addition, automation of model execution with the most recent observation data is dependent on the integration of the data collection, storage and processing elements (Klug and Kmoch, 2014). This paper demonstrates a system that integrates a Sensor Observation Service (SOS) that includinges field observations and internet-based environmental data with a rainfall recharge model that allows near-real time calculation of rainfall recharge in the Upper Rangitaiki catchment, Bay of Plenty region.
The SOS specification is an Open Geospatial Consortium (OGC) standard for the open and standardised integration of environmental sensors into an internet-based environmental data infrastructure (Klug and Kmoch, 2015, Klug, Kmoch, Reichel, 2015).

Figure 1. Process of data flow from field site sensors, to SOS data service to a simulation model process

Results:

• We showed that it is possible to link the collected data directly to a simple rainfall recharge model (Figure 1)
• The low cost sensor and circuit board instrumentation collects data and forwards them to the field computer in 10-minute intervals via robust, low power, ZigBee wireless protocol
• The field computer running a standard Linux operating system, transfers observation data in 10 minute intervals via a 3G mobile data connection to an online SOS server.
• From the service the observations are available in a standardised open format.
• A website can access the raw data from the SOS server and plotted data points within 5-10 minutes of field measurement
• A rainfall recharge model runs with the latest data points from the online SOS server.

References:

Klug, H., & Kmoch, A. (2014). A SMART groundwater portal: An OGC web services orchestration framework for hydrology to improve data access and visualisation in New Zealand. Computers & Geosciences, 69(0), 78–86. http://dx.doi.org/10.1016/j.cageo.2014.04.016

Klug, H., Kmoch, A. (2015). Operationalizing environmental indicators for real time multi-purpose decision making and action support. Ecological Modelling, 295, 66-74. http://dx.doi.org/10.1016/j.ecolmodel.2014.04.009.

White, P. A. (2006). Some Future Directions in Hydrology. Journal of Hydrology (NZ), 45(2), 63–68.

White, P. A., Hong, Y.-S., Murray, D. L., Scott, D. M., & Thorpe, H. R. (2003). Evaluation of regional models of rainfall recharge to groundwater by comparison with lysimeter measurements, Canterbury, New Zealand. Journal of Hydrology (NZ), 42(1), 39–64.

Klug, H., Kmoch, A., & Reichel, S. (2015). Adjusting the Frequency of Automated Phosphorus Measurements to Environmental Conditions. GI_Forum 2015 - Journal for Geographic Information Science - Geospatial Minds for Society, 1, 590–599. http://doi.org/10.1553/giscience2015s590

AGILE 2016 - SensorWeb Semantics on MQTT for responsive Rainfall Recharge Modelling

Integrating Wireless Sensor Networks (WSNs) and spatial data web services is becoming common in ecological applications. However, WSNs were developed in application domains with different sensor and user types, and often with their own low-level metadata semantics, data format and communication protocols. The sensor web enablement initiative (SWE) within the Open Geospatial Consortium (OGC) has released a set of open standards for interoperable interface specifications and (meta) data encodings for the real time integration of sensors and sensor networks into a web services architecture.
Such XML-based web services exhibit disadvantages in terms of payload and connectivity in low-bandwidth low energy unreliable networks, such as remote 3G uplinks. Monitoring stations deliver frequent measurements in real-time, but dynamic implementation of measurement frequencies, adapted to certain environmental conditions, are rarely implemented. We describe a responsive integrated hydrological monitoring prototype to calculate rainfall recharge for water management purposes.
When rainfall is observed, a threshold event triggers a reconfiguration task for the soil moisture sensors, using asynchronous, push-based communication implemented with an MQTT queue. A Sensor Planning Service commits that request via MQTT into the wireless sensor network, and updates the measurement frequency of the target sensors to gain higher resolution for the vertical soil water infiltration.
The system integrates a Sensor Observation Service (SOS) including field observations and internet-based environmental data with a rainfall recharge model that allows near-real time calculation of rainfall recharge in the Upper Rangitaiki catchment, Bay of Plenty region in New Zealand.

Figure 1: Setup and location of the sensor field site, central North Island, New Zealand

The prototype site comprises a main station conducting comprehensive measurements of meteorological, hydrological and pedological parameters. For the wireless data transmission within the local site installation XBee-PRO modules from the Digi Company  ZigBee IEEE 802.15.4 protocol are used. The main station receives continuous sensor measurements from the attached sensor units, and acts as the gateway to the online SOS and SPS services by providing the communication channel from the local sensor network to the web-enabled data management infrastructure.

The field site has been established in the Upper Rangitaiki catchment (Figure 1) and comprises a field computer (Raspberry Pi) with a direct internet link (GPRS/3G) and a sensor board (Waspmote) that has 12 typical meteorological, hydrological and pedological sensors attached (i.e., wind speed, wind direction, rainfall, 1x groundwater probe, 5x temperature and 3x soil moisture). The Raspberry Pi and Waspmote can be monitored and reprogrammed from an online server.

Figure 2: Raw sensor series visualized in a website from a SOS query.

The site setup allows scaling up to a multitude of low cost, low energy sensor stations throughout the catchment, with only one field computer that serves as data logger for backup. The observations were available in a standardized open format. The website accessed the raw data from the SOS server and plotted data points within 5-10 minutes of field measurement. This website was easily accessible via browsers and smartphones (Figure 2).

The paper was was presented at the 19th AGILE International Conference on Geographic Information Science, 15th of June, in  Helsinki, Finlkand.

Kmoch, A., Klug, H., White, P., & Reichel, S. (2016). SensorWeb Semantics on MQTT for responsive Rainfall Recharge Modelling. In 19th AGILE International Conference on Geographic Information Science. Helsinki.