A Python Pandas Coding Meetup

On October, 12th, I organised a local meetup event at the Department of Geography. We had around 15 attendees, both from Tartu industry and university students.

In this meetup we played with the Jupyter Notebook (http://jupyter.org/) and did some data wrangling with Python, Pandas (http://pandas.pydata.org/) and co., some data analysis and visualisation.

I presented on how we can use Miniconda, an encapsulated versatile virtual python environment installer, that works under the hood of the big Anaconda python distribution. Miniconda is basically a mini version of Anaconda that includes only the conda package manager and its dependencies. We used a Python version 3.4 because some of the libraries/tools we wanted to work with supposedly only are supported up to Python 3.4. Here it becomes obvious how practical virtual environments can be. They help you to keep various Python versions around without messing up your system.

After we had set-up our local Python working environment, I introduced the Jupyter Notebook. The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

I prepared a variety of resources that we worked through in the meetup. I shared the collated tutorials and course materials on my GitHub account, The repository contains a curated collection of Jupyter Notebooks of introductory materials about programming in Python in general, the Pandas data manipulation and visualisation toolkit, and the Geopandas geospatial library.

This repository aims at different degrees of experience with the Python programming language and Python data manipultation and visualisation libraries. Each folder contains a README.md (for the the online repository) and README.notebook.md (readme as readable Notebook) file that list the contents and some details for each of the 3 big modules here:

• 01 - Introduction to Python and the Jupyter Notebook
• 02 - Introduction to the Pandas library
• 03 - Introduction to the Geopandas library

In order to get started, download the full Meetup-Repositiry from https://github.com/allixender/meetup-notes/archive/master.zip and extract it to a LOCAL folder. Go back into the console/commandline prompt and change into that folder. There you should then start the Jupyter notebook.

OGC OWS GeoJSON

We are implementing OWC GeoJSON in the SAC groundwater portal as part of the SAC research program. In the latest OGC OWS Context GeoJSON Encoding Standard 14-055r2 we came across a few ambiguities that we couldn't solve for ourselves immediately.

I am summarising those subsequently here. The examples in the OGC Schemas repository are mostly still based on the former OWS-10 testbed results. Hut the OWC working group announced to update the official samples.

7.1.1 Class OWC:Context:

specReference: <xz>.properties.links.profiles requires the Specification Reference (requirements class) identifying that this is an OWC Context document and its version. Table 1 defines the data type as an Array of links profiles (as defined in Table 10 - data type "links") where one element SHALL have the href value “http://www.opengis.net/spec/owc-geojson/1.0/req/core”

But the multiplicity defines: One (mandatory) and the GeoJSON example 7.1.1.1 shows another different picture, which is only an array of String.

"links" : {
 "profiles" : [
  "http://www.opengis.net/spec/owc-geojson/1.0/req/core"
 ], ...
},

We implemented links.profiles as an array of links as objects of data type links from 7.1.10, table 10 with href, type, lang, title and length. And one mandatory link object element is required that has the specified core profile url.

Is that the desired behaviour?

Secondly, we found a quirk with the "<>.properties.links.via" description.

In 7.1.2 Class OWC:Resource "contextMetadata" is an Array of links as defined in Table 10 (Zero or more), and the example shows a JSON array. all good so far. However in OWC:Resource it is "resourceMetadata" also a "<>.properties.links.via" but explicitly call "Link object" (whereas links.data, links.previews and links.alternatives are Array of Link objects). But then in the column multiplicity it states "Zero or more" as with all the other link object arrays? But the example explicitly shows a single link object under the path "<>.properties.links.via":

"via" : {
"href" : "http://www.acme.com/products/algal20090123090856.xml",
"type" : "application/xml",
"length" : "435",
"title" : "XML metadata file for the entry 2009-01-23 09:08:56"
}

This is confusing. What is the intended behaviour? We chose an array of link objects to mimic the other link object arrays general behaviour. Is that appropriate?


Finally, there seems to be an exceptionally different behaviour for "7.1.1.9 creator". In OWC:Context there is only mentioned the path: <xz>.properties.creator

with a presumably wrong and irrelevant example:

"creator" : "ACME CSW Server”,

Because the abstract 7.1.7 DataType OWC:Creator is later typed into specific:

- owc:CreatorApplication - <xz>.properties.generator
  (as defined in Table 8)

- owc:CreatorDisplay - <xz>.properties.display
  (as defined in Table 9)

But the actual issue that I mean is that in 7.1.7 DataType OWC:Creator specifies at first a generic extension (without a JSON path, because it is abstract) - then extension for <xz>.properties.display.* in Table 9 - Definitions of owc:CreatorDisplay elements is explicitly described BUT COMPLETELY ABSENT for <xz>.properties.generator.* in Table 8 - Definitions of owc:Creator/OWC:CreatorApplication elements.

That seems either accidental or the reason is not obvious. We chose the freedom to allow for an extension in OWC:CreatorApplication the same way the extension is declared for ALL other data types.

If I can be of any help to improve the standard or provide material or examples, please don't hesitate to contact me. I would be happy to join an OWC Context working group if possible.


- Our GitHub repository: https://github.com/ZGIS/smart-owc-geojson
- OpenHub Open Source statistics: https://www.openhub.net/p/smart-owc-geojson
- Bintray Maven/Ivy artefacts download: https://bintray.com/allixender/ivy2/smart-owc-geojson

Looking forward to support the use of OGC OWC Context for data exchange in the geospatial web.

Alex