Info Session for MSc programme in Geoinformatics for Urbanised Society

Today we broadcast a webinar info session about the new MSc programme in Geoinformatics for Urbanised Society at the University of Tartu.

One of the recurring core topics in this new MSc curriculum is Data and GIS use in Urban Planning.

Until just a decade ago spatial planning and analytics projects had problems with getting enough data. But nowadays there is so much data available, that it is increasingly hard to make sense of it – because of the 3 V’s of big data - volume, variety, velocity. The open data movement, government agencies, research institutes and citizen scientists alike make more and more data available publicly, mobile phones, sensor networks and satellites generate a multitude of datasets every day

In order solve the Interdisciplinary challenges of urban planning we want to empower you with skills and knowledge to analyse, visualise and understand processes and data. For that we teach the Full cycle of spatial data management, from the various methods of data acquisition, followed by efficient and practical processing techniques, to subsequent meaningful analysis and visualisation; in order to consequently make successful planning decisions for a sustainable future.

So what does it mean to study Geoinformatics for Urbanised Society with us in Tartu?

You will learn how to combine geography and IT in the age of BIG data. This is essentially what we believe modern Geoinformatics is representing. Mastering Geoinformatics will provide you with tools to analyse social and natural processes in space for interdisciplinary decision- and policy-making.

Watch the whole recorded session for more info:

Links:

• https://www.ut.ee/gis 
• https://www.facebook.com/GISTartu

A Python Pandas Coding Meetup

On October, 12th, I organised a local meetup event at the Department of Geography. We had around 15 attendees, both from Tartu industry and university students.

In this meetup we played with the Jupyter Notebook (http://jupyter.org/) and did some data wrangling with Python, Pandas (http://pandas.pydata.org/) and co., some data analysis and visualisation.

I presented on how we can use Miniconda, an encapsulated versatile virtual python environment installer, that works under the hood of the big Anaconda python distribution. Miniconda is basically a mini version of Anaconda that includes only the conda package manager and its dependencies. We used a Python version 3.4 because some of the libraries/tools we wanted to work with supposedly only are supported up to Python 3.4. Here it becomes obvious how practical virtual environments can be. They help you to keep various Python versions around without messing up your system.

After we had set-up our local Python working environment, I introduced the Jupyter Notebook. The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

I prepared a variety of resources that we worked through in the meetup. I shared the collated tutorials and course materials on my GitHub account, The repository contains a curated collection of Jupyter Notebooks of introductory materials about programming in Python in general, the Pandas data manipulation and visualisation toolkit, and the Geopandas geospatial library.

This repository aims at different degrees of experience with the Python programming language and Python data manipultation and visualisation libraries. Each folder contains a README.md (for the the online repository) and README.notebook.md (readme as readable Notebook) file that list the contents and some details for each of the 3 big modules here:

• 01 - Introduction to Python and the Jupyter Notebook
• 02 - Introduction to the Pandas library
• 03 - Introduction to the Geopandas library

In order to get started, download the full Meetup-Repositiry from https://github.com/allixender/meetup-notes/archive/master.zip and extract it to a LOCAL folder. Go back into the console/commandline prompt and change into that folder. There you should then start the Jupyter notebook.

SMART Groundwater Portal Dev going full "Cloud"

What a concise quote from Erik Dietrich, founder of DaedTech LLC:

Software developers demand the ability to work effectively from anywhere.  They have attained a coolness factor, and demand for them is so high that there is no need for them to guard their source code like squirrels preparing for winter.  GitHub is a good idea because it effectively captured what software developers really want and offered it to them pretty flawlessly.  GitHub is a zeitgeist that is taking over the world precisely because software developers are taking over the world and software developers really like GitHub. (source)

Although, I work at a research institute which is one half a commercial consultancy with IP to protect, on-going international research collaborations, and governmental research funding require us to be flexible, open and accessible.  I work as a research scientist/analyst programmer in a science department, not in the IT or applications department, and thus, IT infrastructure interaction in commercial entities is "challenging" - for the scientists as well as for the IT folks.

In our current project we embraced the Zeitgeist now, too. For our geodata portal development and deployment processes, we adopted following paradigm:

Google Cloud Platform, Compute and Container Engine with Kubernetes as the our computational platform.

Google Drive, Google Docs and Sheets for assets, functional and implementation specifications development, user stories and use cases.

GitHub as our distributed version control system, which allows us to collaborate, yet, keep contributions transparent and easily and publicly traceable.

Trello eventually serves as our workflow board, for sprints, keeping links of specs, repos and other soft information together. I believe, that we could have done everything in GitHub, but Google Docs and Trello provide gently mechanisms to also invite non-technical folks to contribute. And actually, what we really want is this, right?

Geoscience Data Mining and Visualisation Brainstorming Weekend

Early 2016 New Zealand Ministry for Business, Innovation and Employment (MBIE) have put out calls for interested parties for a business to govt (B2G) data innovation challenge, one "opportunity" in particular is about "Geoscience Data management" - thought that I have some strengths and competency to contribute and was keen to have a say.

The Challenge: 

This so called R9 Accelerator brings the public and private sectors together to make it easier for business to interact with New Zealand government.

http://www.r9accelerator.co.nz/opportunities/opportunity14/

A prototype model could be applied to a range of other large databases managed by government, businesses and science institutions across the country. Data management issues are common internationally, so the model could have applications overseas.

On 29th Jan to 31st Jan 2016 was a full weekend information workshop in Miramar, Wellington. Subsequently, if interested, one would have to apply for the 3 months accelerator programme (either as team member or as mentor/domain expert).

http://www.r9accelerator.co.nz/apply/

http://www.r9accelerator.co.nz/timeline/

If teams would get elected to the full programme they would take part in the three month Accelerator starting 1st of March, then pitch to investors. This could be an opportunity to learn how private sector could better interact with govt data.

Alternatively, one could to consider to be involved on higher level into the process, which would be sort of part-time mentorship govt navigator or domain expert type participation.

http://www.r9accelerator.co.nz/take-part/support-a-team/

http://www.r9accelerator.co.nz/take-part/invest-in-a-team/

The plan was to show up and try to form a team and develop an idea and basic plan over the weekend, which will then be pitched on Sunday in a two minute presentation. This apparently would have the most impact on a team's chance of being accepted into the 3 months intense programme, where a real prototype is supposed to be developed by the team. The official application via an online form is then only a formal act to be completed subsequently for an already consistent and focussed team from the weekend).

The Geoscience Data Management opportunity was only one out of 14 or 15, and it's not obvious how many teams tackle each opportunity and how many applications are thought to go forward.

The Team Brainstorming Weekend:

There was a wild crowd of young and old, but only the team around the geodata challenge seemed to be high profile.

Katalyst / KDM Spectrum Data from Australia, Schlumberger, and the NZ agencies MBIE, LINZ, NIWA, GNS (Guy Maslen / Globe Claritas) had representatives there. So we were locked away over the weekend to brainstorm ideas to address MBIE's and NZPM immediate problem of nicer representation/delivery/visualisation of prospectivity data for possible investors in oil&gas and minerals.

From Dave Darby pitching the challenge...

WELLINGTON, NEW ZEALAND - January 29: R9 Accelerator Day 1: January 29, 2016 in Wellington, New Zealand. (Photo by Mark Tantrum/ http://mark tantrum.com, COPYRIGHT:2016 Mark Tantrum)

over group discussions...

WELLINGTON, NEW ZEALAND - January 30: R9 Accelerator Day 2. January 30, 2016 in Wellington, New Zealand. (Photo by Elias Rodriguez/ eliasrodriguez.co.nz) COPYRIGHT:2015 Elias Rodriguez

WELLINGTON, NEW ZEALAND - January 30: R9 Accelerator Day 2. January 30, 2016 in Wellington, New Zealand. (Photo by Elias Rodriguez/ eliasrodriguez.co.nz) COPYRIGHT:2015 Elias Rodriguez

.. toward the final pitch of what a team could possibly achieve if funded (respectively participate in this accelerator program):

WELLINGTON, NEW ZEALAND - January 31: R9 Accelerator Day 3. January 31, 2016 in Wellington, New Zealand. (Photo by Elias Rodriguez/ eliasrodriguez.co.nz) COPYRIGHT:2015 Elias Rodriguez

The (preliminary) Summary:

It could have been a great set-up for creating specific start-up type business solutions for MBIE across their departments.

We came up with designated/suggested team members , e.g. Guy Holmes and Tony Duffy(KDM Spectrum Data), Marielle Lange, a developer, Gavin Chapman, geodata management team at MBIE and I. We also suggested an advisory group, as far as I get it together: Dave Darby (MBIE), James Johnson (MBIE), Richard Garlick (MBIE), Jochen Schmidt (NIWA), Guy Maslen (GNS / Globe Claritas), Grep Byrom (LINZ).

If the proposal would have been accepted then the team would have had to develop a prototype with a little funding type stipend, and present that prototype to MBIE and other possible investors by June. Based on that further commercialization/contracting may arise. However, the professional team members were mainly supposed to support themselves (presumably KDM as big business, MBIE seconding their participant), and few of us would have to go full in and see if we'd be eligible for a part of the team stipend to basically live the start-up work life for the coming three months.

However, while the team, the idea, and the pitch were great, my situation would of course complicate my personal setting my PhD and within SMART programme. Eventually, I had to make a decision and withdrew to wrap up my PhD first. After all, it was a great opportunity to meet fascinating people and talk about possibly disruptive ways of re-shaping geoscience data management, visualization at governmental and even global scale.

Environment Southland Information Management Conference

Regional councils and government agencies are increasingly under pressure to resolve data questions and discover how best to acquire, manage, collate, analyse, report and disseminate data, while managing  the associated costs. Steering organisations through these complex issues requires a solid understanding of what technologies are available and the information demands of the future *(source).

I had the great opportunity to speak at the Environment Southland Information Management Conference in Invercargill. It was a great event, well organised and very informative. I believe I could contribute my part to the line up and fill a few more gaps in the whole picture.
This was not a business as usual conference, it was obvious that the speakers took it serious to cater their presentations to the needs of the stakeholders. And with 70 attendees from regional and central government, as well as visitors from research and industry.

It was great to see the emerging patterns around NZ and similar approaches to a holistic, comprehensive and modern data strategy. If you are interested, this is a link to programme, and please see below for my slides. Watch the talk on Youtube.

2014 NASA International Space Apps Challenge - TETRIS GroupSpace Apps Challenge 2014 - TETRIS Space Suit HUD demo with a Raspberry

As part of the 2014 NASA International Space Apps Challenge I joined an Auckland AUT University Textile Lab group to support their space suit concept idea with some cool wireless video and sensor streaming based on Raspberry Pi.

TETRIS (Terran Expeditionary Technologically Radical Instrumental Suit) is designing a suit, aimed for use by astronauts in space, which will make the astronaut’s work easier and safer to perform. It achieves this by embedding many of the new technologies being released in the recent years into the suit itself in order to: - monitor the astronaut’s life signs and condition via biometric sensors, such as a galvanic sensor for emotions, pulse sensor for the heart rate - allow simple integrated communication via a shoulder mounted camera and microphone - display biometric information to user via projected HUD - take simple electrical readings (when testing/diagnosing equipment) via embedded measurement tools (threaded conductive fabric in gloves) These hardware devices all interface with a RaspberryPi, being used as a central control unit in our design, as well the transceiver for all wireless communications between the astronaut and their remote operations control room.

Link to the 2014 Space Apps project website

The Raspberry Pi video camera live capture was streamed into the remote browser with an overlay of the HUD mockup.

Sources here https://github.com/TetrisGroup/spaceapps

Exposing geographic information with Geoserver and OpenLayers

Exposing geographic information

Visualising geographic data especially on a map has been scientific subject for ages :-) But one of the basics is actually still colouring and labeling features as means of symbolisation to recognise and distinguish geographic features and to compare specific attribute values.

The OpenGIS® Styled Layer Descriptor (SLD) Profile of the OpenGIS® Web Map Service (WMS) Encoding Standard [http://www.opengeospatial.org/standards/wms] defines an encoding that extends the WMS standard to allow user-defined symbolization and coloring of geographic feature[http://www.opengeospatial.org/ogc/glossary/f] and coverage[http://www.opengeospatial.org/ogc/glossary/c] data. SLD addresses the need for users and software to be able to control the visual portrayal of the geospatial data. The ability to define styling rules requires a styling language that the client and server can both understand. The OpenGIS® Symbology Encoding Standard (SE) [http://www.opengeospatial.org/standards/symbol] provides this language, while the SLD profile of WMS enables application of SE to WMS layers using extensions of WMS operations. Additionally, SLD defines an operation for standardized access to legend symbols.  (http://www.opengeospatial.org/standards/sld)

I would like to show three nice examples I used in the SMART web mapping application. The first is a a point style - a neat triangle with a label, the 2nd is colouring of contour lines and labeling and the use of Geoserver's FeatureInfo template system.

The first example is a point feature, a State of the Environment monitoring well in New Zealand. Besides other data fields It has an ID, which is actually an officially assigned number.

<?xml version="1.0" encoding="UTF-8"?>
<StyledLayerDescriptor version="1.0.0"
  xsi:schemaLocation="http://www.opengis.net/sld http://schemas.opengis.net/sld/1.0.0/StyledLayerDescriptor.xsd" xmlns="http://www.opengis.net/sld"
  xmlns:ogc="http://www.opengis.net/ogc" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <NamedLayer>
    <Name>soe_gwl_monitoring_wells</Name>
    <UserStyle>
      <Name>soe_gwl_monitoring_wells:withLabel</Name>
      <Title>withLabel</Title>
      <FeatureTypeStyle>
        <Name>triangleWithLabel</Name>
        <Rule>
          <Name>triangleWithLabel</Name>
          <MinScaleDenominator>0</MinScaleDenominator>
          <MaxScaleDenominator>9999999</MaxScaleDenominator>
          <TextSymbolizer>
            <Label>
              <ogc:PropertyName>ID</ogc:PropertyName>
            </Label>
            <Font>
              <CssParameter name="font-family">Sans-Serif</CssParameter>
              <CssParameter name="font-style">italic</CssParameter>
              <CssParameter name="font-size">10</CssParameter>
              <CssParameter name="font-color">#000000</CssParameter>
            </Font>
            <LabelPlacement>
              <PointPlacement>
                <AnchorPoint>
                  <AnchorPointX>
                    <ogc:Literal>0.0</ogc:Literal>
                  </AnchorPointX>
                  <AnchorPointY>
                    <ogc:Literal>0.0</ogc:Literal>
                  </AnchorPointY>
                </AnchorPoint>
                <Displacement>
                  <DisplacementX>
                    <ogc:Literal>2.0</ogc:Literal>
                  </DisplacementX>
                  <DisplacementY>
                    <ogc:Literal>2.0</ogc:Literal>
                  </DisplacementY>
                </Displacement>
                <Rotation>
                  <ogc:Literal>0.0</ogc:Literal>
                </Rotation>
              </PointPlacement>
            </LabelPlacement>
            <Halo>
              <Fill>
                <CssParameter name="fill">#FF4000</CssParameter>
                <CssParameter name="fill-opacity">0.3</CssParameter>
              </Fill>
            </Halo>
            <Fill>
              <CssParameter name="fill">#000000</CssParameter>
            </Fill>
          </TextSymbolizer>
          <PointSymbolizer>
            <Graphic>
              <Mark>
                <WellKnownName>triangle</WellKnownName>
                <Fill>
                  <CssParameter name="fill">
                    <ogc:Literal>#FF4000</ogc:Literal>
                  </CssParameter>
                </Fill>
              </Mark>
              <Opacity>
                <ogc:Literal>1.0</ogc:Literal>
              </Opacity>
              <Size>
                <ogc:Literal>10</ogc:Literal>
              </Size>
            </Graphic>
          </PointSymbolizer>
        </Rule>
      </FeatureTypeStyle>
    </UserStyle>
  </NamedLayer>
</StyledLayerDescriptor>

 The main things I do here, are essentially defining the text representation and then the point symbolisation by giving it the shape of small orange triangle with no surrounding line. For the text symbolisation I define, what data field shall the label represent, the  font, the label placement in relation to the actual point and I use the halo-tag to give the font I nice colourful background "glow" :-)

The next is a contour line clouring description. The contour lines represent mean annual rainfall and regarding the data sets I used a Jenks classification with 5 classes, which I hard-coded in the SLD document.

<?xml version="1.0" encoding="ISO-8859-1"?>
<StyledLayerDescriptor version="1.0.0" xsi:schemaLocation="http://www.opengis.net/sld StyledLayerDescriptor.xsd" xmlns="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <!-- a Named Layer is the basic building block of an SLD document -->
  <NamedLayer>
    <Name>horowhenua_mean_rainfall_contours</Name>
    <UserStyle>
      <Name>horowhenua_mean_rainfall_contours:withContourLabel</Name>
      <Title>withCountourLabels</Title>
      <Abstract>withCountourLabels</Abstract>
      <FeatureTypeStyle>
        <Name>withCountourLabels</Name>
        <Title>withCountourLabels</Title>
<!-- 5 breaks natural jenks, 900-1200, 1201-1700, 1701-2200, 2201-2700, 2701-3200 -->
        <Rule>
          <Name>900-1200</Name>
          <Title>900-1200</Title>
          <Abstract>900-1200</Abstract>
          <ogc:Filter>
            <ogc:PropertyIsLessThanOrEqualTo>
              <ogc:PropertyName>CONTOUR</ogc:PropertyName>
              <ogc:Literal>1200</ogc:Literal>
            </ogc:PropertyIsLessThanOrEqualTo>
          </ogc:Filter>
          <MinScaleDenominator>0</MinScaleDenominator>
          <MaxScaleDenominator>9999999</MaxScaleDenominator>
          <TextSymbolizer>
            <Label>
              <ogc:PropertyName>CONTOUR</ogc:PropertyName>
            </Label>
            <LabelPlacement>
              <LinePlacement />
            </LabelPlacement>
            <Halo>
              <Fill>
                <CssParameter name="fill">#CEF6F5</CssParameter>
                <CssParameter name="fill-opacity">0.6</CssParameter>
              </Fill>
            </Halo>
            <Fill>
              <CssParameter name="fill">#0B0B3B</CssParameter>
            </Fill>
            <VendorOption name="followLine">true</VendorOption>
          </TextSymbolizer>
          <LineSymbolizer>
            <Stroke>
              <CssParameter name="stroke">#2ECCFA</CssParameter>
              <CssParameter name="stroke-opacity">0.7</CssParameter>
              <CssParameter name="stroke-width">
                <ogc:Literal>3</ogc:Literal>
              </CssParameter>
            </Stroke>
          </LineSymbolizer>
        </Rule>
<!-- 5 breaks natural jenks, 900-1200, 1201-1700, 1701-2200, 2201-2700, 2701-3200 -->
        <Rule>
          <Name>1201-1700</Name>
          <Title>1201-1700</Title>
          <Abstract>1201-1700</Abstract>
          <ogc:Filter>
            <ogc:And>
              <ogc:PropertyIsGreaterThan>
                <ogc:PropertyName>CONTOUR</ogc:PropertyName>
                <ogc:Literal>1201</ogc:Literal>
              </ogc:PropertyIsGreaterThan>
              <ogc:PropertyIsLessThanOrEqualTo>
                <ogc:PropertyName>CONTOUR</ogc:PropertyName>
                <ogc:Literal>1700</ogc:Literal>
              </ogc:PropertyIsLessThanOrEqualTo>
            </ogc:And>
          </ogc:Filter>
          <MinScaleDenominator>0</MinScaleDenominator>
          <MaxScaleDenominator>9999999</MaxScaleDenominator>
          <TextSymbolizer>
            <Label>
              <ogc:PropertyName>CONTOUR</ogc:PropertyName>
            </Label>
            <LabelPlacement>
              <LinePlacement />
            </LabelPlacement>
            <Halo>
              <Fill>
                <CssParameter name="fill">#CEF6F5</CssParameter>
                <CssParameter name="fill-opacity">0.6</CssParameter>
              </Fill>
            </Halo>
            <Fill>
              <CssParameter name="fill">#0B0B3B</CssParameter>
            </Fill>
            <VendorOption name="followLine">true</VendorOption>
          </TextSymbolizer>
          <LineSymbolizer>
            <Stroke>
              <CssParameter name="stroke">#2E9AFE</CssParameter>
              <CssParameter name="stroke-opacity">0.7</CssParameter>
              <CssParameter name="stroke-width">
                <ogc:Literal>3</ogc:Literal>
              </CssParameter>
            </Stroke>
          </LineSymbolizer>
        </Rule>
<!-- 5 breaks natural jenks, 900-1200, 1201-1700, 1701-2200, 2201-2700, 2701-3200 -->
        <Rule>
          <Name>1701-2200</Name>
          <Title>1701-2200</Title>
          <Abstract>1701-2200</Abstract>
          ...
<!-- 5 breaks natural jenks, 900-1200, 1201-1700, 1701-2200, 2201-2700, 2701-3200 -->
        <Rule>
          <Name>2201-2700</Name>
          <Title>2201-2700</Title>
          <Abstract>2201-2700</Abstract>
          ...
<!-- 5 breaks natural jenks, 900-1200, 1201-1700, 1701-2200, 2201-2700, 2701-3200 -->
        <Rule>
          <Name>2701-3200</Name>
          <Title>2701-3200</Title>
          <Abstract>2701-3200</Abstract>
          ...
        </Rule>
      </FeatureTypeStyle>
    </UserStyle>
  </NamedLayer>
</StyledLayerDescriptor>

I cut the last three declarations, as anybody might derive that from the first two ones, I think :-) Here I used the OGC filter encoding specification (http://www.opengeospatial.org/standards/filter) to define that only these lines, which have values of a specific range in the named attribute. Additionally the vendor-option <VendorOption name="followLine">true</VendorOption> followLine aligns the contour's elevation labels along the contour lines :-). A nice fact is that if you use not only the name-tag within the rule, but also title and/or abstract (I am actually not 100% sure :-p ) that naming will be represented in the WMS layer legend.

Finally I would like to shortly introduce my first experiences with Geoserver's freemarker template engine (http://docs.geoserver.org/latest/en/user/tutorials/GetFeatureInfo/index.html and http://geoserver.org/display/GEOS/Templates) to customise featureInfo output with OpenLayers. The first code example is a pretty basic template that fetches attribute names and values from the Geoserver provided featureCollection and then fetches a hydrograph picture based on the ID of the current feature, to demonstrate the flexibilty. In future developments  a bit more dynamic functionality like rendering such a graph online by requesting the time-series from a SOS server would way cooler :-) 

<#--

Body section of the GetFeatureInfo template, it's provided with one feature collection, and

will be called multiple times if there are various feature collections

-->

<table class="featureInfo">

  <caption class="featureInfo">SOE Wells</caption>

  <tr>

<#list type.attributes as attribute>

  <#if !attribute.isGeometry>

    <th >${attribute.name}</th>

  </#if>

</#list>

  </tr>

<#assign odd = false>

<#list features as feature>

  <#if odd>

    <tr class="odd">

  <#else>

    <tr>

  </#if>

  <#assign odd = !odd>

  <#list feature.attributes as attribute>

    <#if !attribute.isGeometry>

      <td>${attribute.value}</td>

    </#if>

  </#list>

  </tr>

  <tr>

        <td colspan="4">

                <img src="demorequests/${feature.attributes["ID"].value}.png" width="400" height="216" alt="SOE Well ID ${feature.attributes["ID"].value} Hydrograph">

        </td>

  </tr>

</#list>

</table>

<br/>

In the JavaScript implementation of your OpenLayers map-object you fetch this template "auomagically" filled with data from Geoserver. You need to patch together following things:

• featureInfo = OpenLayers.Control.WMSGetFeatureInfo()
• map.addControl(featureInfo)
• featureInfo.activate()
• document.getElementById('map').style.cursor='pointer';

And finally you could display that generated html via an Ext.Window

The JavaScript stuff is a big hassle in my opinion. It gives you a lot of flexibility and nice cool things directly in the client browser, but it costs a lot of nerves. But maybe I am just not the real JavaScript developer at all :-p