SMART Groundwater Portal Dev going full "Cloud"

What a concise quote from Erik Dietrich, founder of DaedTech LLC:

Software developers demand the ability to work effectively from anywhere.  They have attained a coolness factor, and demand for them is so high that there is no need for them to guard their source code like squirrels preparing for winter.  GitHub is a good idea because it effectively captured what software developers really want and offered it to them pretty flawlessly.  GitHub is a zeitgeist that is taking over the world precisely because software developers are taking over the world and software developers really like GitHub. (source)

Although, I work at a research institute which is one half a commercial consultancy with IP to protect, on-going international research collaborations, and governmental research funding require us to be flexible, open and accessible.  I work as a research scientist/analyst programmer in a science department, not in the IT or applications department, and thus, IT infrastructure interaction in commercial entities is "challenging" - for the scientists as well as for the IT folks.

In our current project we embraced the Zeitgeist now, too. For our geodata portal development and deployment processes, we adopted following paradigm:

Google Cloud Platform, Compute and Container Engine with Kubernetes as the our computational platform.

Google Drive, Google Docs and Sheets for assets, functional and implementation specifications development, user stories and use cases.

GitHub as our distributed version control system, which allows us to collaborate, yet, keep contributions transparent and easily and publicly traceable.

Trello eventually serves as our workflow board, for sprints, keeping links of specs, repos and other soft information together. I believe, that we could have done everything in GitHub, but Google Docs and Trello provide gently mechanisms to also invite non-technical folks to contribute. And actually, what we really want is this, right?

Amazon IoT and OGC SensorThings vs SOS/SPS on MQTT

Today I stumbled over a blog post on the OGC Blog about "Amazon IoT and the candidate OGC SensorThings API Standard"

It was great to learn that Amazon IoT uses MQTT as network / transport layer. The more interesting is the move towards REST (which is HTTP) to address semantic issues with IoT systems:

AWS IoT, MQTT and many other network interoperability standards (LWM2M, CoAP, etc.) enable message interchange.  However, IoT network interoperability doesn’t enable the systems that are exchanging messages to interpret those messages. To realize the many-to-many system-of-systems vision, IoT applications need to implement standard ways of communicating sensor locations, sensor and data parameters, and sensor instruction sets. This is what the OGC SensorThings API provides.

It is understandable from the point of view, that OGC OWS is in general intrinsically interwoven with HTTP semantics and the application of SWE in the IoT context is desirable, well, even crucial.

However, MQTT stems from the constraints of low bandwidth, low energy consumption and unreliable connectivity, and implements a robust, small footprint asynchronous messaging platform. Earlier this we have developed a prototype system, that aims to add SWE semantics into MQTT messaging. We keep the specialisation of MQTT, where for example connections can be resumed which under SSL/TLS security can be a significant power saver. MQTT would also better address the "addressability" of distributed "things" which might be isolated behind IP NAT or run with dynamically assigned addresses, because in MQTT context the "things" would subscribe to SPS management topics and therefore higher order SDI services don't need to direct data queries or task requests to an address but place it on the queue with respective topics.

For an agricultural research project we distributed monitoring stations as a wireless sensor network throughout the catchment to deliver frequent measurements in near-realtime, which mimics OGC SOS.

A dynamic implementation of measurement frequencies, adapted to certain environmental conditions like heavy rainfall events require re-tasking of the nodes, which mimics OGC SPS. Within this paper we provide a framework where a threshold event triggers a reconfiguration task for a phosphorus measurement device, using asynchronous, push-based communication, on an MQTT queue, which links the ground stations with a cloud-based control system. OGC WPS algorithms continuously analyse incoming SOS measurements commits such a request into the wireless sensor network, and updates the measurement frequency of the target nodes to enable nutrient peak flow estimation during storm events.

This on-going work is based on original thoughts on the  interlinking  of  OGC  SWE  semantics  for  sensor  descriptions  and  observations (OGC Sensor Observation Service, SOS), sensor configuration and planning (OGC  Sensor Planning Service, SPS)  with  the  open  Message  Queue  Telemetry  Transport  (MQTT)  protocol.

MQTT  is  a  simple,  yet  very performing  and  robust  publish/subscribe  message  passing  system,  which  can  also  serve  for  event  brokering as  described  in  the  SWE  Service  Model.  Actual  addressing  and  data  transmission  is  done  by  a  topic  string and an arbitrary payload. The topic string and the payload need to reflect the necessary standard semantics to be  mapped  back  and  forth  to  allow  for  a  seamless,  and  preferably  lossless  bi-directional  multi-lateral communication  between  the  WSN  as  data  provider  (SOS),  web  clients  as  data  consumers  (SOS)  and  the management system, which interacts with the WSN in a standardised way, too (SOS and SPS). 

Geoscience Data Mining and Visualisation Brainstorming Weekend

Early 2016 New Zealand Ministry for Business, Innovation and Employment (MBIE) have put out calls for interested parties for a business to govt (B2G) data innovation challenge, one "opportunity" in particular is about "Geoscience Data management" - thought that I have some strengths and competency to contribute and was keen to have a say.

The Challenge: 

This so called R9 Accelerator brings the public and private sectors together to make it easier for business to interact with New Zealand government.

http://www.r9accelerator.co.nz/opportunities/opportunity14/

A prototype model could be applied to a range of other large databases managed by government, businesses and science institutions across the country. Data management issues are common internationally, so the model could have applications overseas.

On 29th Jan to 31st Jan 2016 was a full weekend information workshop in Miramar, Wellington. Subsequently, if interested, one would have to apply for the 3 months accelerator programme (either as team member or as mentor/domain expert).

http://www.r9accelerator.co.nz/apply/

http://www.r9accelerator.co.nz/timeline/

If teams would get elected to the full programme they would take part in the three month Accelerator starting 1st of March, then pitch to investors. This could be an opportunity to learn how private sector could better interact with govt data.

Alternatively, one could to consider to be involved on higher level into the process, which would be sort of part-time mentorship govt navigator or domain expert type participation.

http://www.r9accelerator.co.nz/take-part/support-a-team/

http://www.r9accelerator.co.nz/take-part/invest-in-a-team/

The plan was to show up and try to form a team and develop an idea and basic plan over the weekend, which will then be pitched on Sunday in a two minute presentation. This apparently would have the most impact on a team's chance of being accepted into the 3 months intense programme, where a real prototype is supposed to be developed by the team. The official application via an online form is then only a formal act to be completed subsequently for an already consistent and focussed team from the weekend).

The Geoscience Data Management opportunity was only one out of 14 or 15, and it's not obvious how many teams tackle each opportunity and how many applications are thought to go forward.

The Team Brainstorming Weekend:

There was a wild crowd of young and old, but only the team around the geodata challenge seemed to be high profile.

Katalyst / KDM Spectrum Data from Australia, Schlumberger, and the NZ agencies MBIE, LINZ, NIWA, GNS (Guy Maslen / Globe Claritas) had representatives there. So we were locked away over the weekend to brainstorm ideas to address MBIE's and NZPM immediate problem of nicer representation/delivery/visualisation of prospectivity data for possible investors in oil&gas and minerals.

From Dave Darby pitching the challenge...

WELLINGTON, NEW ZEALAND - January 29: R9 Accelerator Day 1: January 29, 2016 in Wellington, New Zealand. (Photo by Mark Tantrum/ http://mark tantrum.com, COPYRIGHT:2016 Mark Tantrum)

over group discussions...

WELLINGTON, NEW ZEALAND - January 30: R9 Accelerator Day 2. January 30, 2016 in Wellington, New Zealand. (Photo by Elias Rodriguez/ eliasrodriguez.co.nz) COPYRIGHT:2015 Elias Rodriguez

WELLINGTON, NEW ZEALAND - January 30: R9 Accelerator Day 2. January 30, 2016 in Wellington, New Zealand. (Photo by Elias Rodriguez/ eliasrodriguez.co.nz) COPYRIGHT:2015 Elias Rodriguez

.. toward the final pitch of what a team could possibly achieve if funded (respectively participate in this accelerator program):

WELLINGTON, NEW ZEALAND - January 31: R9 Accelerator Day 3. January 31, 2016 in Wellington, New Zealand. (Photo by Elias Rodriguez/ eliasrodriguez.co.nz) COPYRIGHT:2015 Elias Rodriguez

The (preliminary) Summary:

It could have been a great set-up for creating specific start-up type business solutions for MBIE across their departments.

We came up with designated/suggested team members , e.g. Guy Holmes and Tony Duffy(KDM Spectrum Data), Marielle Lange, a developer, Gavin Chapman, geodata management team at MBIE and I. We also suggested an advisory group, as far as I get it together: Dave Darby (MBIE), James Johnson (MBIE), Richard Garlick (MBIE), Jochen Schmidt (NIWA), Guy Maslen (GNS / Globe Claritas), Grep Byrom (LINZ).

If the proposal would have been accepted then the team would have had to develop a prototype with a little funding type stipend, and present that prototype to MBIE and other possible investors by June. Based on that further commercialization/contracting may arise. However, the professional team members were mainly supposed to support themselves (presumably KDM as big business, MBIE seconding their participant), and few of us would have to go full in and see if we'd be eligible for a part of the team stipend to basically live the start-up work life for the coming three months.

However, while the team, the idea, and the pitch were great, my situation would of course complicate my personal setting my PhD and within SMART programme. Eventually, I had to make a decision and withdrew to wrap up my PhD first. After all, it was a great opportunity to meet fascinating people and talk about possibly disruptive ways of re-shaping geoscience data management, visualization at governmental and even global scale.

Dreamteam FOSSGIS NZ eResearch 2013 Ignite Talk

In July 2013, NeSI, the New Zealand E-Science Infrastructure, organised lightning talks in Ignite format (20 slides, 15 seconds each) at the eResearch NZ conference in Christchurch.

I had the chance to present a little piece on the dream team combination of FOSSGIS (free and open source software for Geographic Information Systems) and OGC (Open Geospatial Consortium) standards, and how they are great enablers for Science and Education. Stumbled over this presi in my archives and thought I could share it:


A rough transcript :-)

1. Hi, I’m Alex and I’m a Geoinformatics PhD student at AUT University and I’m working in a New Zealand groundwater research project with GNS Science in Taupo.

2. Today I’d like to tell you the story of my journey. I think it is a great story and I want to share it with you - and being in my first year I needed to get my head around stuff.

3. So my part in the so called SMART aquifer characterisation project is about a groundwater web-portal for New Zealand. I need to collect, analyse, mash-up, visualise and share again a lot of different groundwater related data sets – and everything shall happen in the web

4. Little did I know before …The diversity of available content, formats and technologies is quite overwhelming – So I had to figure out where to start.

5. So my story is basically about discovery –a step by step exploration – to build something for the better of humanity (well, definitely for New Zealand)

6. And what helped me in this rather iterative and incremental process? Open Source software, especially Free and Open Source Software for Geoinformation Systems (FOSSGIS)

7. It worked for me for two reasons: If it didn’t perform, I could have a look inside and fiddle around with it to make it fit my demands – well, or I’ll just try something else, you know, freedom of choice.

8. And if you even want to give back to the community, you often can propose and contribute enhancements to make such software better – however all things optional

9. Alright, second thing, data sharing - Let’s have a look what’s happening out there in the vast expanses(expansiveness) of the internet.

10. So there is the Open Geospatial Consortium, the OGC, is an international consortium of 482 companies, government agencies and universities participating in an open consensus-based process to develop publicly available standards that "geo-enable" the Web, and thus fostering interoperability. Interoperability is key.

11. They create specifications for data formats and web services and so forth, to interact, integrate and communicate with each other. You can download the full specifications for free and again, you could also participate.

12. So now, does that OGC stuff work with open source? Bamm, there’s a huge open source software ecosystem supporting OGC standards –  go to osgeo.org or 52north.org and you will likely find everything you need to start.

13. And that’s actually really high quality software, They are often even the reference implementations for particular OGC standards - and you still can do with it whatever you want.

14. Ok, I don’t want to pull out what’s happening internationally, you know, Aussie, North America or Europe … Well, I learned New Zealand is just really awesome, too

15. So NZ has a Geospatial Office that published the NZ Geospatial strategy in 2007 which actually says “USE OGC STANDARDS” and make environmental and other spatial datasets available.

16. And boom, agencies, research institutes and even Kiwi-based commercial companies open up massive NZ datasets to the public – accessible through OGC interfaces and open source – OPEN STANDARDS work

17. So data and technology was available. I could start with small steps, be open, be agile and flexible. I really felt enabled to discover, grow and share in return.

18. So how can you facilitate that wealth of knowledge? I heard in New Zealand there is a GIS Masters programme available, is it covering OGC and Open Source software? I don’t know actually, but I’ve been asked to co-author an “Open GIS” module in another international GIS Masters (I could make it open access perhaps)

19. Well, having OGC and FOSSGIS at hand – you can conduct as well as support top-notch science and research, and also foster education of kids at school, students at uni even citizens at home with the same set of tools … communicate science and knowledge about this country and its beautiful nature that deserves protection and sustainable development

20. So when do you start? Thanks everyone, acknowledgements to GNS, AUT, eResearch NZ, MBIE/MSI

Environment Southland Information Management Conference

Regional councils and government agencies are increasingly under pressure to resolve data questions and discover how best to acquire, manage, collate, analyse, report and disseminate data, while managing  the associated costs. Steering organisations through these complex issues requires a solid understanding of what technologies are available and the information demands of the future *(source).

I had the great opportunity to speak at the Environment Southland Information Management Conference in Invercargill. It was a great event, well organised and very informative. I believe I could contribute my part to the line up and fill a few more gaps in the whole picture.
This was not a business as usual conference, it was obvious that the speakers took it serious to cater their presentations to the needs of the stakeholders. And with 70 attendees from regional and central government, as well as visitors from research and industry.

It was great to see the emerging patterns around NZ and similar approaches to a holistic, comprehensive and modern data strategy. If you are interested, this is a link to programme, and please see below for my slides. Watch the talk on Youtube.

A Spatial Data Infrastructure Approach for the Characterization of New Zealand's Groundwater Systems

I was very happy when I got informed that our latest research article was published in "Transactions in GIS". While being embedded in our New Zealand SMART Aquifer Characterisation programme, it was a great joint effort which results went beyond SMART.

Kmoch, A., Klug, H., Ritchie, A. B. H., Schmidt, J. and White, P. A. (2015), A Spatial Data Infrastructure Approach for the Characterization of New Zealand's Groundwater Systems. Transactions in GIS. doi: 10.1111/tgis.12171

It explains technical and methodological approaches of web groundwater data infrastructure and uses NIWA's CLIDB and GNS's NGMP as examples and describes the long way from stakeholder interaction, workshops and meetings towards an NZ data standard for the National Environmental Monitoring Standards (NEMS) framework - the Environmental Observation Data Profile (EODP) as data access and transfer blueprint for NEMS.

Wiley Online Library "Transactions in GIS" Journal

Alexander Kmoch on ResearchGate.

The technical work with colleagues, collaborators and partners from New Zealand research institutes (CRIs) like GNS, NIWA, Landcare Research, New Zealand regional councils like Horizons and Waikato (WRC), Hawke's Bay (HBRC) and and Bay of Plenty (BOPRC) regional councils which culminated in a draft OGC profile and is being incorporated into national standards in New Zealand, GitHub EODP and are based on the work over the last two years including a great review paper on OGC standards for groundwater,  customising the 52°North SOS server to interchangeably encode WaterML2 time-series data along O&M2 through a Google Summer of Code (GSoC) programme, and then customising the SOS server database access to demonstrate it as an adaptor on legacy databases of national significance, in specific an "NGMP/GGW-SOS" and a "CLIDB-SOS" demo through a NZ eResearch programme.

Digital Earth Conference 2015, Canada

"Towards a One-World Vision for the Blue Planet" was the slogan for the 9th Symposium of the International Society for Digital Earth (ISDE) from 5th - 9th of October, 2015, in Halifax, Nova Scotia, Canada.

At Digital Earth 2015 scientists, engineers, technologists, and environmental managers from around the world will meet to share concepts, research findings, technologies, and practical applications relating to the Digital Earth vision. *()

I presented an experimental approach towards integrating scientific legacy codes into the OGC web services framework, on the example of exposing USGS MODFLOW through a vanilla and open source WPS (without using proprietary tools like ESRI Arc Server and Arc Hydro Tools etc).

I was very honoured and greatly appreciated the "Best Student Poster" award. It seems I really had the right audience at ISDE.

Furthermore it was extremely informative, and in the session about "Discrete Global Grid Systems" I learned a lot about this new approach of unifying traditional raster/coverage data with an equal area per pixel advantage ... one of the great shortfalls of todays popular Web Mercator projections. In fact spinning this further it could be a  great new approach and how groundwater, geology, ocean modelling and atmospheric sciences, which would benefit from an equal volume grid, particularly when re-mashing resources from different environmental/scientific/governmental/industrial domains.

There is even an DGGS OGC Standards Working Group on the way.