ResearchGate milestone and GSoC finale ahead

What an exciting start of this week:

Google is re-organising itself into Alphabet, Google Summer of Code 2015 announced 'Pencils down' date in two weeks and ResearchGate informed me about reaching 200 publication downloads.

Well, being in the 3rd year of my PhD the number of publications, views and citations might not be outstanding but it is continuous progress along the early researcher path and a form of acknowledgement.

The GSoC project work with GeoTreliis and Azavea is also great opportunity to get more involved with Big Data technologies like Apache Spark and Cassandra and cloud technologies. The support from the project team, notably Rob and Chris, but also vibe on the GeoTrellis Gitter channel is fantastic.

Although it's time to wrap up GSoC in the next weeks, it is also the starting point of getting these new software development insights applied to my research in the SMART aquifer characterisation (SAC) programme, where I develop tools and web platforms for the SMART data portal. If all goes well, the SMART data portal will be overhauled end of this year and my PhD completed early next year :-)

Google Summer of Code 2015 with GeoTrellis, Cassandra and Spark!

Awesome news :-) Found this in my inbox:

<snip>
 Congratulations! Your proposal 'GeoTrellis: Cassandra Backend to GeoTrellis-Spark' submitted to The Eclipse Foundation has been accepted for Google Summer of Code 2015.

Welcome to Google Summer of Code 2015! We look forward to having you with us.

With best regards,
The Google Summer of Code Program Administration Team
</snip>


So, what is this about (Full proposal) ?

Cassandra Backend to GeoTrellis-Spark

1. Introduction

GeoTrellis is a Scala-based LocationTech project that is a framework for fast, parallel processing of geospatial data. Recent development efforts have allowed GeoTrellis to give the Apache Spark cluster compute engine geospatial capabilities, focusing on the processing of large scale raster data.

GeoTrellis's recent integration to with Apache Spark currently supports Hadoop HDFS and Accumulo as backends to store and retrieve raster data across a cluster. Cassandra is another distributed data store that could provide a rich set of features and performance opportunities to GeoTrellis running on top of Spark. It's also a popular distributed data store that a number of people interested in doing large scale geospatial computations are already using. A prototypical GeoTrellis catalog implementation for raster data in Cassandra is in development, yet it doesn't filter in the way we need.

This project would improve the GeoTrellis Catalog implementation for Cassandra, which allows us to save and load raster layers as Spark RDD's, as well as metadata. An important factor for distinction of GeoTrellis to other geospatial libraries the focus on performance. A performance-based indexing scheme needs to be integrated for being able to do spatial and spatio-temporal queries against Cassandra data as fast as possible. Eventually we will also be storing vector data in these data stores, and this project should support the efficient storing, indexing and retrieving of vector data with high-performance spatial and spatio-temporal filtering as well.

2. Background

GeoTrellis is Scala and Akka based high-performance geospatial processing framework. With the linking to Spray/Akka-Http GeoTrellis functionality can easily be exposed via Web Services, and with the latest integration with Apache Spark, GeoTrellis can now be run in massive large scale big data environments. Originally GeoTrellis had an own raster file type and a native catalog implementation that allowed for fast tile access, filtering and additional metadata. Now GeoTrellis also supports the widely used open GeoTIFF format.
To efficiently read, ingest, process and write data on Spark, data sources need to be exposed as SparkRDDs (Resilient Distributed Datasets). This allows to use Spark's great native algorithms to widely parallelise data processing and manipulation. GeoTrellis as of now supports the Hadoop HDFS filesystem for distributed file access and storage, and the Accumulo database for the GeoTrellis catalogue and advanced raster array access. A Cassandra raster integration is in an early stage development, yet filtering and indexing need to be improved as well.

3. The idea

Apache Cassandra is zero-dependency distributed NoSQL schema-less data base. It doesn't support referential integrity or joins like a relational database. It also has no spatial capabilities. With GeoTrellis on top of Cassandra this high-performance data store could be used in massively big data application with spatio-temporal requirements. There's is an active development community around the Apache Spark Cassandra connector, and the use of Cassandra data stores in the big data framework Spark can almost be considered main stream production.

GeoTrellis has its roots in high-performance raster data processing, and as rasters are in fact basically arrays of values with spatial and non-spatial metadata, GeoTrellis has its own (metadata) catalog implementation and can flexibly support an arbitrary variety of data stores for array data. Recent developments are prototyping raster storage and retrieval with Cassandra and vector support is planned. As Cassandra doesn't support any spatial indexing on its own (as of now), a high-performance indexing scheme needs to be implemented to be able to do spatial and spatio-temporal queries against Cassandra data as fast as possible. Here the vector and raster-based indexing and filtering methods against Cassandra and the GeoTrellis catalog need to be optimized for vector data in general and Cassandra in particular. Cassandra does support custom indexes on columns by reference to (presumably Java-based) an Index-class. Here typical discrete spatial indexes (in reconciliation with GeoTrellis catalog) like R-Trees, QuadTrees or possibly Space Filling Curves might be directly added to Cassandra to support spatial data indexing. The raster indexing/filtering is dependent on the GeoTrellis raster/array ingestion and referencing via the GeoTrellis catalog (documentation is a bit sparse, so I could also support in documenting the functionality in the course).

4. Future ideas / How can your idea be expanded?

There are several other great GIS, hydro-climate and geo-science (FOSS4G) toolkits that are based on Java and it would be great, if they or their functionality could also be subsequently be exposed and used under the GeoTrellis framework (like JTS is already, or the alternative GeoTIFF reader), particularly under large scale parallel processing capabilities on top of Apache Spark. This would allow for big data (business as well environmental science) applications under an enormously powerful framework.

Many scientific codes have been and are still developed and coded in Fortran, mainly because if its superior fast and efficient Matrix manipulation functionality. Yet, parallelising Fortran is hard, and in fact only viable on actual HPC facilities. With Scala, Akka and Breeze (or directly Spark and Java native BLAS...etc) parallelising scientific codes should be nicely scalable across commodity cloud servers. Maybe it could be evaluated at some point how to easily port popular Fortran codes to Scala/Spark and run them in unprecedented simplicity in main stream IT/server/cloud deployments at very comparable rate of performance.

About Google Summer of Code

Google Summer of Code (GSoC) is a global programme that offers stipends to students to write code for open source projects.  Google works with the open source community to identify and fund exciting projects. Alexander Kmoch, a student supported by the SMART Aquifer Characterisation programme (SAC), has been accepted as one of 1051 students in this year’s Google Summer of Code programme. Alex will help to improve the GeoTrellis database implementation for the cloud database Cassandra to allow processing of raster layers and vector data via the fast cluster engine for large-scale data processing Apache Spark. The GeoTrellis software will also be incorporated into the groundwater data portal that is being developed through the SMART project.

ZOO-Project WPS Java-API and JGrasstools Java Hydrological Toolbox

The ZOO-Project (http://zoo-project.org/) is a solid Open Geospatial Consortium (OGC) Web Processing Service (WPS - http://www.opengeospatial.org/standards/wps) standard server implementation with an open flexible API that works well with many different programming languages. The Java bindings have never been tested in advanced configurations and complex data types, and to date only implement the minimum necessary interfaces. The JGrasstools project is a modular processing library and its highly annotated nature makes it possible to adapt quite easily to other toolboxes. JGrasstools contains a wide variety of powerful and efficient GIS, hydrology and geomorphological tools and processes, that can be exposed to and used by other libraries and toolkits. One example has been the adaptation to the Geotools Process API. The JGrasstools project, as well as other java based projects (as JTS, Sextante or even Geotools) would benefit greatly from the possibility to be used within a web-enabled WPS execution environment, as well as being integrated with the open standards suite of the OGC.  Some time ago Moovida tried integrating the JGrasstools libraries with the ZOO-Project Java binding to expose them as native WPS processes. This would allow them to work inside the ZOO-Project and serve its modules under the WPS standard.

Some Background

The ZOO-Project WPS implementation is a flexible, modular high performance HTTP CGI implementation. ZOO-Kernel is a powerful server-side C Kernel which makes it possible to manage and chain Web services, by loading dynamic libraries and handling them as on-demand Web services. The ZOO Kernel is written in C language, and supports several common programming languages in order to connect to numerous libraries and models (http://zoo-project.org/trac/wiki/ZooWebSite/ZooKernel). The generic ZOO API is basically accessible for every possible programming and web scripting language that can be run under the CGI interface. Main API implementations, the ZOO services, are available for C/C++, Python, JavaScript, PHP, Fortran and Java. Some API bindings are more advanced and complete and make the full ZOO-API (http://zoo-project.org/trac/wiki/ZooWebSite/ZOOAPI/Classes#ZOOAPIClasses) accessible to the ZOO service in the particular programming language (e.g. C, Python or JavaScript). In comparison the the Java API binding only exposes the minimum functionality to run from the ZOO Kernel.

JGrasstools (http://moovida.github.io/jgrasstools/) is a powerful GIS toolkits with functionality reaching from standard geoprocessing algorithms to advanced processing features used in hydrology and geomorphology. JGrasstools is based on a Maven (http://maven.apache.org/) build process, which takes care of dependency resolution and creating the succinct jar packages with the compiled classes. Maven is a defacto standard for managing (source and dependencies) building and deploying (jar packaging, resources, copying, publishing, archiving and installing) Java-based software projects. JGrasstools is also used as a toolbox in the uDig desktop GIS software (http://udig.refractions.net/). If JGrasstools could be exposed via a open standards and interfaces, web-based processing and execution environment (like ZOO-Project provides) it can be widely used in WebGIS deployments and large scale cloud based processing chains.

The next level

Andrea from Moovida said he didn't have enough time to continue developing this idea. He developed a generator which would programmatically scan through the annotated JGrasstools modules and generate respective ZooJavaWps classes per JGrasstools module/method and the corresponding ZOO-Project .zcfg config file. The only struggle I had was getting the CLASSPATH properly set up, as the ZOO-Project is basically an HHTP CGI application which will start a JVM per request. When I picked up on this in preparation for a GSoC proposal, I found that there were a few little botches with the parameter mapping from ZOO Java API into the very nicely annotated JGrasstools methods. So I took one example generated (WPS-ified) JGrasstool process and adjusted the parameter mapping and got it running with ZOO-Project. Additionally I adjusted the JGrasstools Maven config files to download the necessary dependencies in the target folders to copy them collectively in the ZOO-Project Java CLASSPATH.

Unfortunately I also didn't have time to drive this further still. However, it is just soooo close really :-) Alternatively a 52North WPS implementation based on the super practical JGrasstools annotations and Andrea's generator would also be relatively straightforward.

GSoC and Master Thesis completed

Busy summer, well or winter, depends on the hemisphere and the full time extent. Last 2-3 months have been packed with work and included about 25.000 air miles (which means concatenated spending several days in airplanes).
I finally managed to finish my Master Thesis "Towards a 4D WebGIS using harmonised datasets: Examined on a New Zealand Example" which examines in the first part existing groundwater-related geoportal projects and scientific applications around the globe, existing and useful, web-based technologies and standards (e.g. OGC) to implement such a portal, a draft architecture. In the second part actual prototypical implementations of a selected subset of the examined standards and and technologies demonstrate their feasibility for the SMART project. I also hold a talk at the GI_Forum conference in Salzburg (proceedings/conference paper).
Furthermore I successfully completed my Google Summer of Code (GSoC) project with the 52°North Initiative, where I implemented an exchangeable encodings mechanism for their Sensor Observation Service (SOS). I additionally implemented a plugin that provides SOS time-series output in the WaterML2.0 format. I could acquire the latest schema, which will be published soon.. we were supposed to write a series of blog articles to document or progress in the project:

• Dynamic output formats for the Sensor Observation Service
• Exchangeable Encodings getting in shape
• Demonstrating Exchangeable Encodings: A Quantum Leap for me

After flying back to New Zealand I was lucky to visit the GeoSciML Face-to-Face-Meeting at GNS Science in Wellington. It was really exciting to meet these people, working on the international standard of a Geoscience Markup Language. also issues around the OneGeology portal have been disussed, as it uses GeoSciML to build up a world geology map. Additionally I could get in touch with one of the inspiring driving persons behind the Canadian Groundwater Information Network and their its sematic foundation, the Groundwater Markup Language (GWML), which itself is derived from GeoSciML.
Now I will dedicate my research to the development of a New Zealand groundwater geoportal within the SMART project, using OGC webservices, GeoSciML and/or GWML, WaterML2.0 and X3D as its foundations to transform and consume all available data relevant to aquifer characterisation and create a completely new, visually appealing 3D/4D view on New Zealand's groundwater resources.

Exciting ;-)

Demonstrating Exchangeable Encodings for SOS - A Quantum Leap

The final sprint towards the end of the Google Summer of Code (GSoC) is over now. After fixing bugs, working on documentation and actually producing time-series in different formats, the “Exchangeable Encodings for SOS”
project met most of its ambitious goals – and it is demonstrated on the 52°North Google summer of Code Demo Server. Users and developers documenation on the encoding mechanism has also found a place in the 52°North wiki.

We have integrated a plugin mechanism into an SOS server (source code branch), which is based on the main 52°North SOS development line (source code trunk).

Link to the article on the 52°North blog

Exchangeable Encodings Getting in Shape

The time is flying by and soon we reach midway of the Google Summer of Code (GSoC) with 52°North. It is the typical time in between – I already managed to meet the first milestones, but there is no proper product yet. Catching up with the changes from the SOS main development branch is a constant challenge, but the infrastructural changes within the “Exchangeable Encodings for SOS” project are slowly but continuously leading towards a robust and standardized, yet limited, encoding plugin API (application programming interface).

The main use case is the request for observations (GetObservation request) that could be delivered in CSV (comma separated values) format for legacy applications, WaterML2.0, an upcoming OGC (Open Geospatial Consortium)
data transfer standard for the hydrological domain, possibly netCDF (Network Common Data Form), also an OGC standard for meteorological applications, and in O&M (Observations & Measurements) for the generic use within sensor networks and spatial data infrastructures.

Link to the article on the 52°North blog

Dynamic Output Formats for the Sensor Observation Service

As one of the 52°North Google Summer of Code projects, “Exchangeable Encodings for the 52°North Sensor Observation Service (SOS)” aims to implement a customisable result set encoding mechanism. The users would then be able to add their own compiled libraries (like plugins), which contain all the necessary code to provide the requested data in different formats, such as CSV or GeoJSON, to the SOS. When that mechanism works, we can easily add more encodings, e.g. the planned WaterML2.0 – but for now, where to start?

The internal architecture of the 52°North SOS server is generally well structured. Based on the user request, the respective Listener will get the data from the data backend and send it to the corresponding Encoder. The response document, very likely a collection of observations, is then returned to the user in the requested encoding, also referred to as the response format. The typical application flow of such a request is depicted in the comic strip below. Available response formats can be queried from SOS server and are listed in the capabilities document. To date the two integrated encoders return XML documents based on the OGC© O&M schema and the OGC© SOS versions.

Link to the article on the 52°North blog

Exchangeable Encodings for SOS - extension support WaterML2

The 52°North SOS implements only a subset of the O&M data format, because O&M is too flexible to be supported completely. It would be better if users could just implement a simple encoding (for example to support their own CSV or JSON data format) and upload a compiled jar file to the SOS with a browser-based user interface. Then a client selects the newly available response format to retrieve the data in a new format. As an extension it would be very useful to researchers and data maintainers in the hydrological and meteorological domain to have the SOS server deliver WaterML2.0 encoded time-series. Based on the SOS adminstrator project, the desired encodings could be prepared as .jar-files and added via a web frontend.

Link to the 2012 GSoC proposal