Starting to play with spatial-temporal data

Thinking  the big picture is a different thing to actually implement it :-) Well, who doesn't know that. Having some hydrological time-series (groundwater levels) in place in a sensor observation service (SOS), the hydrogeological all-in-one-wonder-portal is going to get a glance of the next level :-p
The last weeks I started to play around with the R environment for statistical computing and visualization (The R Project). 52°North developed a neat R toolkit to access and digest SOS time-series - sos4R.

It is well documented and pretty easy to connect to a SOS server, and query observations. So for the fun of it and to demonstrate the general feasibilty, I quickly queried the groundwater levels of the Horowhenua area in New Zealand, where I got some sample data (courtesy by the regional council).

With the R sos4R, fields and akima packages from the CRAN R packages archive I (quite coarsely) interpolated the groundwater surfaces for the years 1991-2009 and put the images together as an animated gif (meters above mean sea level over time).

Discussion

I am aware of the total uselessness of this particualr way presenting :-) No years, the scale changes slightly, and the exact spatial extent and north orientiation are not reliable :-p

Nevertheless, for just playing around, this was a motivating simple first shot to easily visualise changes over time.

I would like to play around with the gstat and spacetime R package, integrate a more sophisticated script as a 52°North WPS process and have those things happening automagically in the interwebz.

GSoC and Master Thesis completed

Busy summer, well or winter, depends on the hemisphere and the full time extent. Last 2-3 months have been packed with work and included about 25.000 air miles (which means concatenated spending several days in airplanes).
I finally managed to finish my Master Thesis "Towards a 4D WebGIS using harmonised datasets: Examined on a New Zealand Example" which examines in the first part existing groundwater-related geoportal projects and scientific applications around the globe, existing and useful, web-based technologies and standards (e.g. OGC) to implement such a portal, a draft architecture. In the second part actual prototypical implementations of a selected subset of the examined standards and and technologies demonstrate their feasibility for the SMART project. I also hold a talk at the GI_Forum conference in Salzburg (proceedings/conference paper).
Furthermore I successfully completed my Google Summer of Code (GSoC) project with the 52°North Initiative, where I implemented an exchangeable encodings mechanism for their Sensor Observation Service (SOS). I additionally implemented a plugin that provides SOS time-series output in the WaterML2.0 format. I could acquire the latest schema, which will be published soon.. we were supposed to write a series of blog articles to document or progress in the project:

• Dynamic output formats for the Sensor Observation Service
• Exchangeable Encodings getting in shape
• Demonstrating Exchangeable Encodings: A Quantum Leap for me

After flying back to New Zealand I was lucky to visit the GeoSciML Face-to-Face-Meeting at GNS Science in Wellington. It was really exciting to meet these people, working on the international standard of a Geoscience Markup Language. also issues around the OneGeology portal have been disussed, as it uses GeoSciML to build up a world geology map. Additionally I could get in touch with one of the inspiring driving persons behind the Canadian Groundwater Information Network and their its sematic foundation, the Groundwater Markup Language (GWML), which itself is derived from GeoSciML.
Now I will dedicate my research to the development of a New Zealand groundwater geoportal within the SMART project, using OGC webservices, GeoSciML and/or GWML, WaterML2.0 and X3D as its foundations to transform and consume all available data relevant to aquifer characterisation and create a completely new, visually appealing 3D/4D view on New Zealand's groundwater resources.

Exciting ;-)

Demonstrating Exchangeable Encodings for SOS - A Quantum Leap

The final sprint towards the end of the Google Summer of Code (GSoC) is over now. After fixing bugs, working on documentation and actually producing time-series in different formats, the “Exchangeable Encodings for SOS”
project met most of its ambitious goals – and it is demonstrated on the 52°North Google summer of Code Demo Server. Users and developers documenation on the encoding mechanism has also found a place in the 52°North wiki.

We have integrated a plugin mechanism into an SOS server (source code branch), which is based on the main 52°North SOS development line (source code trunk).

Link to the article on the 52°North blog

First X3D example to visualise geological layers in the web

Recently I have played around with tools to visualise geological layers in 3D in the web, preferably without any browser plugins. With HTML5 and WebGL some really cool possibilities arise. WebGL is not supported by every browser, but apparently  all newer cool browsers like Chrome or Firefox, as well as Safari and Opera support WebGL at least experimentally. What a surprise that Microsoft Internet Explorer does not net yet support neither HTML5 canvas nor WebGL. But luckily there is the Chromeframe plugin :-)

If anyone has ever been working with OpenGL for 3D stuff probably in C or C++, well, WebGL is quite arcane, too, but in JavaScript ^^

Nevertheless, X3D for the rescue. X3D is a) an ISO standard (ISO/IEC 19775-1.2:2008 ), b) the successor of the working, but not really successful VRML97 (ISO/IEC 14772-1.2:1997) and c) a fully XML-based scenegraph declarative language. And the final ingredient is the Fraunhofer IGD experimental open source framework x3dom that thrives to integrate X3D content into HTML5.

Well, to the actual task. I am supposed to visualise a 3D geological model that originally has been designed with EarthVision(c). EarthVision(c) has its own binary format to store the 3D models, but they van be exported to a simpe XYZ-ASCII file:

2696105 6047205 150 20 11
2696605 6047205 150 21 11
2697105 6047205 150 22 11
2697605 6047205 150 23 11
2701105 6047205 150 30 11

...

The first two columns are Easting and Northing - implicitly known that the spatial reference system is New Zealand Map Grid. Third column is the height value and the further columns represent some additional attribute data. Based on the resolution, the surfaces from the roundabout 20 km  by 30 km range from 50KB (500m), 1MB (100m) to 20MB (20m) per layer (5 layers altogether).
X3D provides two easy (point set based) possibilities to show surfaces (right now the geological layers are represented as surfaces, they are not described as full bodies).

• ElevationGrid
• IndexedFaceSet

Furthermore there are the NURBS and extrusion implementations, which describes surfaces through splines. But that's rather complicated for the first shot :-) ElevationGrid requires an evenly spaced grid of height values, whereas IndexedFaceSet  notes all point coordinates and then defines coordinate indexes to define a mesh of single surfaces (similar to TINs, but not necessarily triangles).
I decided for the ElevationGrid. X3D has a geospatial extension (X3D Earth), which can geographically reference and place 3D objects in a defined spatial reference system. I didn't try this feature yet. And apparently it does not make sense to load 100MB for a 3D model into the browser. Therefore the 500m grid has been used here. The following example outlines a rather simple definition of such an elevation grid in X3D:

<shape> 

    <elevationgrid colorpervertex="false" creaseangle="3.14" def="Greywacke_top_500" 
        normalPerVertx="true" colorPerVertex='false' xDimension='37'
        zDimension='44' xSpacing='500' zSpacing='500' creaseAngle='3.14' solid='false'
        height='
150 150 150 ...
           '>
    </elevationgrid> 
    <appearance>
        <material ambientintensity="0.1" diffusecolor="red" id="Greywacke" 
            shininess="0.2" specularcolor="lightred" transparency="0.0">
        </material> 
    </appearance>
<shape>

To prepare that grid, you need to take care of some things:

• you need to know the extent and resolution of the dataset, from that you calculate and define the x- and zSpacing (how many values will be filled, because you only need the height values)
• the coordinate system orientation of the X3D 3-dimensional space is probably from the 2/2,5 coordinate system from the source dataset
• the source datasets only contained points with actual values, to fill the ElevationGrid properly, NODATA values need to added

Finally I got my data sorted on built a neat first little demo :-) There might happen some improvement, as we are intending to visualise wells, bores and other hydro(geo)logical data in such a scene.

Fig. 1: Preliminary (X)3D  model with five layers and a (not aligned) image as an underlayer

Exchangeable Encodings Getting in Shape

The time is flying by and soon we reach midway of the Google Summer of Code (GSoC) with 52°North. It is the typical time in between – I already managed to meet the first milestones, but there is no proper product yet. Catching up with the changes from the SOS main development branch is a constant challenge, but the infrastructural changes within the “Exchangeable Encodings for SOS” project are slowly but continuously leading towards a robust and standardized, yet limited, encoding plugin API (application programming interface).

The main use case is the request for observations (GetObservation request) that could be delivered in CSV (comma separated values) format for legacy applications, WaterML2.0, an upcoming OGC (Open Geospatial Consortium)
data transfer standard for the hydrological domain, possibly netCDF (Network Common Data Form), also an OGC standard for meteorological applications, and in O&M (Observations & Measurements) for the generic use within sensor networks and spatial data infrastructures.

Link to the article on the 52°North blog

Dynamic Output Formats for the Sensor Observation Service

As one of the 52°North Google Summer of Code projects, “Exchangeable Encodings for the 52°North Sensor Observation Service (SOS)” aims to implement a customisable result set encoding mechanism. The users would then be able to add their own compiled libraries (like plugins), which contain all the necessary code to provide the requested data in different formats, such as CSV or GeoJSON, to the SOS. When that mechanism works, we can easily add more encodings, e.g. the planned WaterML2.0 – but for now, where to start?

The internal architecture of the 52°North SOS server is generally well structured. Based on the user request, the respective Listener will get the data from the data backend and send it to the corresponding Encoder. The response document, very likely a collection of observations, is then returned to the user in the requested encoding, also referred to as the response format. The typical application flow of such a request is depicted in the comic strip below. Available response formats can be queried from SOS server and are listed in the capabilities document. To date the two integrated encoders return XML documents based on the OGC© O&M schema and the OGC© SOS versions.

Link to the article on the 52°North blog

The Devil is in the Details

As one of the 52°North Google Summer of Code projects, “Exchangeable Encodings for SOS” aims to implement a customisable result set encoding mechanism.   If that mechanism works,  users could add an own pre-compiled .jar-file to the SOS that contains all the necessary code to encode the requested data in simpler formats like CSV or GeoJSON – but for now, where to start?

The architecture of the SOS server is generally well structured. Based on the request – GetCapabilities, GetObservation – and given parameters, like Offering, FeatureOfInterest and/or time the RequestOperator calls the respective Listener. The Listener calls the DAO, which fills an internal response object and gives it back to the Listener. The response object, most likely a SosObservationCollection is then sent to the respective encoding module actually based on the SOS version from the request. Then the Listener decides, which Encoder will be used - to date the decision is mainly based on the requested SOS protocol version. The Encoder now creates an XML object based on the OGC O&M schema and gives it back to the Listener. Finally the Listener puts the textual representation of that XML object into the response document that will be returned via http. This is good, because other encodings are likely to be plain/text, too.

There are different entry points to dynamacally add encodings, either as additional full-fledged SOS server modules or as another responseFormat internally be evaluated and the respective encoder selected through the SosConfigurator instance. But following constraints are to be regarded:

• The SOS protocol versions 1.0 and 2.0 must be honoured
• responseFormat O&M 2.0 ships as standard
• Listener and Encoder are coupled through the SOS protocol version, the response encoding needs to be decoupled and dynamically selectable
• Other formats / encoders need to be “dynamically” registered in the SoSConfigurator and to be identified and allocated based on the requested ResponseFormat value from the request

Simple sequence diagram of a GetObservation request to the SOS server

Finally one of the challenges I suppose, is the generation of a non-XML-response in this XML dominated workflow. As the SOS internal response object is merely of the String datatype, a simple encoder would need to implement the ISosResponse for the return value and should accept the SOS internal data collection representation as delivered from the DAO backend. Here in between we will put the lever to enhance the versatility of the 52 North SOS server.