Showing posts with label GIS. Show all posts
Showing posts with label GIS. Show all posts

Monday, 7 December 2020

Teaching with GIS or about GIS? University educator's dilemma


Teaching who: for us, for companies?

About GIS, with GIS
Education or training
Tools or techniques/methods
Geography side approach towards Geoinformatics (the big picture for geographers)
European Context -> INSPIRE?

Terminology: GIS geoinformatics, GI Science … GI Society, GI Science and Technology
Spatial thinking
Geoinformatik in Germany -> spatial process with methods of informatics
What about Geomatics (origin surveying)
Cartography & geodesy as sub sections of …
NCGIA curriculum from the 1990s





Friday, 3 April 2020

Tartu Geo, a podcast about geography and geoinformatics

Tahmin, a student in our Geoinformatics for Urbanised society MSc programme, and I have been working hard over the last months to record, edit, and polish on our podcast project: a GIS-themed educational podcast from our Department of Geography (Geograafia osakond) here in Tartu, Estonia.

Introducing: Tartu Geo, a podcast about geography and geoinformatics related fields along with education, research, history, philosophy, ground breaking ideas and innovation. The show will also host guests from different sectors who will share insight about their research and work. We try to bring up new episodes every 15 days with exciting and newer topics of geoinformatics and geography. Stay tuned and happy listening.

https://tartugeo-podcast.com/

Please bear with us, we are still amateurs :-D The sound quality in some guest interviews is not great, but overall it is getting better and better as we progress. We have already several episodes lined up, and the podcast can be loaded from all the convenient places like iTunes, Stitcher, Spotify and of course you can add the feed also directly to your podcast app.




Tartu Geo Podcast


Please let us know any suggestions or advice that can help us grow. Like always comment, like and subscribe.

Wednesday, 25 September 2019

A review of selected latest journal articles in the field of Geoinformatics



I spontaneously filled an open slot for our regular PhD seminar in the Chair of Geoinformatics. As I described in an earlier blog post, I occasionally browse through latest published articles like scanning for headlines in the daily news. And with an app like Feedly or Inoreader you can tag interesting articles for later and group them. Now was the chance to re-iterate through some of the latest articles - literally aiming for only few months old to a maximum of 1-2 years old. I initially came up with a group of ca 30-35 articles, which was still too much for a single seminar.

GIScience is not a homogenous and strictly defined discipline, and there is no consensus among GIScience researchers about the relevant publication outlets.
Filip Biljecki (2016) “A scientometric analysis of selected GIScience journals”, International Journal of Geographical Information Science, 30:7, 1302-1335, DOI: 10.1080/13658816.2015.1130831

The original idea was to cover a few different topics that are also relevant for the teaching and research we do in the department, such as:

- cloud processing and Google Earth engine
- machine learning and GIS (statistical modelling)
- at least like European level/scale modelling
- some modern cartography/visualization topics
- terrain modelling

I'd then give a short overview of journals, and the selected papers hand out copies one each. 5 minutes reading/skimming and then one circulation, and another 5 minutes, and then discussion. The timing turned out to be too optimistic and we spent more time discussion the papers and interesting facts the participants found for themselves.




  • Comparison of FOSS4G Supported Equal-Area Projections Using Discrete Distortion Indicatrices, ISPRS Int. J. Geo-Inf. 2019, 8(8), 351; https://doi.org/10.3390/ijgi8080351
  • Performance Testing on Marker Clustering and Heatmap Visualization Techniques: A Comparative Study on JavaScript Mapping Libraries, ISPRS Int. J. Geo-Inf. 2019, 8(8), 348; https://doi.org/10.3390/ijgi8080348
  • Examining the sensitivity of spatial scale in cellular automata Markov chain simulation of land use change, International Journal of Geographical Information Science, 33:5, 1040-1061, DOI: 10.1080/13658816.2019.1568441
  • The scale effects of the spatial autocorrelation measurement: aggregation level and spatial resolution, International Journal of Geographical Information Science, 33:5, 945-966, DOI: 10.1080/13658816.2018.1564316
  • Comparative usability of an augmented reality sandtable and 3D GIS for education, International Journal of Geographical Information Science, DOI: 10.1080/13658816.2019.1656810
  • Deeply integrating Linked Data with Geographic Information Systems, Transactions in GIS 2019 https://doi.org/10.1111/tgis.12538
  • GIS&T pedagogies and instructional challenges in higher education: A survey of educators, Transactions in GIS 2019 https://doi.org/10.1111/tgis.12534
  • The spatial allocation of population: a review of large-scale gridded population data products and their fitness for use, Earth Syst. Sci. Data, 11, 1385–1409, https://doi.org/10.5194/essd-11-1385-2019
  • Google Earth Engine: Planetary-scale geospatial analysis for everyone, 2017, Remote Sensing of Environment, https://doi.org/10.1016/j.rse.2017.06.031
  • Big spatial vector data management: a review, Big Earth Data, 2:1, 108-129, DOI: 10.1080/20964471.2018.1432115
  • Contemporary American cartographic research: a review and prospective, Cartography and Geographic Information Science, 46:3, 196-209, DOI: 10.1080/15230406.2019.1571441
  • Automated and semi-automated map georeferencing, Cartography and Geographic Information Science, DOI: 10.1080/15230406.2019.1604161

Monday, 24 September 2018

Some really nice geospatial podcasts to enlighten your day





Geodorable: https://geodorable.com/

  • If you like maps and location then you might find listening to a couple of middle-aged guys from the land of the long white cloud (New Zealand) chat irreverently about geospatial stuff just the ticket!

Isn't That Spatial: http://isntthatspatial.net/

  • "Isn't That Spatial" is the podcast bringing everyday geography and urbanism into your earspace.

The Mappyist Hour: http://themappyisthour.com/
  • Geographer and Geo types talking about how incredible their profession is "after hours"

The Scene From Above Podcast: https://www.geoger.co.uk/podcast

  • The #scenefromabove podcast aims to present an informal podcast looking at the world of modern remote sensing and Earth observation, fuelled by their passion for all things raster and geospatial: a mix of news, opinion, discussion and interviews.

GeoGedöns (in German): http://geogedoens.de/

  • Ein Podcast von zwei enthusiastischen Satellitennavigationsbegeisterten, die gerne Technik testen und Spiele spielen, die sich am Handy bzw Smartphone abspielen und für die meist eine Lokalisation via Satelliten nötig ist.

Radio OSM (in German)http://podcast.openstreetmap.de/

  • Berichte und Neuigkeiten rund um OpenStreetMap, ​die freie Wiki-Weltkarte


  • JBGeoPro - Joe Bob Penor (United States), Soundcloud Podcast

A VerySpatial Podcast: http://www.veryspatial.com/
  • Discussions on Geography and geospatial technologies. The VerySpatial blog is intended to be a location for the hosts and participants of A VerySpatial Podcast to link to interesting sites and articles on Geography and related information.

Wednesday, 7 February 2018

Info Session for MSc programme in Geoinformatics for Urbanised Society

Today we broadcast a webinar info session about the new MSc programme in Geoinformatics for Urbanised Society at the University of Tartu.

One of the recurring core topics in this new MSc curriculum is Data and GIS use in Urban Planning.

Until just a decade ago spatial planning and analytics projects had problems with getting enough data. But nowadays there is so much data available, that it is increasingly hard to make sense of it – because of the 3 V’s of big data - volume, variety, velocity. The open data movement, government agencies, research institutes and citizen scientists alike make more and more data available publicly, mobile phones, sensor networks and satellites generate a multitude of datasets every day

In order solve the Interdisciplinary challenges of urban planning we want to empower you with skills and knowledge to analyse, visualise and understand processes and data. For that we teach the Full cycle of spatial data management, from the various methods of data acquisition, followed by efficient and practical processing techniques, to subsequent meaningful analysis and visualisation; in order to consequently make successful planning decisions for a sustainable future.

So what does it mean to study Geoinformatics for Urbanised Society with us in Tartu?

You will learn how to combine geography and IT in the age of BIG data. This is essentially what we believe modern Geoinformatics is representing. Mastering Geoinformatics will provide you with tools to analyse social and natural processes in space for interdisciplinary decision- and policy-making.

Watch the whole recorded session for more info:



Links:

Monday, 15 January 2018

Staying informed with Literature Review on the smartphone

As a scientist / researcher you have to stay informed about the latest research findings in your field. Typically, this means that you should follow the publications of the most important journals in your research domain. I found that many researchers (including myself so far) only conduct proper literature reviews when they work on a specific problem, when writing articles and grant applications. Often they wouldn't find the time to go to all the journal websites and scroll through the article lists etc.

I recently discovered the Feedly RSS reader. RSS is long-known internet feed syndication protocol that is used to subscribe to updates on websites and blogs. I found that many, if not all journals, more or less provide RSS feeds for their latest articles, often with an abstract provided.  Feedly is a website application that helps you to organise RSS feeds and read them online. There is also an Android and an iPhone app. With these you can then read and manage your feeds on the phone. Now I quickly scroll through the latest articles every day via my mobile phone. This way it is just like scrolling through Facebook, Twitter or Instagram, but for journals. And the very few articles that you find relevant or that are of interest for you, you can save them in your Feedly backlog in order to read them later when you are in your office and take reading time :-)

Update: Inoreader is another very similar application to read your RRS feeds, and there are also the respective smartphone apps available.

One advantage of Feedly is that you can export and import lists of your RSS feed sources with so called OPML files.  OPML is available in many RSS reader web sites and applications, so you can both import and export OPML files of RSS subscriptions.

I prepared an OPML Export file for you, a standard list format for your feed URLs,  so you don't have to aggregate all the RSS URLs again. Happy reading and feel confident that you are not missing out on latest papers in your field.

Link to my export.opml file: https://www.dropbox.com/s/li0yjvwyt8cezsc/export.opml?dl=0

It includes links and feeds to the following resources:


  • Information systems and information technology : nature.com subject feeds - rss url and webpage url
  • Google Cloud Big Data and Machine Learning Blog - Google Cloud Big Data and Machine Learning Blog - rss url
  • Research Participant Portal - Funding Opportunities - Recently published Calls - rss url  - webpage url



Happy reading :-)

Wednesday, 30 November 2016

AGILE 2016 - SensorWeb Semantics on MQTT for responsive Rainfall Recharge Modelling

Integrating Wireless Sensor Networks (WSNs) and spatial data web services is becoming common in ecological applications. However, WSNs were developed in application domains with different sensor and user types, and often with their own low-level metadata semantics, data format and communication protocols. The sensor web enablement initiative (SWE) within the Open Geospatial Consortium (OGC) has released a set of open standards for interoperable interface specifications and (meta) data encodings for the real time integration of sensors and sensor networks into a web services architecture.
Such XML-based web services exhibit disadvantages in terms of payload and connectivity in low-bandwidth low energy unreliable networks, such as remote 3G uplinks. Monitoring stations deliver frequent measurements in real-time, but dynamic implementation of measurement frequencies, adapted to certain environmental conditions, are rarely implemented. We describe a responsive integrated hydrological monitoring prototype to calculate rainfall recharge for water management purposes.
When rainfall is observed, a threshold event triggers a reconfiguration task for the soil moisture sensors, using asynchronous, push-based communication implemented with an MQTT queue. A Sensor Planning Service commits that request via MQTT into the wireless sensor network, and updates the measurement frequency of the target sensors to gain higher resolution for the vertical soil water infiltration.
The system integrates a Sensor Observation Service (SOS) including field observations and internet-based environmental data with a rainfall recharge model that allows near-real time calculation of rainfall recharge in the Upper Rangitaiki catchment, Bay of Plenty region in New Zealand.


Figure 1: Setup and location of the sensor field site, central North Island, New Zealand

The prototype site comprises a main station conducting comprehensive measurements of meteorological, hydrological and pedological parameters. For the wireless data transmission within the local site installation XBee-PRO modules from the Digi Company  ZigBee IEEE 802.15.4 protocol are used. The main station receives continuous sensor measurements from the attached sensor units, and acts as the gateway to the online SOS and SPS services by providing the communication channel from the local sensor network to the web-enabled data management infrastructure.
The field site has been established in the Upper Rangitaiki catchment (Figure 1) and comprises a field computer (Raspberry Pi) with a direct internet link (GPRS/3G) and a sensor board (Waspmote) that has 12 typical meteorological, hydrological and pedological sensors attached (i.e., wind speed, wind direction, rainfall, 1x groundwater probe, 5x temperature and 3x soil moisture). The Raspberry Pi and Waspmote can be monitored and reprogrammed from an online server.

Figure 2: Raw sensor series visualized in a website from a SOS query.

The site setup allows scaling up to a multitude of low cost, low energy sensor stations throughout the catchment, with only one field computer that serves as data logger for backup. The observations were available in a standardized open format. The website accessed the raw data from the SOS server and plotted data points within 5-10 minutes of field measurement. This website was easily accessible via browsers and smartphones (Figure 2).

The paper was was presented at the 19th AGILE International Conference on Geographic Information Science, 15th of June, in  Helsinki, Finlkand.

Kmoch, A., Klug, H., White, P., & Reichel, S. (2016). SensorWeb Semantics on MQTT for responsive Rainfall Recharge Modelling. In 19th AGILE International Conference on Geographic Information Science. Helsinki.


Friday, 29 April 2016

Geoscience Data Mining and Visualisation Brainstorming Weekend

Early 2016 New Zealand Ministry for Business, Innovation and Employment (MBIE) have put out calls for interested parties for a business to govt (B2G) data innovation challenge, one "opportunity" in particular is about "Geoscience Data management" - thought that I have some strengths and competency to contribute and was keen to have a say.

The Challenge: 

This so called R9 Accelerator brings the public and private sectors together to make it easier for business to interact with New Zealand government.

http://www.r9accelerator.co.nz/opportunities/opportunity14/

A prototype model could be applied to a range of other large databases managed by government, businesses and science institutions across the country. Data management issues are common internationally, so the model could have applications overseas.

On 29th Jan to 31st Jan 2016 was a full weekend information workshop in Miramar, Wellington. Subsequently, if interested, one would have to apply for the 3 months accelerator programme (either as team member or as mentor/domain expert).

http://www.r9accelerator.co.nz/apply/

http://www.r9accelerator.co.nz/timeline/

If teams would get elected to the full programme they would take part in the three month Accelerator starting 1st of March, then pitch to investors. This could be an opportunity to learn how private sector could better interact with govt data.

Alternatively, one could to consider to be involved on higher level into the process, which would be sort of part-time mentorship govt navigator or domain expert type participation.

http://www.r9accelerator.co.nz/take-part/support-a-team/

http://www.r9accelerator.co.nz/take-part/invest-in-a-team/

The plan was to show up and try to form a team and develop an idea and basic plan over the weekend, which will then be pitched on Sunday in a two minute presentation. This apparently would have the most impact on a team's chance of being accepted into the 3 months intense programme, where a real prototype is supposed to be developed by the team. The official application via an online form is then only a formal act to be completed subsequently for an already consistent and focussed team from the weekend).

The Geoscience Data Management opportunity was only one out of 14 or 15, and it's not obvious how many teams tackle each opportunity and how many applications are thought to go forward.

The Team Brainstorming Weekend:

There was a wild crowd of young and old, but only the team around the geodata challenge seemed to be high profile.

Katalyst / KDM Spectrum Data from Australia, Schlumberger, and the NZ agencies MBIE, LINZ, NIWA, GNS (Guy Maslen / Globe Claritas) had representatives there. So we were locked away over the weekend to brainstorm ideas to address MBIE's and NZPM immediate problem of nicer representation/delivery/visualisation of prospectivity data for possible investors in oil&gas and minerals.

From Dave Darby pitching the challenge...


WELLINGTON, NEW ZEALAND - January 29: R9 Accelerator Day 1: January 29, 2016 in Wellington, New Zealand. (Photo by Mark Tantrum/ http://mark tantrum.com, COPYRIGHT:2016 Mark Tantrum)



over group discussions...

WELLINGTON, NEW ZEALAND - January 30: R9 Accelerator Day 2. January 30, 2016 in Wellington, New Zealand. (Photo by Elias Rodriguez/ eliasrodriguez.co.nz) COPYRIGHT:2015 Elias Rodriguez
WELLINGTON, NEW ZEALAND - January 30: R9 Accelerator Day 2. January 30, 2016 in Wellington, New Zealand. (Photo by Elias Rodriguez/ eliasrodriguez.co.nz) COPYRIGHT:2015 Elias Rodriguez

.. toward the final pitch of what a team could possibly achieve if funded (respectively participate in this accelerator program):

WELLINGTON, NEW ZEALAND - January 31: R9 Accelerator Day 3. January 31, 2016 in Wellington, New Zealand. (Photo by Elias Rodriguez/ eliasrodriguez.co.nz) COPYRIGHT:2015 Elias Rodriguez

The (preliminary) Summary:

It could have been a great set-up for creating specific start-up type business solutions for MBIE across their departments.

We came up with designated/suggested team members , e.g. Guy Holmes and Tony Duffy(KDM Spectrum Data), Marielle Lange, a developer, Gavin Chapman, geodata management team at MBIE and I. We also suggested an advisory group, as far as I get it together: Dave Darby (MBIE), James Johnson (MBIE), Richard Garlick (MBIE), Jochen Schmidt (NIWA), Guy Maslen (GNS / Globe Claritas), Grep Byrom (LINZ).

If the proposal would have been accepted then the team would have had to develop a prototype with a little funding type stipend, and present that prototype to MBIE and other possible investors by June. Based on that further commercialization/contracting may arise. However, the professional team members were mainly supposed to support themselves (presumably KDM as big business, MBIE seconding their participant), and few of us would have to go full in and see if we'd be eligible for a part of the team stipend to basically live the start-up work life for the coming three months.

However, while the team, the idea, and the pitch were great, my situation would of course complicate my personal setting my PhD and within SMART programme. Eventually, I had to make a decision and withdrew to wrap up my PhD first. After all, it was a great opportunity to meet fascinating people and talk about possibly disruptive ways of re-shaping geoscience data management, visualization at governmental and even global scale.



Thursday, 24 March 2016

Dreamteam FOSSGIS NZ eResearch 2013 Ignite Talk

In July 2013, NeSI, the New Zealand E-Science Infrastructure, organised lightning talks in Ignite format (20 slides, 15 seconds each) at the eResearch NZ conference in Christchurch.

I had the chance to present a little piece on the dream team combination of FOSSGIS (free and open source software for Geographic Information Systems) and OGC (Open Geospatial Consortium) standards, and how they are great enablers for Science and Education. Stumbled over this presi in my archives and thought I could share it:


A rough transcript :-)

1. Hi, I’m Alex and I’m a Geoinformatics PhD student at AUT University and I’m working in a New Zealand groundwater research project with GNS Science in Taupo.

2. Today I’d like to tell you the story of my journey. I think it is a great story and I want to share it with you - and being in my first year I needed to get my head around stuff.

3. So my part in the so called SMART aquifer characterisation project is about a groundwater web-portal for New Zealand. I need to collect, analyse, mash-up, visualise and share again a lot of different groundwater related data sets – and everything shall happen in the web

4. Little did I know before …The diversity of available content, formats and technologies is quite overwhelming – So I had to figure out where to start.

5. So my story is basically about discovery –a step by step exploration – to build something for the better of humanity (well, definitely for New Zealand)

6. And what helped me in this rather iterative and incremental process? Open Source software, especially Free and Open Source Software for Geoinformation Systems (FOSSGIS)

7. It worked for me for two reasons: If it didn’t perform, I could have a look inside and fiddle around with it to make it fit my demands – well, or I’ll just try something else, you know, freedom of choice.

8. And if you even want to give back to the community, you often can propose and contribute enhancements to make such software better – however all things optional

9. Alright, second thing, data sharing - Let’s have a look what’s happening out there in the vast expanses(expansiveness) of the internet.

10. So there is the Open Geospatial Consortium, the OGC, is an international consortium of 482 companies, government agencies and universities participating in an open consensus-based process to develop publicly available standards that "geo-enable" the Web, and thus fostering interoperability. Interoperability is key.

11. They create specifications for data formats and web services and so forth, to interact, integrate and communicate with each other. You can download the full specifications for free and again, you could also participate.

12. So now, does that OGC stuff work with open source? Bamm, there’s a huge open source software ecosystem supporting OGC standards –  go to osgeo.org or 52north.org and you will likely find everything you need to start.

13. And that’s actually really high quality software, They are often even the reference implementations for particular OGC standards - and you still can do with it whatever you want.

14. Ok, I don’t want to pull out what’s happening internationally, you know, Aussie, North America or Europe … Well, I learned New Zealand is just really awesome, too

15. So NZ has a Geospatial Office that published the NZ Geospatial strategy in 2007 which actually says “USE OGC STANDARDS” and make environmental and other spatial datasets available.

16. And boom, agencies, research institutes and even Kiwi-based commercial companies open up massive NZ datasets to the public – accessible through OGC interfaces and open source – OPEN STANDARDS work

17. So data and technology was available. I could start with small steps, be open, be agile and flexible. I really felt enabled to discover, grow and share in return.

18. So how can you facilitate that wealth of knowledge? I heard in New Zealand there is a GIS Masters programme available, is it covering OGC and Open Source software? I don’t know actually, but I’ve been asked to co-author an “Open GIS” module in another international GIS Masters (I could make it open access perhaps)

19. Well, having OGC and FOSSGIS at hand – you can conduct as well as support top-notch science and research, and also foster education of kids at school, students at uni even citizens at home with the same set of tools … communicate science and knowledge about this country and its beautiful nature that deserves protection and sustainable development

20. So when do you start? Thanks everyone, acknowledgements to GNS, AUT, eResearch NZ, MBIE/MSI

Tuesday, 11 August 2015

ResearchGate milestone and GSoC finale ahead

What an exciting start of this week:

Google is re-organising itself into Alphabet, Google Summer of Code 2015 announced 'Pencils down' date in two weeks and ResearchGate informed me about reaching 200 publication downloads.


Well, being in the 3rd year of my PhD the number of publications, views and citations might not be outstanding but it is continuous progress along the early researcher path and a form of acknowledgement.

The GSoC project work with GeoTreliis and Azavea is also great opportunity to get more involved with Big Data technologies like Apache Spark and Cassandra and cloud technologies. The support from the project team, notably Rob and Chris, but also vibe on the GeoTrellis Gitter channel is fantastic.

Although it's time to wrap up GSoC in the next weeks, it is also the starting point of getting these new software development insights applied to my research in the SMART aquifer characterisation (SAC) programme, where I develop tools and web platforms for the SMART data portal. If all goes well, the SMART data portal will be overhauled end of this year and my PhD completed early next year :-)

Friday, 8 May 2015

Google Summer of Code 2015 with GeoTrellis, Cassandra and Spark!

Awesome news :-) Found this in my inbox:

<snip>
 Congratulations! Your proposal 'GeoTrellis: Cassandra Backend to GeoTrellis-Spark' submitted to The Eclipse Foundation has been accepted for Google Summer of Code 2015.

Welcome to Google Summer of Code 2015! We look forward to having you with us.

With best regards,
The Google Summer of Code Program Administration Team
</snip>


So, what is this about (Full proposal) ?

Cassandra Backend to GeoTrellis-Spark


1. Introduction


GeoTrellis is a Scala-based LocationTech project that is a framework for fast, parallel processing of geospatial data. Recent development efforts have allowed GeoTrellis to give the Apache Spark cluster compute engine geospatial capabilities, focusing on the processing of large scale raster data.

GeoTrellis's recent integration to with Apache Spark currently supports Hadoop HDFS and Accumulo as backends to store and retrieve raster data across a cluster. Cassandra is another distributed data store that could provide a rich set of features and performance opportunities to GeoTrellis running on top of Spark. It's also a popular distributed data store that a number of people interested in doing large scale geospatial computations are already using. A prototypical GeoTrellis catalog implementation for raster data in Cassandra is in development, yet it doesn't filter in the way we need.

This project would improve the GeoTrellis Catalog implementation for Cassandra, which allows us to save and load raster layers as Spark RDD's, as well as metadata. An important factor for distinction of GeoTrellis to other geospatial libraries the focus on performance. A performance-based indexing scheme needs to be integrated for being able to do spatial and spatio-temporal queries against Cassandra data as fast as possible. Eventually we will also be storing vector data in these data stores, and this project should support the efficient storing, indexing and retrieving of vector data with high-performance spatial and spatio-temporal filtering as well.

2. Background


GeoTrellis is Scala and Akka based high-performance geospatial processing framework. With the linking to Spray/Akka-Http GeoTrellis functionality can easily be exposed via Web Services, and with the latest integration with Apache Spark, GeoTrellis can now be run in massive large scale big data environments. Originally GeoTrellis had an own raster file type and a native catalog implementation that allowed for fast tile access, filtering and additional metadata. Now GeoTrellis also supports the widely used open GeoTIFF format.
To efficiently read, ingest, process and write data on Spark, data sources need to be exposed as SparkRDDs (Resilient Distributed Datasets). This allows to use Spark's great native algorithms to widely parallelise data processing and manipulation. GeoTrellis as of now supports the Hadoop HDFS filesystem for distributed file access and storage, and the Accumulo database for the GeoTrellis catalogue and advanced raster array access. A Cassandra raster integration is in an early stage development, yet filtering and indexing need to be improved as well.

3. The idea


Apache Cassandra is zero-dependency distributed NoSQL schema-less data base. It doesn't support referential integrity or joins like a relational database. It also has no spatial capabilities. With GeoTrellis on top of Cassandra this high-performance data store could be used in massively big data application with spatio-temporal requirements. There's is an active development community around the Apache Spark Cassandra connector, and the use of Cassandra data stores in the big data framework Spark can almost be considered main stream production.

GeoTrellis has its roots in high-performance raster data processing, and as rasters are in fact basically arrays of values with spatial and non-spatial metadata, GeoTrellis has its own (metadata) catalog implementation and can flexibly support an arbitrary variety of data stores for array data. Recent developments are prototyping raster storage and retrieval with Cassandra and vector support is planned. As Cassandra doesn't support any spatial indexing on its own (as of now), a high-performance indexing scheme needs to be implemented to be able to do spatial and spatio-temporal queries against Cassandra data as fast as possible. Here the vector and raster-based indexing and filtering methods against Cassandra and the GeoTrellis catalog need to be optimized for vector data in general and Cassandra in particular. Cassandra does support custom indexes on columns by reference to (presumably Java-based) an Index-class. Here typical discrete spatial indexes (in reconciliation with GeoTrellis catalog) like R-Trees, QuadTrees or possibly Space Filling Curves might be directly added to Cassandra to support spatial data indexing. The raster indexing/filtering is dependent on the GeoTrellis raster/array ingestion and referencing via the GeoTrellis catalog (documentation is a bit sparse, so I could also support in documenting the functionality in the course).

4. Future ideas / How can your idea be expanded?


There are several other great GIS, hydro-climate and geo-science (FOSS4G) toolkits that are based on Java and it would be great, if they or their functionality could also be subsequently be exposed and used under the GeoTrellis framework (like JTS is already, or the alternative GeoTIFF reader), particularly under large scale parallel processing capabilities on top of Apache Spark. This would allow for big data (business as well environmental science) applications under an enormously powerful framework.

Many scientific codes have been and are still developed and coded in Fortran, mainly because if its superior fast and efficient Matrix manipulation functionality. Yet, parallelising Fortran is hard, and in fact only viable on actual HPC facilities. With Scala, Akka and Breeze (or directly Spark and Java native BLAS...etc) parallelising scientific codes should be nicely scalable across commodity cloud servers. Maybe it could be evaluated at some point how to easily port popular Fortran codes to Scala/Spark and run them in unprecedented simplicity in main stream IT/server/cloud deployments at very comparable rate of performance.



About Google Summer of Code

Google Summer of Code (GSoC) is a global programme that offers stipends to students to write code for open source projects.  Google works with the open source community to identify and fund exciting projects. Alexander Kmoch, a student supported by the SMART Aquifer Characterisation programme (SAC), has been accepted as one of 1051 students in this year’s Google Summer of Code programme. Alex will help to improve the GeoTrellis database implementation for the cloud database Cassandra to allow processing of raster layers and vector data via the fast cluster engine for large-scale data processing Apache Spark. The GeoTrellis software will also be incorporated into the groundwater data portal that is being developed through the SMART project.

Thursday, 26 March 2015

ZOO-Project WPS Java-API and JGrasstools Java Hydrological Toolbox


The ZOO-Project (http://zoo-project.org/) is a solid Open Geospatial Consortium (OGC) Web Processing Service (WPS - http://www.opengeospatial.org/standards/wps) standard server implementation with an open flexible API that works well with many different programming languages. The Java bindings have never been tested in advanced configurations and complex data types, and to date only implement the minimum necessary interfaces. The JGrasstools project is a modular processing library and its highly annotated nature makes it possible to adapt quite easily to other toolboxes. JGrasstools contains a wide variety of powerful and efficient GIS, hydrology and geomorphological tools and processes, that can be exposed to and used by other libraries and toolkits. One example has been the adaptation to the Geotools Process API. The JGrasstools project, as well as other java based projects (as JTS, Sextante or even Geotools) would benefit greatly from the possibility to be used within a web-enabled WPS execution environment, as well as being integrated with the open standards suite of the OGC.  Some time ago Moovida tried integrating the JGrasstools libraries with the ZOO-Project Java binding to expose them as native WPS processes. This would allow them to work inside the ZOO-Project and serve its modules under the WPS standard.

Some Background


The ZOO-Project WPS implementation is a flexible, modular high performance HTTP CGI implementation. ZOO-Kernel is a powerful server-side C Kernel which makes it possible to manage and chain Web services, by loading dynamic libraries and handling them as on-demand Web services. The ZOO Kernel is written in C language, and supports several common programming languages in order to connect to numerous libraries and models (http://zoo-project.org/trac/wiki/ZooWebSite/ZooKernel). The generic ZOO API is basically accessible for every possible programming and web scripting language that can be run under the CGI interface. Main API implementations, the ZOO services, are available for C/C++, Python, JavaScript, PHP, Fortran and Java. Some API bindings are more advanced and complete and make the full ZOO-API (http://zoo-project.org/trac/wiki/ZooWebSite/ZOOAPI/Classes#ZOOAPIClasses) accessible to the ZOO service in the particular programming language (e.g. C, Python or JavaScript). In comparison the the Java API binding only exposes the minimum functionality to run from the ZOO Kernel.

JGrasstools (http://moovida.github.io/jgrasstools/) is a powerful GIS toolkits with functionality reaching from standard geoprocessing algorithms to advanced processing features used in hydrology and geomorphology. JGrasstools is based on a Maven (http://maven.apache.org/) build process, which takes care of dependency resolution and creating the succinct jar packages with the compiled classes. Maven is a defacto standard for managing (source and dependencies) building and deploying (jar packaging, resources, copying, publishing, archiving and installing) Java-based software projects. JGrasstools is also used as a toolbox in the uDig desktop GIS software (http://udig.refractions.net/). If JGrasstools could be exposed via a open standards and interfaces, web-based processing and execution environment (like ZOO-Project provides) it can be widely used in WebGIS deployments and large scale cloud based processing chains.

The next level

Andrea from Moovida said he didn't have enough time to continue developing this idea. He developed a generator which would programmatically scan through the annotated JGrasstools modules and generate respective ZooJavaWps classes per JGrasstools module/method and the corresponding ZOO-Project .zcfg config file. The only struggle I had was getting the CLASSPATH properly set up, as the ZOO-Project is basically an HHTP CGI application which will start a JVM per request. When I picked up on this in preparation for a GSoC proposal, I found that there were a few little botches with the parameter mapping from ZOO Java API into the very nicely annotated JGrasstools methods. So I took one example generated (WPS-ified) JGrasstool process and adjusted the parameter mapping and got it running with ZOO-Project. Additionally I adjusted the JGrasstools Maven config files to download the necessary dependencies in the target folders to copy them collectively in the ZOO-Project Java CLASSPATH.

Unfortunately I also didn't have time to drive this further still. However, it is just soooo close really :-) Alternatively a 52North WPS implementation based on the super practical JGrasstools annotations and Andrea's generator would also be relatively straightforward.

Sunday, 15 September 2013

IAH 2013, Perth, Australia




This presentation provides a great overview of my PhD research.
(presented at IAH 2013, Perth, Australia)

Sunday, 15 July 2012

First X3D example to visualise geological layers in the web

Recently I have played around with tools to visualise geological layers in 3D in the web, preferably without any browser plugins. With HTML5 and WebGL some really cool possibilities arise. WebGL is not supported by every browser, but apparently  all newer cool browsers like Chrome or Firefox, as well as Safari and Opera support WebGL at least experimentally. What a surprise that Microsoft Internet Explorer does not net yet support neither HTML5 canvas nor WebGL. But luckily there is the Chromeframe plugin :-)

If anyone has ever been working with OpenGL for 3D stuff probably in C or C++, well, WebGL is quite arcane, too, but in JavaScript ^^

Nevertheless, X3D for the rescue. X3D is a) an ISO standard (ISO/IEC 19775-1.2:2008 ), b) the successor of the working, but not really successful VRML97 (ISO/IEC 14772-1.2:1997) and c) a fully XML-based scenegraph declarative language. And the final ingredient is the Fraunhofer IGD experimental open source framework x3dom that thrives to integrate X3D content into HTML5.

Well, to the actual task. I am supposed to visualise a 3D geological model that originally has been designed with EarthVision(c). EarthVision(c) has its own binary format to store the 3D models, but they van be exported to a simpe XYZ-ASCII file:

2696105 6047205 150 20 11
2696605 6047205 150 21 11
2697105 6047205 150 22 11
2697605 6047205 150 23 11
2701105 6047205 150 30 11

...

The first two columns are Easting and Northing - implicitly known that the spatial reference system is New Zealand Map Grid. Third column is the height value and the further columns represent some additional attribute data. Based on the resolution, the surfaces from the roundabout 20 km  by 30 km range from 50KB (500m), 1MB (100m) to 20MB (20m) per layer (5 layers altogether).
X3D provides two easy (point set based) possibilities to show surfaces (right now the geological layers are represented as surfaces, they are not described as full bodies).

Furthermore there are the NURBS and extrusion implementations, which describes surfaces through splines. But that's rather complicated for the first shot :-) ElevationGrid requires an evenly spaced grid of height values, whereas IndexedFaceSet  notes all point coordinates and then defines coordinate indexes to define a mesh of single surfaces (similar to TINs, but not necessarily triangles).
I decided for the ElevationGrid. X3D has a geospatial extension (X3D Earth), which can geographically reference and place 3D objects in a defined spatial reference system. I didn't try this feature yet. And apparently it does not make sense to load 100MB for a 3D model into the browser. Therefore the 500m grid has been used here. The following example outlines a rather simple definition of such an elevation grid in X3D:

<shape> 

    <elevationgrid colorpervertex="false" creaseangle="3.14" def="Greywacke_top_500" 
        normalPerVertx="true" colorPerVertex='false' xDimension='37'
        zDimension='44' xSpacing='500' zSpacing='500' creaseAngle='3.14' solid='false'
        height='
150 150 150 ...
           '>
    </elevationgrid> 
    <appearance>
        <material ambientintensity="0.1" diffusecolor="red" id="Greywacke" 
            shininess="0.2" specularcolor="lightred" transparency="0.0">
        </material> 
    </appearance>
<shape>

To prepare that grid, you need to take care of some things:

  • you need to know the extent and resolution of the dataset, from that you calculate and define the x- and zSpacing (how many values will be filled, because you only need the height values)
  • the coordinate system orientation of the X3D 3-dimensional space is probably from the 2/2,5 coordinate system from the source dataset
  • the source datasets only contained points with actual values, to fill the ElevationGrid properly, NODATA values need to added
Finally I got my data sorted on built a neat first little demo :-) There might happen some improvement, as we are intending to visualise wells, bores and other hydro(geo)logical data in such a scene.

Fig. 1: Preliminary (X)3D  model with five layers and a (not aligned) image as an underlayer

Monday, 30 January 2012

Hello World

"Hello World"

Obligatory, awkward first post :-)

I am going to post infos about stuff I am actually doing. That mainly includes my UNIGIS studies and the Master Thesis in particular. I am officially matriculated at the University of Salzburg, Austria. I am contributing to their research in the SMART project as a working student in New Zealand. My main interests and recently my main activities lie in the convergence zone of web technologies and GIS. Pretty cool stuff ^^

Cheers,
Alex