Showing posts with label Postgis. Show all posts
Showing posts with label Postgis. Show all posts

Wednesday, 7 February 2018

Info Session for MSc programme in Geoinformatics for Urbanised Society

Today we broadcast a webinar info session about the new MSc programme in Geoinformatics for Urbanised Society at the University of Tartu.

One of the recurring core topics in this new MSc curriculum is Data and GIS use in Urban Planning.

Until just a decade ago spatial planning and analytics projects had problems with getting enough data. But nowadays there is so much data available, that it is increasingly hard to make sense of it – because of the 3 V’s of big data - volume, variety, velocity. The open data movement, government agencies, research institutes and citizen scientists alike make more and more data available publicly, mobile phones, sensor networks and satellites generate a multitude of datasets every day

In order solve the Interdisciplinary challenges of urban planning we want to empower you with skills and knowledge to analyse, visualise and understand processes and data. For that we teach the Full cycle of spatial data management, from the various methods of data acquisition, followed by efficient and practical processing techniques, to subsequent meaningful analysis and visualisation; in order to consequently make successful planning decisions for a sustainable future.

So what does it mean to study Geoinformatics for Urbanised Society with us in Tartu?

You will learn how to combine geography and IT in the age of BIG data. This is essentially what we believe modern Geoinformatics is representing. Mastering Geoinformatics will provide you with tools to analyse social and natural processes in space for interdisciplinary decision- and policy-making.

Watch the whole recorded session for more info:



Links:

Monday, 21 January 2013

A side note on geospatial data sharing and spatial data infrastructures (SDI)

This blog post is dedicated to provide a general overview over the field of geospatial and environmental data sharing. The term geospatial is actually tautologous: The prefix “geo” implies geography, which always relates things to each other based on their location, where nearer things are stronger related than things further away from each other (Tobler’s 1st law of geography). And the word “spatial” also means having an extent and location in a space. However “geospatial” nowadays is almost exclusively used in the field of digital data with geographical context. Therefore software that delivers, analyses, presents, processes, stores and retrieves is also often called geospatial software (having its origins in the good old GIS – Geographical Information Systems).

Part 1: GIS and GI Science intro

Research areas that work on the science behind GIS and spatial data, on the analytical methods, processes techniques, on ways of (standardised) spatial data exchange, effective and efficient storage and retrieval, are called GI Science, Geoinformatics, Geomatics, Geocomputation, Spatial Computation or Spatial Science … There is actually quite some discussion, if the necessity, even an entitlement for such a dedicated mixed branch of computer science and geography exists, a similar discussion when geography emerged as an accepted field of research (ref). On the other hand you’ll often find the quote that “80% of data has a spatial component” or something like that. Apparently I can get the source right anymore, therefore you might handle this with care. It might go back to the 1990s when GIS software for PCs wanted to get their feet into the market (gis lounge). However a lot of data in the geosciences have a spatial context and - a lot - of those data are needed for governmental agencies to manage land and water resources properly :smile: Agreed?

Part 2: The Opengeospatial Consortium, aka OpenGIS, aka OGC

“The Open Geospatial Consortium (OGC) is an international industry consortium of 479 companies, government agencies and universities participating in a consensus process to develop publicly available interface standards. OGC® Standards support interoperable solutions that ‘geo-enable’ the Web, wireless and location-based services and mainstream IT.”
The OGC standards framework provides means to build a spatial Data infrastructure (LINZ - Land Information New Zealand) – which is “the technology, policies, standards, and human resources necessary to acquire, process, store, distribute and improve the usability of geospatial data. (SDI) facilitates the connections between these important sources of information, and allows people to find and access them.” Quite some of the OGC standards and web services are also ISO international standards. There are interface and service descriptions on the one hand and data encodings/formats and conceptual data models on the other side. I will provide a short summary:

WMS – Web Mapping Service (ISO 19128 WMS v1.3.0)

Essentially provides (web) maps (as images like png or jpg) output data Geographiclly correct images, png, jpg, view or portrayal service
Major methods: GetCapabilties and GetMap

WFS – Web Feature Service (ISO 19142 WFS v2.0)

Provides an interface to access, query, store and retrieve vector “features”, aka discrete data – like in ESRI shapefiles. Data is accessible by their data schema, which can be soft-typed and values in schema fields and location queries are utilized. Output GML (ISO 19136), which is in a particular “domain-specific” XML schema. With WFS-T – transactional – there is also support to write back to the WFS server.
GetCapabilities GetFeature – get the data DescribeFeatureType – get schema

WCS – Web Coverage Service

Provides an interface to access, query and retrieve raster imagery and coverages, grids (aka “fields”) as in continuous data. eg NetCDF-CF, GeoTIFF, ArcGRID
GetCapabilities DescribeCoverage GetCoverage

CSW – Catalogue Service for Web (ISO 19115 CSW 2.0.2)

Provides an interface to access, query, store and retrieve metadata, aka data about the geospatial data, which is accessible through other geospatial webservices. Output is usually XML ISO 19139 metadata or Dublin core.
GetCapabilities GetRecords – find metadata record by search criteria GetRecordById – get one record by its unique id DescribeRecord – metadata type GetDomain – get range of values and/or keywords

SOS – Sensor Observation Service

Provides an interface to access, query, store and retrieve time-series based data that has been measured at locations, eg through sensors or field surveying/sampling. But the focus is to query on temporal and then on spatial or value comparison basis. Standard output formats are O&M, WaterML2.0 time-series and SensorML sensor/procedure metadata.
SOS is part of the sensor web enablement initiative (SWE) which advances to its version 2, where a lot of things become more flexible, but also moe complicated. I will write about that later ☺ SOS explicitly also describes a group of methods to insert sensor and observation data. This is handled through different profiles.
GetCapabilities GetObservation DescribeSensor
There are quite some commercial and Open Source software packages and tool kits available, for the desktop and server-based for the web that support or where explicitly written for OGC webservices and splendid Open Source resources in the web:
OSGeo Foundation: http://www.osgeo.org

(Spatially enabled) Databases:

  • Postgresql/Postgis
  • MySQL
  • Oracle Database
  • Microsoft SQL Server
  • ESRI Geodatabase / ArcSDE
  • SpatiaLite
  • Rasdaman
  • GeoCouch

Data Servers (store data in databases):

  • Geoserver (WMS, WFS, WCS)
  • Mapserver (WMS, WFS, SOS)
  • 52°North SOS server
  • Geonetwork (CSW)
  • ESRI ArcIMS / ArcServer
  • Thredds (WCS, netCDF)

Web mapping tool kits / frameworks (take data from data servers):

  • openlayers
  • Mapbender
  • MapFish
  • Geomajas
  • Flash
  • Silverlight

Desktop clients supporting (at least partially) OGC standards:

  • QuantumGIS
  • uDig
  • ESRI ArcGIS
  • Intergraph Geomedia
And on it goes … in the next weeks, I will write about New Zealand and international examples of OGC webservices implementations and SDIs for geospatial data publishing (where the data is basically public domain and needs to made accessible) and provide some closer insights to OGC SWE and the next generation sensor networks initiative SWE 2.0 et al.

Thursday, 5 April 2012

Head in the clouds

Head in the clouds


My supervisor gave me the opportunity to establish a demo webgis in "the cloud". So I worked out, what all this fuzz with cloud computing is about and initialised an Ubuntu image in the Amazon EC2 (Amazons Elastic Computing Cloud).

Installation and Basic Configuration

  • Login to amazon web services: http://aws.amazon.com/ec2/ 
  • Login to security access for accessing X509 key: http://aws.amazon.com/security-credentials
  • Select AMI, launch instance EC2
  • ubuntu 11.10, t1.micro, "dev-instance", your key pair
  • created some rules that allow for SSH, web, FTP and a ftp passive port range/ reflected in /etc/proftpd/proftpd.conf for passive ports, if firewall issues prevent active ports, typical ftp issue
  • instance home and availability zone in Singapore, test for performance between Europe and NZ
  • login with user ubuntu instead of root with the private key, it’s ok, just use sudo for root commands
  • http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/putty.html does work, converted to putty private key

Software
  • Apache and Tomcat6 with Geoserver
#> apt-get update
#> apt-get install apache2 
#> apt-get install openjdk-7-jdk (for geoserver etc)
#> apt-get install postgresql-9.1-postgis postgis postgresql-9.1
#> useradd -m geoserver (dedicated non-root-user for geoserver instance)
#> update-alternatives --config java (because I installed  openjdk7)
#> vi /etc/apache2/sites-enabled/000-default (password protection for complete site)
#> htpasswd -c /home/geoserver/includes/htpasswd sbg (http password for sbg user is that from amazon test user)
#> a2enmod proxy auth_basic authn_file proxy_http rewrite
  • configured proxying and password security for webserver in /etc/apache2/sites-enabled/default
  • configured tomcat security in /home/geoserver/tomcat6/conf/tomcat-users.xml (role manager-gui)
  • configured tomcat for localhost only in /home/geoserver/tomcat6/conf/server.xml (address=127.0.0.1)\
  • deployed geoserver.war in /home/geoserver/tomcat6/webapps
  • ftp daemon proftpd
#> apt-get install proftpd-basic
  • edited config file /etc/proftpd/proftpd.conf for virtual users under /vol1 and shared identity for easy file maintenance
  • ftp users and passwords in /vol1/conf
#> apt-get install libapache2-mod-php5 php5-pgsql  (for web php based postgre sql admin console)
  • enabled php in apache2
  • downloaded PhpPgAdmin, edited apache2 config again to include it
  • configured includes/phppgadmin/conf/conig.inc.php for localhost
  • Customised postgresql db for postgis and changed user password
#> su postgres -c psql template1
psql>  ALTER USER postgres WITH PASSWORD 'TheSecr3tP@ss';
psql>  CREATE DATABASE template_postgis WITH TEMPLATE = template1 ENCODING = 'UTF8';
psql>  \i /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql
psql>  \i /usr/share/postgresql/9.1/contrib/postgis-1.5/spatial_ref_sys.sql
psql>  UPDATE pg_database set datistemplate = TRUE where datname = 'template_postgis';
psql>  grant all on geometry_columns to public;
psql>  grant all on spatial_ref_sys to PUBLIC ;
psql>  create database my_geoserver_data_store with template = template_postgis;
psql>  \q
#>
#> passwd -d postgres
#> su postgres -c passwd
#> 

Some Sysadmin considerations

Take care, terminating a standard instance running from public AMI will be gone with all the data!!! So to be more flexible, I added a EBS volume (Elastic Block storage, independent from the AMI), formatted it and mounted it to /vol1 and made changes to the typical directory layout.
  • /vol1   (supposed to be used as a mount point when on different file system, so file system is almost easily movable)
  • /vol1/www  (base for web stuff, also ftp entry point/root )
  • /vol1/www/logs  (log-dir from apache2 and ftp daemon)
  • /vol1/www/htdocs (general Webserver Document Root)
  • /vol1/www/includes (some additional web stuff/libraries that should not directly be accible via HTTP, like phppgadmin)
  • /vol1/www/tmp (temp dir for user and apache2 stuff)
  • /vol1/www/geoserver (geoserver directory)
  • /vol1/conf (some config  and password files, for apache2 and ftp)
  • /vol1/data (data directories of postgres db moved to /vol1/data/postgresql and symlinked to /var/lib/postgresql (origin) )
  • main apache2 config moved to /vol1/conf/apache2/smartsite1.conf and symlinked to /etc/apache2/sites-available/
  • main proftpd config moved to /vol1/conf/proftpd/proftpd.conf and symlinked to /etc/proftpd/proftpd.conf (origin)
Pros of virtualisation

To be safe I created snapahots and registered the AMI as a private one, because config options under /etc and the installed programs are already available.
  • created one snapshot each (running AMI, and /vol1 EBS volume) wrote some descriptive tags to them
  • from snapshot of the root device I created an image (AMI) to start a new instance from
  • from the vol1 snapshot I created a new volume, to attach to the new instance
  • added some descriptive tags to the AMI and the new volume
  • we now have a kind of useful base AMI and base volumes
  • probably I should check the mount options, maybe the new instance does not react
  • by attaching the cloned volume and reboot in the AWS webconsole (reboot works)  does the system initialise properly
Cheers

Designing the big picture

Designing the big picture


After literature review and further research and gathering a lot of information about state-of-the-art technology and paradigms, I designed a bigger concept of a possible, full-fledged WebGIS that is more than capable to support the data-related and computational needs of the research of the SMART aquifer characterisation program. after writing it down, we already got a paper to submit to the GI_Forum Geoinformatics conference.
.

SMART Portal archtitecture concept

As shown in the picture, the whole portal / WebGIS is based on standardised interfaces, mainly the OGC geo-spatial web services. as a requirement from the New Zealand Ministry of Science and Innovation all software produced (and used) should be Open Source, the main building blocks for the portal will be tested and evaluated in the prototype, I am going to develop:
  • PostGres/PostGis - data storage, spatial database
  • Geoserver - for WMS and WFS (subsequently WCS, too)
  • OpenLayers as web mapping client
  • 52North SOS Server, customise DAO backend to serve data from an existing legacy DB
  • Thredds server to serve netCDF coverage data
  • X3D as 3D file format to display 3D geological model in the browser
  • Geonetwork - CSW, metadata, to connect to upstream NZ spatial data infrastructure

Groundwater Resources in New Zealand

Data Synthesis and visualisation for groundwater resources in New Zealand


“As a conservative estimate, better understanding of groundwater resources could lead to improvements in water management with potential to prevent social, cultural, environmental and economic losses of $1 billion or more, about 1% of New Zealand’s GDP. Understandably, central and regional government has identified improved characterisation of groundwater resources as a top research priority.”
The SMART project is about characterisation of New Zealand’s aquifers, to improve ground- and freshwater management (www.smart-project.info).


The main research aims of the SMART project

  • focus on novel, passive suite of methods to measure groundwater volume and changes in it over time
  • aquifer hydraulic properties
  • fluxes of groundwater interchange with surface waters
  • water age
  • with methods like ...
    • ambient noise seismic tomography
    • airborne geophysical surveying
    • satellite remote sensing
    • fibre optic temperature sensing
    • novel age tracers
Outcomes shall made available via a web portal and harmonised 3D groundwater database that will meet stakeholder needs for open access, ease of use, and interoperability with existing systems. Furthermore the integration of existing data is crucial.

My Thesis Topics

I contribute in the field of “data synthesis and visualisation” in the form of a prototype 3D/4D WebGIS, data processing and web services to provide harmonised hydro(geo)logical data from multiple sources. Within the works for my Master Thesis until June 2012, I identified following milestones:
  • stakeholder consultation - groundwater workshop, visited HBRC and EW
  • gather and analyse (sample) hydro(geo)logical datasets from wells, boreholes or pump tests, managed in systems/databases like GGW/NGMP (GNS), Hilltop and Wiski (very popular among regional councils) – Feb 2012
  • find a common data scheme/data model, migrate and/or transform data online and provide via web services to connect to a (NZ) SDI or integrate/link in web portal for easy access – April 2012
  • identify appropriate technology and standards (ISO, OGC, OASIS … ) 3D/4D visualisation in the web, as aquifers are inherently 3D structures plus time (water time series data -> 4D) – June 2012
The prototype implementations of this thesis will be in test mode, accessible by designated staff and stakeholders. The research about “data synthesis and visualisation” including the topics of this Master Thesis will be continued in more detail by a PhD student from July 2012.
We focus on hydro(geo)logical data. The first steps are about accessing multiple existing sources of data and find a way to publish them via web services (most likely OGC).

Major Challenges include

  • complexity of water-related data – many parameters and measurement properties, parameter naming, units etc.
  • data policy issues (ownership, restricted access ...)
  • data quality and quality and assurance issues – uncertainty, hint/description in metadata
  • as existing data often is not accessible outside their environments (GNS network, regional councils’ local data storage), or mostly via manual file-based email/ftp interchange or as “pdf”-reports, the publishing of existing data from multiple sources is anticipated via the web e.g. via WMS, WFS, WCS, WaterML2.0?, SOS, download (shapefile, raster coverage) to allow easy recombination, correlation and analysis at national scale (surpassing regional boundaries)
  • find a common data scheme, probably “just” implement a software to access databases and deliver WaterML2.0?! Good idea might be to transform online (e.g. XSLT/WPS web service)
  • provide infrastructure, for storage, access, delivery and manipulation of different geodata (vector, raster, time series …)
  • Prepare catalogue and provide metadata in standard encodings (XML, OGC web services, ISO 19115/19139 metadata, CSW, CS-W ebRIM/ISO)
  • Regarding exisiting methods and infrstructures, like Auscope / SISS (AU), GeoNet (NZ), GENESIS FP7 (EU), CUAHSI (North America)
  • Think about GeoSciML for description of geology and geometry, as WaterML is more about time series and measurement data prototypical web 3D/4D map/aquifer visualisation (X3D earth ...)

Monday, 30 January 2012

Hello World

"Hello World"

Obligatory, awkward first post :-)

I am going to post infos about stuff I am actually doing. That mainly includes my UNIGIS studies and the Master Thesis in particular. I am officially matriculated at the University of Salzburg, Austria. I am contributing to their research in the SMART project as a working student in New Zealand. My main interests and recently my main activities lie in the convergence zone of web technologies and GIS. Pretty cool stuff ^^

Cheers,
Alex