2017 LA Conference on diffuse pollution

It was an awkward conference - not knowing anyone, but also, really experiencing America first hand, in LA. Presenting went well, with barely connecting to the audience, a comparatively anonymous dinner.

UCLA has a pompous campus, it was nice weather, dry and warm. Comparing it to other conferences like EGU, or ANZ Hydrological conferences, Hydrogeology Association and Digital Earth summits, that conference was a strange experience.

OGC OWS GeoJSON

We are implementing OWC GeoJSON in the SAC groundwater portal as part of the SAC research program. In the latest OGC OWS Context GeoJSON Encoding Standard 14-055r2 we came across a few ambiguities that we couldn't solve for ourselves immediately.

I am summarising those subsequently here. The examples in the OGC Schemas repository are mostly still based on the former OWS-10 testbed results. Hut the OWC working group announced to update the official samples.

7.1.1 Class OWC:Context:

specReference: <xz>.properties.links.profiles requires the Specification Reference (requirements class) identifying that this is an OWC Context document and its version. Table 1 defines the data type as an Array of links profiles (as defined in Table 10 - data type "links") where one element SHALL have the href value “http://www.opengis.net/spec/owc-geojson/1.0/req/core”

But the multiplicity defines: One (mandatory) and the GeoJSON example 7.1.1.1 shows another different picture, which is only an array of String.

"links" : {
 "profiles" : [
  "http://www.opengis.net/spec/owc-geojson/1.0/req/core"
 ], ...
},

We implemented links.profiles as an array of links as objects of data type links from 7.1.10, table 10 with href, type, lang, title and length. And one mandatory link object element is required that has the specified core profile url.

Is that the desired behaviour?

Secondly, we found a quirk with the "<>.properties.links.via" description.

In 7.1.2 Class OWC:Resource "contextMetadata" is an Array of links as defined in Table 10 (Zero or more), and the example shows a JSON array. all good so far. However in OWC:Resource it is "resourceMetadata" also a "<>.properties.links.via" but explicitly call "Link object" (whereas links.data, links.previews and links.alternatives are Array of Link objects). But then in the column multiplicity it states "Zero or more" as with all the other link object arrays? But the example explicitly shows a single link object under the path "<>.properties.links.via":

"via" : {
"href" : "http://www.acme.com/products/algal20090123090856.xml",
"type" : "application/xml",
"length" : "435",
"title" : "XML metadata file for the entry 2009-01-23 09:08:56"
}

This is confusing. What is the intended behaviour? We chose an array of link objects to mimic the other link object arrays general behaviour. Is that appropriate?


Finally, there seems to be an exceptionally different behaviour for "7.1.1.9 creator". In OWC:Context there is only mentioned the path: <xz>.properties.creator

with a presumably wrong and irrelevant example:

"creator" : "ACME CSW Server”,

Because the abstract 7.1.7 DataType OWC:Creator is later typed into specific:

- owc:CreatorApplication - <xz>.properties.generator
  (as defined in Table 8)

- owc:CreatorDisplay - <xz>.properties.display
  (as defined in Table 9)

But the actual issue that I mean is that in 7.1.7 DataType OWC:Creator specifies at first a generic extension (without a JSON path, because it is abstract) - then extension for <xz>.properties.display.* in Table 9 - Definitions of owc:CreatorDisplay elements is explicitly described BUT COMPLETELY ABSENT for <xz>.properties.generator.* in Table 8 - Definitions of owc:Creator/OWC:CreatorApplication elements.

That seems either accidental or the reason is not obvious. We chose the freedom to allow for an extension in OWC:CreatorApplication the same way the extension is declared for ALL other data types.

If I can be of any help to improve the standard or provide material or examples, please don't hesitate to contact me. I would be happy to join an OWC Context working group if possible.


- Our GitHub repository: https://github.com/ZGIS/smart-owc-geojson
- OpenHub Open Source statistics: https://www.openhub.net/p/smart-owc-geojson
- Bintray Maven/Ivy artefacts download: https://bintray.com/allixender/ivy2/smart-owc-geojson

Looking forward to support the use of OGC OWC Context for data exchange in the geospatial web.

Alex

Hydrological Society Conference - Web-based real-time processing of environmental measurements

This case study was presented as poster abstract.

Kmoch, A., White, P. A., & Klug, H. (2015). Sensor Observation Service and web-based real-time Processing of environmental Measurements in the Upper Rangitaiki Catchment (Poster). In The NZ Hydrological Society Conference 2015, 26th November, in Hamilton, New Zealand

Abstract:

Environmental assessments naturally depend on field observations and technological advancements. , such as tTelemetry, allow the automated collection, transmission and processing of these measurements. However, modelling of natural processes is typically a complex challenge and involves applying expertise of scientists as well as a host of data preparation steps (White, 2006, White et al., 2003).
In addition, automation of model execution with the most recent observation data is dependent on the integration of the data collection, storage and processing elements (Klug and Kmoch, 2014). This paper demonstrates a system that integrates a Sensor Observation Service (SOS) that includinges field observations and internet-based environmental data with a rainfall recharge model that allows near-real time calculation of rainfall recharge in the Upper Rangitaiki catchment, Bay of Plenty region.
The SOS specification is an Open Geospatial Consortium (OGC) standard for the open and standardised integration of environmental sensors into an internet-based environmental data infrastructure (Klug and Kmoch, 2015, Klug, Kmoch, Reichel, 2015).

Figure 1. Process of data flow from field site sensors, to SOS data service to a simulation model process

Results:

• We showed that it is possible to link the collected data directly to a simple rainfall recharge model (Figure 1)
• The low cost sensor and circuit board instrumentation collects data and forwards them to the field computer in 10-minute intervals via robust, low power, ZigBee wireless protocol
• The field computer running a standard Linux operating system, transfers observation data in 10 minute intervals via a 3G mobile data connection to an online SOS server.
• From the service the observations are available in a standardised open format.
• A website can access the raw data from the SOS server and plotted data points within 5-10 minutes of field measurement
• A rainfall recharge model runs with the latest data points from the online SOS server.

References:

Klug, H., & Kmoch, A. (2014). A SMART groundwater portal: An OGC web services orchestration framework for hydrology to improve data access and visualisation in New Zealand. Computers & Geosciences, 69(0), 78–86. http://dx.doi.org/10.1016/j.cageo.2014.04.016

Klug, H., Kmoch, A. (2015). Operationalizing environmental indicators for real time multi-purpose decision making and action support. Ecological Modelling, 295, 66-74. http://dx.doi.org/10.1016/j.ecolmodel.2014.04.009.

White, P. A. (2006). Some Future Directions in Hydrology. Journal of Hydrology (NZ), 45(2), 63–68.

White, P. A., Hong, Y.-S., Murray, D. L., Scott, D. M., & Thorpe, H. R. (2003). Evaluation of regional models of rainfall recharge to groundwater by comparison with lysimeter measurements, Canterbury, New Zealand. Journal of Hydrology (NZ), 42(1), 39–64.

Klug, H., Kmoch, A., & Reichel, S. (2015). Adjusting the Frequency of Automated Phosphorus Measurements to Environmental Conditions. GI_Forum 2015 - Journal for Geographic Information Science - Geospatial Minds for Society, 1, 590–599. http://doi.org/10.1553/giscience2015s590

AGILE 2016 - SensorWeb Semantics on MQTT for responsive Rainfall Recharge Modelling

Integrating Wireless Sensor Networks (WSNs) and spatial data web services is becoming common in ecological applications. However, WSNs were developed in application domains with different sensor and user types, and often with their own low-level metadata semantics, data format and communication protocols. The sensor web enablement initiative (SWE) within the Open Geospatial Consortium (OGC) has released a set of open standards for interoperable interface specifications and (meta) data encodings for the real time integration of sensors and sensor networks into a web services architecture.
Such XML-based web services exhibit disadvantages in terms of payload and connectivity in low-bandwidth low energy unreliable networks, such as remote 3G uplinks. Monitoring stations deliver frequent measurements in real-time, but dynamic implementation of measurement frequencies, adapted to certain environmental conditions, are rarely implemented. We describe a responsive integrated hydrological monitoring prototype to calculate rainfall recharge for water management purposes.
When rainfall is observed, a threshold event triggers a reconfiguration task for the soil moisture sensors, using asynchronous, push-based communication implemented with an MQTT queue. A Sensor Planning Service commits that request via MQTT into the wireless sensor network, and updates the measurement frequency of the target sensors to gain higher resolution for the vertical soil water infiltration.
The system integrates a Sensor Observation Service (SOS) including field observations and internet-based environmental data with a rainfall recharge model that allows near-real time calculation of rainfall recharge in the Upper Rangitaiki catchment, Bay of Plenty region in New Zealand.

Figure 1: Setup and location of the sensor field site, central North Island, New Zealand

The prototype site comprises a main station conducting comprehensive measurements of meteorological, hydrological and pedological parameters. For the wireless data transmission within the local site installation XBee-PRO modules from the Digi Company  ZigBee IEEE 802.15.4 protocol are used. The main station receives continuous sensor measurements from the attached sensor units, and acts as the gateway to the online SOS and SPS services by providing the communication channel from the local sensor network to the web-enabled data management infrastructure.

The field site has been established in the Upper Rangitaiki catchment (Figure 1) and comprises a field computer (Raspberry Pi) with a direct internet link (GPRS/3G) and a sensor board (Waspmote) that has 12 typical meteorological, hydrological and pedological sensors attached (i.e., wind speed, wind direction, rainfall, 1x groundwater probe, 5x temperature and 3x soil moisture). The Raspberry Pi and Waspmote can be monitored and reprogrammed from an online server.

Figure 2: Raw sensor series visualized in a website from a SOS query.

The site setup allows scaling up to a multitude of low cost, low energy sensor stations throughout the catchment, with only one field computer that serves as data logger for backup. The observations were available in a standardized open format. The website accessed the raw data from the SOS server and plotted data points within 5-10 minutes of field measurement. This website was easily accessible via browsers and smartphones (Figure 2).

The paper was was presented at the 19th AGILE International Conference on Geographic Information Science, 15th of June, in  Helsinki, Finlkand.

Kmoch, A., Klug, H., White, P., & Reichel, S. (2016). SensorWeb Semantics on MQTT for responsive Rainfall Recharge Modelling. In 19th AGILE International Conference on Geographic Information Science. Helsinki.

Using Liquid Democracy for Water Resources Management: A Review

... and how to link governmental policy processes and geosciences via an interactive decision making platform for water resources co-management.

Initial Context - Case Study New Zealand

The responsibility of management of natural resources, in particular water, is typically delegated to the Regional Councils by the Resource Management Act. Decisions regarding water management are regulated through the National Policy Statement for Freshwater Management. However, those decisions need to be backed by thorough science and in consultation with all stakeholders, be it Iwi, domestic, agricultural or industrial water users, or the general public in regards to recreational services that water resources provide.

Current policy and management decision processes follow a rigid procedure, 1) the science to understand the resources, 2) a consultation process (if at all) about the plan how to manage the resource and 3) the development of a long term strategy and policy decision.

The main concern that I’d like to address is the agility of that process. These steps follow the traditional waterfall project management model, which probably is due to the limitations of current tools. Scientific models that aim to characterise the availability and of natural resources and dynamics of environmental process are developed, possible impact assessed and then described in reports for further use. The data and assumptions utilised in the research process are limited snapshots in time, dependent on the quality of data collection and curation processes, and scope and depth vary with available budget. Based on the information from these reports resource managers discuss strategies how to manage the water resources in reconciliation with demands. The consultation with users is again a cost- intensive process, because it is time-consuming. Furthermore, assumptions e.g. limits or local/regional differences in water allocation for the modelling process cannot be changed flexibly.

The consultation process therefore seems limited to very few pre-decided scenarios, which might not have considered all important stakeholders (who are all important stakeholders anyway?). Thus, a final management and policy decision is often not satisfactory.

I would like to propose more research into an agile, aka “liquid resource management system”. Recent advances in computer and web technologies for example allow dynamic exposure of datasets and scientific software. So it might be time to link data with models and a “delegated voting” mechanism into an online resource management feedback system. Such an online platform would provide the capabilities to run different scenarios transparently within a democratic discussion forum. Through delegates stakeholders can have their interests represented in a co-management approach which allows a close link of the resource managers with the affected communities while having the current science at hand.

Background Liquid Democracy

Probably one of the first explicit thoughts on Liquid Democracy originate from Bryan Ford's draft named Delegative Democracy from 2002. Back then it was uncertain what scientific venue(s) it would be suitable for. Bryan unfortunately didn’t manage to get back to exploring and developing it in any rigorous scientific fashion.

• Delegative Democracy: http://www.brynosaurus.com/deleg/deleg.pdf

However, he published a well cited informal blog post revisiting the idea and pointing to some of the interesting developments since 2002:

• Delegative Democracy Revisited: http://bford.github.io/2014/11/16/deleg.html

The closest academic work Bryan referred to as a reasonably serious, rigorous exploration of the topic of any kind (and the only peer-reviewed and published work directly on the topic that by then was James Green-Armytage’s political-economic analysis:

• Direct Voting and Proxy Voting: http://jamesgreenarmytage.com/proxy.pdf

• Green-Armytage, J. (2015). Direct Voting and Proxy Voting. Constitutional Political Economy, 26(2), 190–220. http://doi.org/10.1007/s10602-014-9176-9

This paper took a first step at theoretically defining and analyzing the idea from a cost/benefit perspective, but it’s only a first step and leaves a lot of unanswered questions and issues, and of does not contributes to the empirical space.

Helene Landemore, a colleague of Bryan in political science at Yale, has been interested in this and related “collective intelligence/decision-making” topics for a while. She is working with Rob Reich and Lucy Bernholz at Stanford on some events in the near future exploring this and other related “digital democracy” topics. One might try reaching out to any or all of them as additional points of contacts with more experience in the political and social sciences space.

Further academic or applied works around Liquid Democracy can be found occasionally, for example:

• Blum, C. & Zuber, C.I., 2015. Liquid Democracy: Potentials, Problems, and Perspectives. The Journal of Political Philosophy, 24(August 2014), pp.6–9. Available at: http://doi.wiley.com/10.1111/jopp.12065

• Rodriguez, M. et al., 2007. Smartocracy: Social Networks for Collective Decision Making. In 2007 40th Annual Hawaii International Conference on System Sciences (HICSS’07). IEEE, pp. 90–90. Available at: http://public.lanl.gov/jhw/Jen/Publications_files/Smarto-Share.pdf

• Paulin, A., 2014. Through Liquid Democracy to Sustainable Non-Bureaucratic Government. CeDEM, 6(2), pp.216–230. Available at: http://research.apaulin.com/research/2014/cedem14-ld-snbg

• Boldi, P. et al., 2011. Viscous democracy for social networks. Communications of the ACM, 54(6), p.129. Available at: http://portal.acm.org/citation.cfm?id=1953154

• Zwattendorfer, B., Hillebold, C. & Teufl, P., 2013. Secure and Privacy-Preserving Proxy Voting System. 2013 IEEE 10th International Conference on e-Business Engineering, 0, pp.472–477

• Link, J.Á.B., Tchorbadjiiski, A. & Wehrle, K., 2014. Opportunistic Vote Delegation for E-Voting Based on Liquid Democracy. In Proceedings of the 6th Extreme Conference on Communication. Galapagos Islands: ACM. Available at: https://www.comsys.rwth-aachen.de/fileadmin/papers/2014/2014-bitsch-extremecom-liquid-democracy.pdf

• Behrens, J. et al., 2014. The Liquid Democracy Journal - Issue 1, Berlin. Available at: http://www.liquid-democracy-journal.org/.

• Behrens, J. et al., 2014. The Principles of LiquidFeedback. Interaktive Demokratie e. V. Berlin. ISBN: 978–3–00–044795–2, p.240

Liquid Democracy as an Integrating Technology Platform

Possibly the most prominent of the application of Liquid Democracy is the German Pirate Party [1], for which the Software Liquid Feedback was developed [2]. A Medium article describes the distinctive features [3]:

Liquid Democracy is a new form for collective decision making that gives voters full decisional control. Voters can either vote directly on issues, or they can delegate their voting power to delegates (i.e. representatives) who vote on their behalf. Delegation can be domain specific, which means that voters can delegate their voting power to different experts in different domains. This is in contrast with direct democracy, where participants are required to personally vote on all issues; and in contrast with representative democracy, where participants vote for representatives once in a certain election cycle and then never worry about voting anymore.

Coming back to the water sciences and data issues around the water resources management process. In the last decade so called geoportals evolved to integrated systems of systems, not only providing data, but also processing routines and visualisations of the processed geospatial data to support science and education as well as policy and decision making for particular environmental domains. From a scientific and data-centric point of view, it becomes an obvious choice to link the democratic processes with the data in an online platform, e.g. as suggested by Craglia & Shanley (2015), or in an elaborate study "When Water Becomes the New Oil" (Kwiatkowski & Höchli, 2016) from the Swiss Gottlieb Duttweiler Institute (GDI), an independent think tank based in Rüschlikon near Zurich, which frames this as a participatory process.

• Craglia, M. & Shanley, L., 2015. Data democracy - increased supply of geospatial information and expanded participatory processes in the production of data. International Journal of Digital Earth, 8(9), pp.679–693. Available at: http://www.tandfonline.com/doi/full/10.1080/17538947.2015.1008214

• Kwiatkowski, M. & Höchli, B., 2016. Wenn Wasser zum neuen Öl Wird Wie die Schweiz die Konflikte der Zukunft meistert, Zurich. Available at: http://www.gdi.ch/de/Think-Tank/Studien/Wenn-Wasser-zum-neuen-Oel-wird/727

Links

1. http://liquidfeedback.org/
2. http://techpresident.com/news/wegov/22154/how-german-pirate-partys-liquid-democracy-works
3. https://medium.com/organizer-sandbox/liquid-democracy-true-democracy-for-the-21st-century-7c66f5e53b6f
4. https://en.wikipedia.org/wiki/LiquidFeedback

SMART Groundwater Portal Dev going full "Cloud"

What a concise quote from Erik Dietrich, founder of DaedTech LLC:

Software developers demand the ability to work effectively from anywhere.  They have attained a coolness factor, and demand for them is so high that there is no need for them to guard their source code like squirrels preparing for winter.  GitHub is a good idea because it effectively captured what software developers really want and offered it to them pretty flawlessly.  GitHub is a zeitgeist that is taking over the world precisely because software developers are taking over the world and software developers really like GitHub. (source)

Although, I work at a research institute which is one half a commercial consultancy with IP to protect, on-going international research collaborations, and governmental research funding require us to be flexible, open and accessible.  I work as a research scientist/analyst programmer in a science department, not in the IT or applications department, and thus, IT infrastructure interaction in commercial entities is "challenging" - for the scientists as well as for the IT folks.

In our current project we embraced the Zeitgeist now, too. For our geodata portal development and deployment processes, we adopted following paradigm:

Google Cloud Platform, Compute and Container Engine with Kubernetes as the our computational platform.

Google Drive, Google Docs and Sheets for assets, functional and implementation specifications development, user stories and use cases.

GitHub as our distributed version control system, which allows us to collaborate, yet, keep contributions transparent and easily and publicly traceable.

Trello eventually serves as our workflow board, for sprints, keeping links of specs, repos and other soft information together. I believe, that we could have done everything in GitHub, but Google Docs and Trello provide gently mechanisms to also invite non-technical folks to contribute. And actually, what we really want is this, right?

Amazon IoT and OGC SensorThings vs SOS/SPS on MQTT

Today I stumbled over a blog post on the OGC Blog about "Amazon IoT and the candidate OGC SensorThings API Standard"

It was great to learn that Amazon IoT uses MQTT as network / transport layer. The more interesting is the move towards REST (which is HTTP) to address semantic issues with IoT systems:

AWS IoT, MQTT and many other network interoperability standards (LWM2M, CoAP, etc.) enable message interchange.  However, IoT network interoperability doesn’t enable the systems that are exchanging messages to interpret those messages. To realize the many-to-many system-of-systems vision, IoT applications need to implement standard ways of communicating sensor locations, sensor and data parameters, and sensor instruction sets. This is what the OGC SensorThings API provides.

It is understandable from the point of view, that OGC OWS is in general intrinsically interwoven with HTTP semantics and the application of SWE in the IoT context is desirable, well, even crucial.

However, MQTT stems from the constraints of low bandwidth, low energy consumption and unreliable connectivity, and implements a robust, small footprint asynchronous messaging platform. Earlier this we have developed a prototype system, that aims to add SWE semantics into MQTT messaging. We keep the specialisation of MQTT, where for example connections can be resumed which under SSL/TLS security can be a significant power saver. MQTT would also better address the "addressability" of distributed "things" which might be isolated behind IP NAT or run with dynamically assigned addresses, because in MQTT context the "things" would subscribe to SPS management topics and therefore higher order SDI services don't need to direct data queries or task requests to an address but place it on the queue with respective topics.

For an agricultural research project we distributed monitoring stations as a wireless sensor network throughout the catchment to deliver frequent measurements in near-realtime, which mimics OGC SOS.

A dynamic implementation of measurement frequencies, adapted to certain environmental conditions like heavy rainfall events require re-tasking of the nodes, which mimics OGC SPS. Within this paper we provide a framework where a threshold event triggers a reconfiguration task for a phosphorus measurement device, using asynchronous, push-based communication, on an MQTT queue, which links the ground stations with a cloud-based control system. OGC WPS algorithms continuously analyse incoming SOS measurements commits such a request into the wireless sensor network, and updates the measurement frequency of the target nodes to enable nutrient peak flow estimation during storm events.

This on-going work is based on original thoughts on the  interlinking  of  OGC  SWE  semantics  for  sensor  descriptions  and  observations (OGC Sensor Observation Service, SOS), sensor configuration and planning (OGC  Sensor Planning Service, SPS)  with  the  open  Message  Queue  Telemetry  Transport  (MQTT)  protocol.

MQTT  is  a  simple,  yet  very performing  and  robust  publish/subscribe  message  passing  system,  which  can  also  serve  for  event  brokering as  described  in  the  SWE  Service  Model.  Actual  addressing  and  data  transmission  is  done  by  a  topic  string and an arbitrary payload. The topic string and the payload need to reflect the necessary standard semantics to be  mapped  back  and  forth  to  allow  for  a  seamless,  and  preferably  lossless  bi-directional  multi-lateral communication  between  the  WSN  as  data  provider  (SOS),  web  clients  as  data  consumers  (SOS)  and  the management system, which interacts with the WSN in a standardised way, too (SOS and SPS).