Author: Claus Rinner

This is a brief account of two “Mapping for Nepal” sessions at Ryerson University’s Department of Geography and Environmental Studies. In an earlier post found at http://gis.blog.ryerson.ca/2015/04/27/notes-for-nepalquake-mapping-sessions-ryersonu-geography/, I collected information on humanitarian mapping for these same sessions.

Mapathon @RyersonU, Geography & Spatial on Monday, 27 April 2015, 10am-2pm. 1(+1) prof, 2 undergrads, 3 MSAs, 1 PhD, 1 alumnus came together two days after the devastating earthquake to put missing roads, buildings, and villages in Nepal on the map using the Humanitarian OpenStreetMap Team’s (HOT) task manager. Thank you to MSA alumnus Kamal Paudel for initiating and co-organizing this and the following meetings.

Mapathon @RyersonU, Geography & Spatial on Sunday, 3 May 2015, 4pm-8pm. Our second Nepal mapathon brought together a total of 15 volunteers, including undergraduate BA in Geographic Analysis and graduate Master of Spatial Analysis (MSA) students along with MSA alumni, profs, and members of the Toronto-area GIS community. On this Sunday afternoon we focused on completing and correcting the road/track/path network and adding missing buildings to the map of Nepal’s most affected disaster zones. Photos via our tweets:

Helping #NepalQuake relief and learning about relevance of OpenStreetMap, geospatial data modeling, & remote sensing pic.twitter.com/1e0mCz0Emr

— Claus Rinner (@ClausRinner) May 3, 2015

Editing HOTOSM live in MSA lab at Ryerson University-15 Torontonians! Thanks @ClausRinner for organizing. #hotosm pic.twitter.com/5Dn9KXHvtB — Kamal Paudel, GISP (@kpaudel) May 3, 2015

My observations and thoughts from co-organizing and leading these sessions, and participating in the HOT/OSM editing:

• In addition to supporting the #EqResponseNp in a small way, the situation provided an invaluable learning opportunity for everyone involved. Most participants of our sessions had never contributed to OSM, and some did not even know of its existence, despite being Geography students or GIS professionals. After creating OSM accounts and reading up on the available OSM and Nepal-specific documentation, participants got to map hundreds of points, lines, or polygons within just a couple of hours.
• The flat OSM data model – conflating all geometries and all feature types in the same file – together with unclear or inconsistent tagging instructions for features such as roads, tracks, and paths challenged our prior experience with GIS and geographic data. Students in particular were concerned about the fact that their edits would go live without “someone checking”.
• While the HOT task manager and general workflow of choosing, locking, editing, and saving an area was a bit confusing at first, the ID editor used by most participants was found to be intuitive and was praised by GIS industry staff as “slick”.
• The most recent HOT tasks were marked as not suitable for beginners after discussions among the OSM community about poor-quality contributions, leaving few options for (self-identified) beginners. It was most interesting to skim over the preceding discussion on the HOT chat and mailing list, e.g. reading a question about “who we let in”. I am not sure how the proponent would define “we” in a crowd-mapping project such as OSM.
• There was a related Twitter #geowebchat on humanitarian mapping for Nepal: “How can we make sure newbies contribute productively?”, on Tuesday, 5 May 2015 (see transcript at http://mappingmashups.net/2015/05/05/geowebchat-transcript-5-may-2015-how-can-newbies-contribute-productively-to-humanitarian-mapping/).
• The HOT tasks designated for more experienced contributors allowed to add post-disaster imagery as a custom background. I was not able to discern whether buildings were destroyed or where helicopters could land to reach remote villages, but I noticed numerous buildings (roofs) that were not included in the standard Bing imagery and therefore missing from OSM.
• The GIS professionals mentioned above included two analysts with a major GIS vendor, two GIS analysts with different regional conservation authorities, a GIS analyst with a major retail chain, and at least one GIS analyst with a municipal planning department (apologies for lack of exact job titles here). The fact that these, along with our Geography students, had mostly not been exposed to OSM is a concern, which however can be easily addressed by small changes in our curricula or extra-curricular initiatives. I am however a bit concerned as to whether the OSM community will be open to collaborating with the #GIStribe.
• With reference to the #geowebchat, I’d posit that newbie != newbie. Geographers can contribute a host of expertise around interpreting features on the ground, even if they have “never mapped” (in the OSM sense of “mapping”). Trained GIS experts understand how feature on the ground translate into data items and cannot be considered newbies either. In addition, face-to-face instructions by, and discussion with, experienced OSM contributors would certainly help to achieve a higher efficiency and quality of OSM contributions. In this sense, I am hoping that we will have more crowd-mapping sessions @RyersonU Geography, for Nepal and beyond.

In my Master of Spatial Analysis (MSA) course “Cartography and Geographic Visualization” in the Fall 2014 semester, three MSA students experimented with geospatial analysis of tweets. This post provides a brief account of the three student projects and ends with a caution about mapping and spatially analyzing tweets.

Yishi Zhao wrote her “mini research paper” assignment about “Exploring the Thematic Patterns of Twitter Feeds in Toronto: A Spatio-Temporal Approach”. Yishi’s goal was to identify the spatial and thematic patterns of geolocated tweets in Toronto at different times of day, as well as to explore the use of R for spatio-temporal analysis of the Twitter stream. Within the R platform, Yishi used the streamR package to collect geolocated tweets for the City of Toronto and mapped them by ward using a combination of MapTools, GISTools, and QGIS. Additionally, the tm package was used for text mining and to generate word clouds of the most frequent words tweeted at different times of the day.

One general observation is that the spatial distribution of tweets (normalized by residential population) becomes increasingly concentrated in downtown throughout the day, while the set of most frequent words expands (along with the actual volume of tweets, which peaked in the 7pm-9pm period).

MSA student Alexa Hinves pursued a more focused objective indicated in her paper’s title, “Twitter Data Mining with R for Business Analysts”. Her project aimed to examine the potential of geolocated Twitter data towards branding research using the example of singer Taylor Swift’s new album “1989”. Alexa explored the use of both, the streamR and twitteR packages in R. The ggplot2, maps, and wordcloud packages were used for presentation of results.

Alexa’s map of 1,000 Taylor Swift-related tweets suffers from a challenge that is common to many Twitter maps – they basically show population distribution rather than spatial patterns that are specific to tweet topics or general Twitter use. In this instance, we see the major cities in the United States lighting up. The corresponding word cloud (which I pasted onto the map) led Alexa to speculate that businesses can use location-specific sentiment analysis for targeted advertising, for example in the context of product releases.

The third project was an analysis and map poster on “#TOpoli – Geovisualization of Political Twitter Data in Toronto, Ontario”, completed by MSA cand. Richard Wen. With this project, we turn our interest back to the City of Toronto and to the topic of the October 2014 municipal election. Richard used similar techniques as the other two students to collect geolocated tweets, the number of which he mapped by the 140 City neighbourhoods (normalized by neighbourhood area – “bubble map” at top of poster). Richard then created separate word clouds for the six former municipalities in Toronto and mapped them within those boundaries (map at bottom of poster).

Despite the different approach to normalization (normalization by area compared to Yishi’s normalization by population), Richard also finds a concentration of Twitter activity in downtown Toronto. The word clouds contain similar terms, notably the names of the leading candidates, now-mayor John Tory and candidate Doug Ford. An interesting challenge arose in that we cannot tell just from the word count whether tweets with a candidate’s name were written in support or opposition to this candidate.

The three MSA students used the open-ended cartography assignment to acquire expertise in a topic that is “trending” among neo-cartographers. They have already been asked for advice by a graduate student of an environmental studies program contemplating a Twitter sentiment analysis for her Master’s thesis. Richard’s project also led to an ongoing collaboration with journalism and communication researchers. However, the most valuable lesson for the students and myself was an increased awareness of the pitfalls of analyzing and mapping tweets. These pitfalls stem from the selective use of Twitter among population subgroups (e.g., young professionals; globally the English-speaking countries), the small proportion of tweets that have a location attached (less than 1% of all tweets by some accounts), and the limitations imposed by Twitter on the collection of free samples from the Twitter stream.

I have previously discussed some of these data-related issues in a post on “Big Data – Déjà Vu in Geographic Information Science”. An additional discussion of the cartography-related pitfalls of mapping tweets will be the subject of another blog post.

Blog post authored by Richard Wen and Claus Rinner

A great way to demonstrate the manipulation of geospatial raster data is Conway’s Game of Life [1]. The “game” starts with a grid (“board”) of binary cells, which represent either alive (populated) or dead (empty) states. Each cell interacts with its eight adjacent neighbours to determine its next state. At each iteration of the game clock, the following rules are applied [1]:

• A live cell with less than two or more than three live neighbours dies (under-population, overcrowding).
• A live cell with two or three live neighbours continues to live.
• A dead cell with three live neighbours becomes alive (reproduction).

The free and open-source Geographic Information System (GIS) software package QGIS [2] offers support for scripting with the Python programming language (pyQGIS module), which enables the use of powerful libraries such as NumPy and GDAL for dealing with raster data. Numerical Python (NumPy) [3] is a package developed for Python that is geared towards scientific computation with support for multi-dimensional arrays and matrices. The Geospatial Data Abstraction Library (GDAL) [4] is a library for translating raster and vector geospatial data formats available as a binding for Python.

Using NumPy, GDAL, and pyQGIS, we implemented the Game of Life, where NumPy manipulates the arrays, GDAL handles reading and writing of the raster data, and pyQGIS visualizes the rasters and their relative changes. The source code was written by Master of Spatial Analysis student Richard Wen with input from Dr. Claus Rinner and is available at https://github.com/rwenite/QGIS_RasterArray. The project was inspired by Anita Graser’s visit to Ryerson’s Lab for Geocomputation in October 2014, during which Anita developed a vector-based version of the Game of Life in QGIS (see http://anitagraser.com/2014/11/16/more-experiments-with-game-of-life/).

Our implementation takes an object-oriented approach, in which an object of a Game of Life class is instantiated and the gaming board is updated with the cycle() method using the QGIS python console. The core function is the manipulation of individual raster cells based on a coded algorithm – in this case, the rules defined by the Game of Life.

Let’s start by initializing and cycling a gaming board using default parameters:

# Instantiate a starting board
x = GameofLife()

# Cycle the board twice
x.cycle(2)

The gaming board may be initialized with a random raster, a filled raster, a custom raster, or from a pre-defined raster file:

# The default is a random raster, we can set the width and height as well
x = GameofLife(width=3,height=5)
# Cycle the board
x.cycle()

# Fill a cells object with 1s
y = Cells(inRaster=1)
# Create a raster with the filled cells object in the directory
y.toRaster("path\\to\\filledraster\\file.tif")
# Instantiate a starting board with the filled raster
x = GameofLife(raster="path\\to\\filledraster\\file.tif")
# Cycle the board
x.cycle()

# Generate a raster from a list of tuples
y=Cells(inRaster=[
(0,0,0,0,0,0,0,0),
(0,0,0,0,0,0,0,0),
(0,0,1,0,0,1,0,0),
(0,0,0,0,0,0,0,0),
(0,0,1,0,0,1,0,0),
(0,0,0,1,1,0,0,0),
(0,0,0,0,0,0,0,0),
(0,0,0,0,0,0,0,0)])
# Create a raster with the custom cells object in the directory
y.toRaster("path\\to\\customraster\\file.tif")
# Instantiate a starting board with the custom raster
x = GameofLife(raster="path\\to\\customraster\\file.tif")

# Instantiate a starting board with a raster
x = GameofLife(raster="path\\to\\raster\\file.tif")

Date source: City of Toronto Open Data [5]

Some other interesting features include changing animation speed, jumping cycles, and applying customized layer styles:

# Adjust delay to 3 seconds
x.speed=3
# Cycle 10 times normally
x.cycle(10)
# Cycle 5 times and display every 2nd cycle
x.cycle(5,2)
# Set the style to the defined qml file
x.style = “path\\to\\qml\\style\\file.qml”

This post focuses on the functionality of the program, while its inner workings can be grasped from comments in the Python source code posted at https://github.com/rwenite/QGIS_RasterArray. The code was written and tested for QGIS 2.6; feedback on any issues is most welcome. The use of a NumPy array to iterate through the grid cells was found in an answer by user “gene” on GIS StackExchange [6]. Reading and processing raster data does have its challenges. When dealing with large grids, reading raster data in blocks rather than as a whole is advisable, because there may not be enough RAM to store an entire file at once [7].

The aim of implementing the Game of Life with Python and QGIS is to demonstrate some fundamental concepts of raster data analysis and cellular automata modeling, both of which have important applications in Geography and GIS. Existing QGIS functionality and scripts for raster processing seem to focus more on low-level input/output operations than higher-level analysis functions. For example, we did not find advanced local and focal raster operations in QGIS’ raster calculator. Thus, we envision that the RasterArray code could serve as a basis for expanding raster analysis in QGIS. The code will also be used in a yet-to-be-written lab assignment in GEO641 “GIS and Decision Support” in Ryerson’s BA in Geographic Analysis program.

References:

[1] Wikipedia, Conway’s Game of Life
http://en.wikipedia.org/wiki/Conway%27s_Game_of_Life

[2] QGIS
http://www2.qgis.org/en/site/

[3] NumPy, Numerical Python
http://www.numpy.org/

[4] GDAL, Geospatial Data Abstraction Library
http://trac.osgeo.org/gdal/wiki/GdalOgrInPython

[5] Toronto Open Data, Regional Municipal Boundary
http://www.toronto.ca/open

[6] How to do loops on raster cells with python console in QGIS?
http://gis.stackexchange.com/questions/107996/how-to-do-loops-on-raster-cells-with-python-console-in-qgis

[7] Chris Garrard, Utah State University, Reading Raster Data with GDAL
http://www.gis.usu.edu/~chrisg/python/2008/os5_slides.pdf