Analyzing the Usability of an Argumentation Map as a Participatory Spatial Decision Support Tool

Analyzing the Usability of an Argumentation Map as a Participatory Spatial Decision Support Tool

(Version 8/15/06)

Christopher Sidlar and Claus Rinner

ABSTRACT: Argumentation Maps support participants in geographically referenced debates as part of urban planning processes. In a quasi-naturalistic case study, 11 student participants explored and discussed planning issues on the University of Toronto downtown campus. Further examination focuses on general usability aspects of the prototype such as cost of entry, efficiency, interactivity, and connectivity. By applying usability analysis methods from the field of human-computer interaction, we evaluate the learnability and memorability of, and the satisfaction with, the functionality of an Argumentation Map prototype. Our findings indicate that participants were generally satisfied with the tool but we include specific suggestions for improving its functionality, e.g. with respect to map navigation, display of discussion contributions, or online status of participants. On a more general level, we aim at informing research methodology in participatory geographic information science and spatial decision support systems.

Introduction

The Argumentation Map concept originated in a Ph.D. dissertation by Rinner (1999), and is later summarized in Rinner (2001). Rinner proposes Argumentation Maps, abbreviated as Argumaps, to support the planning process allowing for distributed, asynchronous discussions. Argumaps are based on the combination of an online discussion forum and an online geographic information system (GIS). Argumaps were conceived as a method to formalize debates that have geo-spatial elements in the discussion. Due to their online nature, Argumaps are able to take advantage of a number of aspects of the Internet, for example the ability to share information with many stakeholders (Laurini 2004) and the anonymity provided in Internet discussions (Kingston et al. 1999).

Keßler (2004) implements an Argumentation Map prototype as a proof of concept. This Web-based prototype integrates a discussion forum and a simple mapping tool. Technology used in the implementation includes the GeoTools Lite mapping toolkit, a custom-built Java applet for the discussion forum, the MySQL database for storage of geographically referenced discussion contributions, and the University of Minnesota MapServer for the supply of background map layers. Keßler chose these open source software tools on the grounds that they suffice the requirements for the Argumap concept set out by Rinner (1999) and that they minimize development costs. The functionality of the prototype includes map navigation (zoom in, pan, zoom to full extent), layer management (switching of layers on and off), display of map labels (e.g. building names). In the discussion forum, contributions are displayed by subject-author-date in a list with indentation by discussion thread, and selected individual contributions are fully displayed in a text window. Further, when selecting a contribution, its geographic references will be highlighted on the map. When selecting a map object, all discussion contributions referring to this object will be highlighted in the discussion forum. The Argumap also provides a full-text search tool for the discussion forum and summary statistics when browsing the map (number of contributions per map object). Finally, in terms of participation in geo-referenced debates, the tool offers a login feature which in turn enables the user to start a new discussion thread or respond to existing contributions. When editing a message, geographic references can be specified in the map and will be stored with the text of the message. The functionality and architecture of the prototype is also summarized in Keßler et al. (2005). The stakeholders in the planning process consist of a heterogeneous population with a variety of knowledge and skill levels, as demonstrated in previous research (Healey 1997; Simão and Densham 2004). Due to the wide range of possible users any planning support system must be designed in such a way that all are able to learn to use the majority of its functions. This introduces a motivation for a usability study for the prototype.

This paper provides background on usability analysis for participatory spatial decision support tools and describes such a case study. We looked at how Keßler’s (2004) prototype was understood and used by a heterogeneous participant population. The following sections describe the research background, methodology, as well as the preparation and results of the case study. Conclusions are then drawn in the form of recommendations for the Argumap prototype and suggestions for usability analysis for participatory GIS tools in general.

Background on Usability Analyses for Software Prototypes

 “Human-Computer Interaction (HCI) is concerned with the design of computer systems that are safe, efficient, easy and enjoyable to use as well as functional” (Preece 1993, 11). As long as there have been computers, their developers have been concerned with how the machine and its software will be used. The interaction between computers and humans is outlined by Licklider (1960) when he viewed the role of the human as formulating hypotheses for problems, setting standards for the evaluation to follow, and finally evaluating the output whereas the role of the computer was to facilitate the “routinizable work”, in order to free time for analysis and evaluation.

HCI evolved further through the work of Hansen with EMILY, a text editing system for programmers, in the 1970’s (Pew 2003, 8). Hansen is accredited with pioneering the use of the term “User Engineering Principles” (Hansen 1971). Hansen’s principles included knowing the user, minimizing memorization, optimizing operations, and engineering for errors. Hansen’s work was later followed by that of Engle and Granda (1975) at IBM that sets out guidelines for various aspects including the display, recovery procedures, user entry, and response time. The evolution of guidelines peaked with Smith and Mosier (1986) when such a comprehensive set was released that the guidelines themselves were contradictory.

By the late 1980s, the concept of user engineering principles had not only become a consideration for programmers but had also evolved into iterative process that included the production of prototypes, subsequent testing, and ultimately production of modified versions (Rosson and Carroll 2002). This spurred the formation of usability engineering. “Initially, usability engineering focused on the design of the user interface” (Rosson and Carroll 2002, 14). Since the personal computer revolution, usability engineering has also made its way to software engineering with the developers’ main concern being the user interface of the software.

Study Types This leads into the discussion on how best to study the users of a system or software. Systems can be evaluated using different levels of controls, Kirkakowski and Corbett (1990) categorized evaluation procedures into three types:

1. naturalistic study,
2. quasi-naturalistic study, and
3. experimental study.

Studies that are observational, taking advantage of already existing situational contexts, are considered naturalistic studies. Naturalistic studies provide realistically applicable results but in order to complete such a study the investigator must play a background role, and therefore collecting the required information to produce the desired results proves taxing. Quasi-naturalistic studies use a ‘real-world’ context but are used with such controls so that both evaluation and collecting of information are easier, and therefore a deeper investigation can be achieved. Finally, experimental studies use controls to focus on the independent variables that the investigator wishes to study, while mitigating variables that would cause errors in, or cloud, the results, but occurs in the least ‘realistic’ context.

Such a classification scheme can be compared to that described in Preece (1993). Preece outlines five categories of evaluation for the purpose of usability evaluation; analytic, expert, observational, survey, and experimental. An analytical evaluation is described as using “interface descriptions to predict user performance” (Preece 1993, 109). An expert evaluation uses identified experts in the field related to the prototype to analyze and evaluate it. An observational evaluation consists of evaluation of the behavior and reactions of users in using the prototype. A survey evaluation utilizes a questionnaire to solicit users’ opinions on the use of the prototype. And finally an experimental evaluation, similar to that of Kirkakowski and Corbett (1990), utilizes the scientific practice of controls to analyze the prototype.

Usability Evaluation

Meister and Rabideau (1965) outline a seven-step procedure for usability evaluation:

1. determining what a successful application of the prototype would be.
2. identifying of the ultimate goal of using the prototype.
3. segmenting the goal of the prototype, so that it may be analyzed as homogeneous functions.
4. identifying and describing the functions of the prototype.
5. deciding on criteria upon which the use of the functions is to be assessed.
6. allocating functions on the basis of whether they are user functions or prototype functions.
7. performing the experiment and observing on the basis of the identified criteria.

Shackel (1991) describes three types of variables that should be investigated when considering usability of a product: dimensional, performance, and attitude criteria. Dimensional criteria refer to the size of the product or its ergonomics, and do not alone provide a mark of usability but must be considered in conjunction with the performance and attitude criteria. Performance criteria refer to how well the product facilitates its function, and attitude criteria refer to the feelings the user has when using the specific product. Similar representations of these criteria can be found in the evaluation procedures used by Chapanis (1965; 1981; 1991), Meister and Rabideau (1965) and Parsons (1972).

Wong and Chua (2001) investigate four beneficial aspects of the Web that are likely to aid Public Participation GIS (PPGIS): low cost of entry, efficient data transfer, interactivity and connectivity. Wong and Chua also describe four barriers that are particularly present in Web-based PPGIS: cost of interactivity, user diversity, data and copyright costs, and trust and legitimacy. They adapt this methodology to investigate the application of the InfoResources project created by the Center for Community Partnership at the University of Pennsylvania while other researchers (Harrison and Haklay, 2002; Carver, 2001; Andrienko et al., 2002) employ a more ‘classical’ method of usability analysis as defined by human-computer interaction and usability engineers.

Methodology

Following Kirkakowski and Corbett’s (1990) classification, this experiment was designed as a quasi-naturalistic study. A quasi-naturalistic study was chosen on the basis that it allowed for Keßler’s (2004) Argumentation Map prototype to be analyzed in a ‘real world’ application while ensuring the case study was manageable enough so that a substantial investigation could be achieved.

This study used the prototype in an early identification/exploration stage as an example of bottom-up planning. Participants in the discussion were invited to identify planning ideas/concerns with the University of Toronto St. George campus. A map of the St. George campus was therefore used as the map component of the Argumap application. Fig. 1 shows the juxtaposition of the University of Toronto campus map with the discussion forum of this case study. Fig. 2 shows the map and the list of layers. The application provided the participants with the ability to shape the planning process and express the concerns that were prevalent to them. Participants were free to participate in the threads of the discussion that most interested them, and were also given the ability to start new threads in the discussion forum. The discussion was monitored for offensive posts by the investigators.

Fig. 1: Argumap prototype with University of Toronto campus map to the left and discussion forum to the right (data source: DMTI Spatial and University of Toronto, Cartography Office)

Fig. 2: Argumap prototype with layers for personal geographic references (red), other users’ geographic references (orange), and aerial background image (data source: DMTI Spatial, University of Toronto, Cartography Office, and J.D. Barnes First Base Solutions)

The participants were contacted using a snowball sampling procedure. The objective in utilizing this procedure is to isolate stakeholders by targeting campus users beginning from the investigators and snowballing outward through contacts. The creation of such a participant group was referred to as a ‘dutch study group’ by Jankowski and Nyerges (2001) in their description of the EAST2 method for GIS-supported participatory decision making. Invitations were sent out via email. Initially, 39 invitation emails were sent out. A setback occurred when the email was filtered as junk-mail by email providers, such as hotmail and gmail. A follow up email was released and in some instances, direct contact was made to these participants informing them that the email may have been directed to their junk-mail box. From the 39 emails, 11 people replied within one week showing interest in the case study, a 28% response rate. From these 11, the investigators received two additional contacts, one of which showed interest in the case study. In total, there were 12 participants in the study. This response rate can be compared to the response rate achieved in a similar study by Harrison and Hacklay (2002) who achieved a 23% (19 of 82) response and an 11% (9 of 82) participation rate in their second study, which used a similar sampling procedure. As a result of the comparable sampling procedures and sample sizes we are able to make the same conclusion that the participants are, “‘typical’, rather then representative” of their publics (Harrison and Hacklay, 2000: 845).

Meetings were then set up with the participants. There were two types of meetings: a group workshop and individual meetings, but all participants were exposed to the same presentation. Of the 12 participants, four attended the group session. The introductory workshops lasted about 30 minutes, and the participants were briefed about the concept, the case study and shown how to access, log in, and make a contribution. At the workshops, participants stated that they understood the functions of the prototype and what was expected from them. At the end of the session, participants were asked to fill out a pre-discussion questionnaire and an informed consent from. Of those who attended the workshops, all filled out the required forms to become participants.

We noticed that participants attending the individual meetings tended to ask more questions than were asked in the group session. In general, some people will shy away from asking questions in groups. They are more likely to ask questions in an individual setting. Another aspect of consideration is how well a participant retains the instruction. This will be assessed on the basis of their pre-existing knowledge of skills related to the case study (Internet forums, GIS, computers, geography, and planning).

The method chosen for investigating the usability of the Argumentation Map prototype involves a combination of the above explained usability methodologies. Usability of this prototype, we feel, needs to be considered on two levels: the general aspects of the tool, and the specific functions offered. The general usability aspects of the prototype were studied through investigating

• cost of entry,
• efficiency,
• interactivity,
• connectivity, and
• intended users,

following a condensed version of the method employed by Wong and Chua (2001). The specific functions of the prototype are analyzed by investigating the

• learnability,
• memorability, and
• satisfaction by case study participants.

This investigation focuses on the dimensional, performance, and attitude criteria (Shackel 1991) expressed by the users.

Functionality of the tool was evaluated through surveys that the participants were required to fill out at the beginning and end of the trial period. The pre-discussion questionnaire asked participants about their participation in Internet forums, familiarity with GIS, geography and computer knowledge, previous involvement in local planning decisions, as well as demographic variables like sex, year of study/tenure/occupation, age, and hometown. The objective of the pre-discussion questionnaire was to identify the participant characteristics as well as their ability to understand, and contribute to, geographically referenced discussions. A post-discussion questionnaire considered topics that refer to the prototype such as its graphical user interface, clarity of its functions, design and layout of the prototype, and suitability of the prototype for the purpose of spatial planning. The second questionnaire was geared towards analyzing the usability of the prototype and whether it would be beneficial to planning processes.

Analyzing the Usability of the Argumentation Prototype

General Usability

On the general level the prototype must be evaluated with respect to the

• cost of entry,
• efficiency,
• interactivity,
• connectivity, and
• its intended users.

Cost of entry refers to the expenditure imposed on the intended users and administrators when using the prototype. Cost of the prototype includes the price of the prototype, the tools needed to run it or access it, as well as the time it takes to set it up or use it. Efficiency refers to the prototype’s ability to fulfill its functions and objectives while taking minimal amount of resources, albeit time or hardware. Interactivity is measured through the users’ feedback on the responsiveness of the prototype. Connectivity refers to how easy it is for a user to access the prototype. The final criterion, intended users, includes various aspects such as the involvement in using similar software and processes as well as position in society and financial situation. Of particular interest is the relationship between participants and investigators and the level of trust that is present, and how this is reflected in the participation.

Cost of Entry

Keßler’s (2004) Argumentation Map prototype is based on open source software components, namely the MySQL database and the GeoTools Lite mapping toolkit, and was published under an open source license itself. Like many open source projects, this software thus is available at no cost. The prototype further adheres to GIS interoperability standards as defined by the OpenGeospatial Consortium (OGC 2005).

A Java applet is a program written in the Java language that can be downloaded and executed as part of a Web page, provided the user has a Java Runtime Environment (JRE) installed in a Web browser. The prototype uses an applet in order to capitalize on functions that cannot be implemented with HTML or JavaScript as well as displaying file types not supported by Web browsers, e.g. ESRI Shapefiles (Keßler, 2004). The Argumap application is downloaded free-of-charge, and the required JRE is also freely available as a download. Consequently, by using a client side applet the user must endure the downloading of the applet. The downloading time is dependent on the user’s Internet connection and computer speed and will imply connection costs for some users.

On the server side, the prototype uses open source software including the Apache Web server, the tomcat Servlet engine, the UMN MapServer, and the MySQL database. All of these components can be downloaded free of charge. The cost to the administrator is incurred through the requirement of Web server hardware and Web space. The processing speed of the server will affect the performance of the prototype; depending on the load that will be received an appropriate server is needed. This case study used a 3GHz Intel Pentium 4 with Hyper-Threading technology with 1 GB of DDR RAM.

The cost of entry for this prototype must be looked at from two angles; from the client and from the server or administrator. From the client side the cost of entry is minimal; it is dependent on having a computer with a typical configuration and a high-speed Internet connection (to be explained in connectivity section). This need can even be circumvented, as the user could use public terminals in Internet cafes or libraries to access the prototype, ultimately eliminating the cost of entry for the user/client. As for the administrator, the only cost incurred is that of a Web server and Web space, as the programs and administration tools are all available online as free downloads.

Efficiency

The efficiency of this prototype can be understood in two ways. Efficiency can be measured via a qualitative analysis of the discussion, as well as from explicit feedback from the users. The context of this case study was to express and discuss problems or concerns about the University campus. Threads with more posts, by a number of different of users, could be thought of as being more popular than others with fewer or no replies. Therefore, the rate of replies can indicate the importance of the topic being discussed. In the case study there were 20 threads. Of these, only three threads had three or more replies, while the majority of threads only had one or two replies. The three most popular threads consist of 25 contributions or 42% of all contributions. Due to the importance that these three threads had to the overall discussion they will be analyzed in-depth. One indication on why these were the most ‘important’ threads was length the case study. If a the case study had been longer then the threads started later in the discussion may have garnered more attention, but these three threads began within the first two days of the case study therefore experienced more exposure to discussion.

One aspect of efficiency is whether the discussion was kept concise and to the point. The in-depth analysis indicates an efficient discussion. One of the three important discussion topics was concerned with the crossing from one side of campus to the other across Queen’s Park. Starting off as a suggestion, with replies in the form of other suggestions, the thread stayed on topic discussing different methods that would make it safer to cross Queen’s Park Crescent. The second of the important threads related to the aesthetics of the Architecture Building at the corner of Huron and College Streets. This thread was more of an opinion thread as users with conflicting views met. Although the building was concluded to need repairs, some participants were of the opinion that the architecture program could even benefit from a building in need of renovations. The third important thread focused on parking on the campus. This thread was started with a general question. This question was followed up by more specific questions and suggestions for parking as well as alternatives to driving. With this discussion there was an obvious separation between the participants who drive to campus and those who take public transportation, involving back and forth replies.

An analysis of user feedback, acquired with the post-discussion questionnaire shows a different understanding. As participants expressed concerns that the discussion was too general and that it would have been better if the ability to start new threads would be restricted or left to the administrator. For example, one participant wrote that, “One main issue I would identify is the difficulty in maintaining a sense of continuity on a topic” while another stated, “…The prototype would have been enhanced by organizing the threads by topic in the discussion list so that the threads relating to one topic are grouped and seen altogether.” Others felt that it was tedious to manually click to expand each thread, needing to select the contribution to read it and look for new posts. Contrarily, some participants liked the nested discussion forum and felt that it was easy for them to find replies to threads that they had started themselves.

Interactivity

Participants were asked a variety of questions in the post-discussion questionnaire with reference to how they interacted with the prototype. Overall, participants were satisfied with how the prototype facilitated and handled the discussion. Participants found that the prototype “did it well” and “It was helpful to be able to see a map of the campus and visualize the relation between all the buildings. Also, it helped that when there was a posting about a building, that building was highlighted.” Another participant noted that “Having a map, and being able to interact with the map and post comments that way makes the prototype very user friendly, and would probably make it more likely that people will participate in the planning process.” Contrarily, a participant noted that the topic failed to fully engage the participants, while also saying that, “The geographic aspect was nice but in this case it didn’t seem too useful, as most participants knew the involved buildings very well already. If only the message board was there … that would have worked just as well with these users.” But this same participant went on to say that, “Combining a map with a message board is the main point of the prototype, which it succeeds in doing.” Such comments lead to a general indication that the prototype provides sufficient interaction to fulfill the objective it has been set out for.

Connectivity

Being a Web-based application, the potential users of the Argumentation Map prototype ultimately include anyone who has an Internet connection. In order to support interoperability, Keßler (2004) designed the applet using the platform-independent Java programming language. The participant should have a high-speed Internet access to ensure a reasonable connection time since the whole applet must be downloaded at the beginning of each session. It was noticed that even under this condition, it took about 1.5 to 2 minutes to load the prototype. Normally, HTML developers intend to keep their pages loading in under seven seconds as a rule of thumb. Once the applet is loaded refresh times for the map depend on the complexity of the shapefiles and images the applet has to load from the Web Map Server.

Intended Users

Keßler developed this prototype in order to increase public participation in the planning process. Therefore, the prototype “facilitates participation for citizens and stakeholders and gives the planners an opportunity to retrieve, store and organize local knowledge. It must be stated that it is not going to be an expert tool, but rather the opposite – it should be usable by as many people as possible, especially lay-persons” (Keßler 2004, 9). In the formation of this usability study, the intended audience was kept in mind as our participants varied widely in backgrounds and skill levels.

The case study had a mean age of 22.6 and a median age of 22. The largest occupation group was undergraduate student, 36% of the group or 4 of 11 participants. Of those who were graduates, no one had graduated more than two years ago, making the group ideal for discussing the university campus. Of the participants 73% (8 of 11) of them had expressed that they did have experience with Internet discussion forums, on a wide range of topics (no trends within forum usage emerged). Further, only two had stated ever participating in the planning process, the majority citing the reason that they had never had the opportunity. Lastly, 64% (7 of 11) of the participants expressed experience with GIS. The users were therefore qualified to participate in discussions relevant to the campus, while being diverse enough to get differentiated views on the prototype.

Function Usability

Usability of the specific functions of the prototype builds upon three main pillars:

• learnability,
• memorability, and
• satisfaction.

Learnability focuses on how easy it is for a user to understand and recognize the usefulness of the prototype or tools in the prototype. This factor must be analyzed in conjunction with the participants’ general education level and knowledge of specific topics that directly deal with the use of the prototype. Memorability denotes how well a user is able to retain what he or she has learnt about using the prototype, and re-apply this knowledge on another use of the prototype. And satisfaction is a broad category that encompasses both how the user felt while using the prototype, thus relating back to the learnability and memorability of the functions of the prototype, and how the users felt the prototype facilitated its functions.

Learnability

Generally, the participant group included only two people (or 18%) who had experience with the planning process. Further, the majority of participants (8 of 11) categorized themselves as being either experts or advanced users of computers. Also, 7 of 11 participants had GIS experience ranging widely from GIS beginners to GIS experts. In order to analyze how different types of users receive the prototype, the participants are categorized on the basis of planning experience, GIS experience, and level of computer knowledge. Those who are well-versed in all of these fields will be considered as experts, participants that either lack one of the bases but are skilled in the other two will be considered as intermediate users, while low representation in two or three fields will be considered as lay-users, or beginners. Typical planning meetings attract a few experts, a larger number of involved citizens, and a few citizens who are new to the process. Reasons for the low number of ‘beginners’ is their over-whelming unfamiliarity or intimidation with the process, thus, leaving the majority of issue championing to be done by “active publics” (Harrison and Hacklay 2002), or in this case, intermediate users.

Due to the limited number of participants in the case study, the numbers in the categories are also low, but similar to the distribution present in the actual planning process. In this study one of our participants can be categorized as an expert, nine as intermediate and one as a beginner. It is important to note that the time to learn the prototype is estimated based on the user’s self-rated learning time. The participants were also asked whether they felt that the learning time was too long, which works as a better indication of patience for learning the tool. Harrower et al. (2000) note an interesting finding on the learnability of a geographic visualization tool in that “understanding the purpose of a tool and recognizing when it is useful to solve a problem are two quite different issues” (Harrower et al. 2000, 298). Therefore, we must be cautious when we say that a user has learnt the Argumentation Map tool, as to whether the user actually uses the tool as Rinner (1999; 2001) had conceptualized it.

From the perspective of the expert, it took less than ten minutes to “get totally familiar with the prototype”, as he put it. This participant felt that the learning time was excellent. The way this participant used the prototype indicates a full understanding of the tool, since a variety of its functions were used.

For the nine intermediate users, the learning time ranged from 10 to 30 minutes with a median of 10 minutes. The general indication received from the users is that the learning time was not too long, while only a couple had noted differently. One participant stated that, “good software should be usable in 1-3 min” while learning took this user 10-15 minutes. Another intermediate user found the time to learn the basic functions acceptable but it took longer to learn the more advanced functions.

From the perspective of the participant who was categorized as a beginner, it took a “few” minutes to learn how to use the prototype. This echoes the point brought up before of how much of the tool was understood by this user, as it was hypothesized that experts would find it easier to learn the tool than beginners. The participant noted that the “few” minutes that it took to learn the tool were not too long, this being the better indication of how easy the user felt the tool was to learn. Upon investigation into the actual contributions of this user, only the basic functions of selecting buildings and making posts were used. Also, the posts had no more than one building selected, with only the initial map layers being visible. Therefore, the user did not take advantage of the more advanced functions like zooming, multiple geo-references, and layer management.

Memorability

In order to properly test memorability the participants need to undergo a significant time away from the prototype. Due to restrictions on the case study and study period such a break in the middle of the case study was impossible to organize. This is therefore an aspect that should be investigated further at another time in either a longer case study or by asking the participants of this case study to use the same prototype again, but this time without an introductory session.

Satisfaction

In order to study satisfaction with the tool we asked the participants to rate their overall experience with the prototype on a scale from 1 to 5 with 1 being the lowest and 5 being the highest score. The responses only varied slightly, as the overall mean was 3.41. This indicates that the participants did see a benefit in the prototype but there are still some aspects that should be addressed in order to increase the satisfaction level.

Conclusion and Outlook

Using a quasi-naturalistic case study, we have analyzed the usability of an Argumentation Map prototype from two perspectives: on a general overview of the tool and on a functional level. In considering the general aspects of the tool its usability is high. It costs nothing to prepare or use the tool; its loading time is acceptable for the amount of information being loaded, and was not a complaint of the participants; and the audience had little to no problems using it. While quantitatively the discussion looked to be efficient, participants had expressed feelings of confusion with reference to thread organization, in particular concerning who could start new threads. Also, the tool generally engaged the users as indicated by participants’ suggestions for further applications of the tool. In a specific review of its functionality, the tool also fares well. It did not take the participants a long time to ‘learn’ how to use the tool. But there seemed to be a discrepancy between learning the functionality and applying this knowledge, as the rate of advanced functions was limited. This would indicate that a simplification of the tool is in order. Due to the short duration the study could not be used to measure the retention of what the participants learnt while using the tool over a longer period of time, therefore not fully evaluating the memorability of the tool. With respect to the satisfaction level, the rating of 3.4 out of 5 indicates that the participants were not overly enthusiastic by the use of the tool while they were still relatively satisfied with its functions.

Participants were also asked questions referring to confusing events while using the tool, as well as about the most useful and any missing functions. From the responses to these questions and the comments/questions that were emailed to the investigators or brought up in the orientation sessions, a variety of additions and alterations can be suggested.

The largest concern among participants was that once a selection of a reference objects on the map had been made, there was no way of deselecting objects. This was seen as a major frustration. Given the present design of the study, we could not deduce whether the lack of a de-selection button reduced the number of contributions, or the number of selected geographic elements.

Another concern participants had with the prototype was that the discussion was complicated or tedious to read, as it required the user to select each comment. Participants requested a button that would expand all nested elements at once or a way to read all the comments as one contribution, similar to functions present in news readers and Internet forums. Also, participants expressed a concern in the lack of order in the discussion. Since our case study was conducted in an exploratory planning stage, this critique is difficult to avoid. Currently, everyone can start a new thread on the main level. This option could be removed and the administrator left with the ability to add general root topics (e.g. Parking, Noise, Safety, Construction). Users could also ask the administrator to add a root topic. Along similar lines, participants should be given a means to contact the administrator (e.g. via email), whether for help with an issue, to report offensive material, or provide comments to make the tool more user-friendly. The participants also expressed a wish to be able to see all the comments that they made themselves. One participant when asked how he or she would see the tool fit into the planning process felt that the planning process was enhanced with drawings and pictures. Thus the ability to upload photos or images and embed them in the contributions should be developed.

Users also requested a way of formatting contributions. For this, investigation towards a database that will store user formatting is suggested. Other usability concerns dealt with the user interface elements themselves, particularly regarding nested comments in the discussion forum. It was suggested that this element be changed to standard user interface design (i.e. a plus denoting nested elements). Also it was expressed that the zoom function was difficult to manage. This could be changed to a staged zoom similar to popular Web mapping sites. Also the participants were confused by the scale bar showing the wrong units. A minor issue was the label on the “answer” button, as users commented that the label should rather read “reply”. More of a concern was the limited amount of information shown in the tooltips for buildings. When contributions are made, the name of the building should be preserved in addition to contribution titles. Other users wanted to see information like occupants of the building, and built characteristics (e.g. number of floors). The users also had wanted to be able to select/draw areas, thus supporting the expansion of the drawing tools beyond just points, with emphasis on polygons over lines/polylines.

Although the tool was developed to be used in an asynchronous and distributed manner, participants had wished to visualize who and how many people were logged into the discussion at any time. Participants also expressed concerns on the number of tabs and felt that the three tabs could be consolidated into one screen. Most participants generally liked the layout of the user interface.

By applying usability analysis methods from HCI for the evaluation of an Argumentation Map prototype, we hope to contribute to filling a gap in ongoing research on PPGIS. More case studies are required on all levels – on this particular Argumap prototype, on the Argumap concept, and PPGIS in general – to match existing spatial decision support technology to user needs. We are specifically interested in real-world (naturalistic) case studies in conjunction with acute urban planning issues, and in the comparison of the usability of different PPGIS tools.

Acknowledgements We would like to thank the participants in the case study for their contribution to this research. We are indebted to Carsten Keßler for his support in adapting and setting up the Argumap application for the case study. Lauren Beharry and Amrita Hari provided valuable help with preparing the manuscript. Partial funding of this case study from the Natural Sciences and Engineering Research Council of Canada (NSERC) and the GEOIDE Network of Centres of Excellence in Geomatics, project HSS-DSD#17, “Promoting Sustainable Communities through Participatory Spatial Decision Support”, is gratefully acknowledged.

References

Andrienko, N., Andrienko, G., Voss, H., Bernardo, F., Hipolito, J., & Kretchmer, U. (2002). Testing the Usability of Interactive Maps in CommonGIS. Cartography and Geographic Information Science. 29, 325-342.

Carver, S., Evans, A., Kingston, R. & Turton, I. (2001). Public Participation, GIS, and Cyberdemocracy: evaluating on-line spatial decision support systems. Environment and Planning B: Planning and Design. 28, 907-921.

Chapanis, A. (1965) Research Techniques in Human Engineering. Baltimore, MD: The John Hopkins Press. Chapanis, A. (1981). Evaluating ease of use. Proceedings of IBM Software and Information Usability Symposium. Poughkeepsie, NY: IBM. pp. 105-120.

Chapanis, A. (1991). Evaluating Usability. Human Factors for Informatics Usability. (Eds. B. Shackel and S. Richardson). New York, NY: Cambridge University Press.

Engle, S.E. and R.E. Granda. (1975). Guidelines for man/display interfaces. IBM Technical Report TR 00.2720. Poughkeepsie, NY: IBM.

Hansen, W. (1971). User Engineering Principles for Interactive Systems. AFIPS Conference Proceedings 30, Fall Joint Computer Conference. Montvale, NJ: AFIPS Press. pp. 523-532.

Harrison, C. and M. Hacklay. (2002). The Potential of Public Participation Geographic Information Systems in UK Environmental Planning: Appraisals by Active Publics. Journal of Environment Planning and Management. 45(6), 841-863.

Harrower, M., MacEachren, A.M. and A.L. Griffin. (2000). Developing a Geographic Visualization Tool to Support Earth Science Learning. Cartography and Geographic Information Science. 27(4), 279-293.

Healey, P. (1997). Collaborative Planning: Shaping Places in Fragmented Societies. London: Macmillan Press.

Jankowski, P. and T. Nyerges. (2001). Geographic Information Systems for Group Decision Making: towards a participatory, geographic information science. New York, NY: Taylor and Francis.

Keßler, C. (2004). Design and Implementation of Argumentation Maps. Münster, Germany: Diploma Thesis, Westfälische Wilhelms-Universitäty Münster, Germany. Available online at http://www.carstenkessler.de/argumap/

Keßler, C., C. Rinner, and M. Raubal (2005). An Argumentation Map Prototype to Support Decision-Making in Spatial Planning. In F. Toppen, M. Painho (Eds.): Proceedings of AGILE 2005 - 8th Conference on Geographic Information Science, 26-28 May 2005, Estoril, Portugal, pp. 135-142.

Kingston, R ., S. Carver, A. Evans and I. Turton (1999). A GIS for the Public: Enhancing Participation in Local Decision Making. GIS Research UK (GISRUK'99). Available online at http://www.geog.leeds.ac.uk/papers/99-7/.

Kirkakowski, J. and M. Corbett. (1990). Effective Methodology for the Study of HCI. New York, NY: Elsevier Science Publishers. Laurini, R. (2004). Computer Systems for Public Participation. Available online at http://www.gisig.it/vpc_sommet/CD_Sommet/ws3/articololaurini.pdf.

Licklider, J.C.R. (1960). Man-computer Symbiosis. IRE Transaction on Human Factors in Electronics. HFE-1, 4-11. Meister, D. and G.F. Rabideau. (1965). Human Factors Evaluation in System Development. New York: John Wiley and Sons, Inc.

Open Geospatial Consortium. (2005). http://www.opengeospatial.org/about/?page=vision First Accessed: May 5, 2005.

Parsons, H.M. (1972). Man-Machine System Experiments. Baltimore, MD: The John Hopkins Press.

Pew, R. (2003). Evolution of Human-Computer Interaction: From Memex to Bluetooth and Beyond. The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. (Eds. J. Jacko and A. Sears). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Preece, J. (ed.) (1993). A Guide to Usability: Human Factors in Computing. Don Mills, ON: Addison-Wesley.

Rinner, C. (1999). Argumentation Maps - GIS-based Discussion Support for Online Planning. GMD Research Series No. 22. Sankt Augustin, Germany: University of Bonn. Available online at http://docserver.fhg.de/gmd/1999/research/022.pdf.

Rinner, C. (2001). Argumentation maps: GIS-based discussion support for on-line planning. Enivironment and Planning B: Planning and Design. 28, 847-863.

Rosson, M.B. and J.M. Carroll (2002). Usability Engineering: Scenario-based development of Human-Computer Interaction. New York, NY: Morgan Kaufmann Publishers.

Shackel, B. (1991). Usability – Context, Framework, Definition, Design and Evaluation. Human Factors for Informatics Usability. (Eds. B. Shackel and S. Richardson). New York, NY: Cambridge University Press.

Simão, A. and P. Densham. (2004). Designing a Web-Based Public Participatory Decision Support System: The Problem of Wind Farms Location. Proceedings CORP 2004 and Geomultimedia 04, 9th International Symposium on ICT in Urban and Regional Planning. (ed. M. Schrenk). Available online at http://corp.mmp.kosnet.com/CORP_CD_2004/archiv/papers/CORP2004_SIMAO_DENSHAM.PDF.

Smith, S.L. and J.N. Mosier. (1986). Guidelines for designing user interface software. Technical Report ESD TR-86-278. Hansom Air Force Base, MA: USAF Electronic Systems Division.

Wong, S. and Y.L. Chua. (Fall 200/Winter 2001). Data Intermediation and Beyond: Issues for Web-based PPGIS. Cartographica. 38(3/4), 63.