University of Eastern Finland, School of Computing

Workshop of CS education Tools 2008

Koli, Finland, November 13-16th, 2008

Call for Tools

A strength of computing education research is its focus on practice. One important subfield within this area deals with developing various kinds of software tools to assist students' and/or teachers' work. Such tools include, for example, automatic assessment, visualization, simulation and course management tools, as well as dedicated learning environments or learning objects. They have had a considerable effect on teaching practices and students' learning outcome in many institutes, and many successful experiences have been reported. Typical research papers related to this software include system papers, in which the rationale for the new tool is described in addition to presenting its functionality, and evaluation papers, where the focus is on evaluating the impact of the tool on students' learning process, learning outcomes or teacher's work.  Many papers include both of these elements, though.

In many cases developing this kind of software requires a major effort from the researcher(s) and may take from several months to several years or full time work.  Unfortunately, this work is rarely given proper credit in scientific evaluations as typically research papers are considered the only important merit. This practice can well be supported by a claim that from the research point of view, the software is only a tool required to carry out the experiment, or test the feasibility of the novel idea - in a similar way as physical gadgets are needed in experimental physics research. Moreover, the software itself has no scientific peer review system to assess its quality. Even though there are some resource repositories, like SIGCSE Education links, these have not been very successful. For example, SIGCSE education links has currently in total only 103 entries for the whole CS field, of which only 21 are classified as courseware.  Moreover, currently no new submissions are accepted. Another type of evaluation process is applied in Courseware competitions, such as Premier Award of Engineering Education Courseware, but very few software get credit through these kinds of procedures.

A significant drawback of this state of affairs is that there is not much incentive for developing the software to match something more than the needs of the developer's course / class / institute. When the initial contribution has been published in papers, it is quite difficult to get credit for further development work like improving usability, generalizing the available features to allow easier adoption of the tool, writing high quality tutorials and manuals, and extending the tool to cover new topics. Consequently very few tools have achieved a wide international dissemination – most are used only in their developers' home institutes or have been forgotten totally when the initial author has left the institute, and maintenance is no longer available.

We argue that a new procedure is needed to evaluate the quality of software tools to give them proper scientific credit. With a rigorous evaluation process high quality educational software should gain a merit comparable to research papers. Moreover, such software has much better chances for wider dissemination, which would be beneficial for the whole CS education field.

With this rationale, we present a CALL FOR TOOLS for computing education. This workshop on computing education tools will be organized as a part of the Koli Calling 2008 conference. The workshop will include presentations of accepted tools, discussions on relevant directions of future tools research and appropriate actions to promote research, development, and dissemination of educational tools / software for computing education.

Original submissions in all areas related computing education are welcome. Topics of interest include, but are not limited to:

A submission should be an original novel tool for some purpose, or a new version of an existing tool with a clearly identifiable new contribution.  The submission should include:

  1. A short paper (3-5 pages in ACM double column format) that summarizes the rationale of the tool
    • What is the problem in CS education the tool addresses and how does it address it?
    • What are the arguments / evidence that the tool works effectively to meet this aim?
    • A brief description of the main functionalities/features provided by the tool.
    • In which way does the tool differ from other tools solving the same or similar problem? What are the key contributions of the tool?
    • Summary of possible publications about the tool and its evaluation results.
  2. Instructions to access a web site that includes
    1. A tutorial or guided tour of using the tool
    2. Relevant documentation of the tool
    3. The tool itself
    4. Source code of the tool (if appropriate) and instructions to install the tool on a new platform
    5. Any other relevant resources
  3. If the tool is only accessible on a server and requests logging in, the submission must include relevant information to allow reviewers to access the tool. The accounts should preferably be anonymous.

Note that the part two of the submission is more important from the review process point of view than the summary paper (part one).

The submissions will be evaluated based on the following criteria:

Note that a contribution cannot be based on unpublished or proprietary code that is inaccessible. Building the contribution on a commercially available resource is not an obstacle, however, as the main criteria is accessibility.  Yet, for the evaluation process, access to all relevant resources should be free. In addition, the libraries and other resources needed to develop the system further must be obtainable for all researchers.

Koli Calling Educational Tool Award

The workshop program committee will select one of the accepted tools to receive the Koli Calling Educational Tool Award at the conference.

Important dates

Submissions due, September 1st 2008
Notification of acceptance, October 2nd
Registration deadline, October 30th
Workshop, November 13th (evening) - 16th (lunch-time)

The tool papers (part 1 in the submission) will also be published in the proceedings of the Koli Calling conference, after the conference.

Program committee

Ari Korhonen, Helsinki University of Technology, Finland (chair)
Mordechai (Moti) Ben-Ari, Weizmann Institute of Science, Israel
Pierluigi Crescenzi, Università degli Studi di Firenze, Italy
Mike Joy, University of Warwick, UK
Mikko-Jussi Laakso, University of Turku, Finland
Raymond Lister, University of Technology, Sydney, Australia
Lauri Malmi, Helsinki University of Technology, Finland
Arnold Pears, Uppsala University, Sweden
Guide Roessling, Darmstadt University of Technology, Germany
Tapio Salakoski, University of Turku, Finland
Carsten Schulte, Freie Universität Berlin, Germany

For more information, contact Ari Korhonen ( or Lauri Malmi (