Reviewing Guide
The review form is the same for all categories. This means that some of the criteria have a slightly different meaning, depending on the submission category. As a quick reminder, here are the submission categories:
Full paper (12-16 pages + references)
- Research papers
- Empirical papers
- Theoretical papers
- Review papers
- System and tool papers
Discussion paper (7-10 pages + references)
e.g. short empirical paper, proposition paper, …
For more information, check the call for papers: https://www.kolicalling.fi/call-for-papers/
Please make sure that you are reviewing the submission in the category that it was submitted in. If you feel that the submission was submitted in the wrong category (e.g. systems and tools instead of research) you can either alert the chairs or note in your review whether that would change your rating.
Since the formatting in EasyChair is not optimal, below is a better readable version of the review form with additional comments. These are mostly adapted from Amy Ko’s TOCE reviewing guidelines that you can also use as a reference (the criteria mostly align):
https://github.com/acm-toce/documentation/wiki/Reviewing-Criteria
We use a four-point rating scale that, for most criteria, corresponds to the well known levels of “accept”, “minor revision”, “major revision” and “reject”, albeit with specific labels for each criterion. The general idea of the levels is:
Accept: No work needed before camera-ready version, some minor suggestions might exist.
Minor revision: Suggested changes should be made, but can be implemented at the discretion of the authors.
Major revision: Suggested changes should be made and a new review of these is needed. If not too many criteria are rated at that level, a conditional accept might be a suitable option.
Reject: The changes that are necessary for this criterion cannot reasonably be made for a camera-ready version, therefore it should lead to recommending rejection.
Reviewer’s confidence.
What is your level of expertise in the methods used and context of this submission?
5: expert
4: high
3: medium
2: low
1: none
Comment: If you feel that you have low or no experience in these areas, please consider whether asking for a different paper to review is an option. If this is unlikely, for example when writing a last-minute emergency review, please still indicate if you feel that you may lack the specific expertise for this submission.
Contextualisation.
How well situated is this work both in the literature and (esp. for Systems and Tools paper) in comparison to other similar systems or tools? Please provide pointers to related work if you believe some is missing.
4: Excellent contextualization of the submission’s contribution.
3: Submission’s contribution is well contextualized, however some improvements are suggested.
2: Limited contextualization of submission’s contribution, several improvements are necessary before publication.
1: Very limited or no contextualization of the submission’s contribution.
Comment: Contextualization can happen in various ways, it does not have to be in a dedicated “related work” or “background” section. Also, submissions may vary with regard to the contextualization that is deemed necessary. If a submission does not need to make strong links to prior work or theory, it may still be rated as excellent – the rating should depend mostly on how much you think is missing.
Methods.
If the submission follows a reporting standard: To what extent is the chosen reporting standard adhered to?
In general: To what extent does the submission contain the necessary information to follow and (if applicable) replicate the work presented.
4: All necessary information provided with enough detail
3: Some information is missing, but nothing important
2: Important parts of information are missing
1: Significant amount of information is missing
Comment: Please check the appropriate reporting standard as part of your review. You can use the checklist:
to report missing aspects in the justification. Feel free to add further aspects that you feel are missing. The overall goal is to make the research presented “recoverable” (for more on this, see the TOCE guidelines). When reading the submission, the reader should have all the necessary information to understand what has been done and how and can thus make an informed judgment on the soundness (and possibly built upon the work in the future).
Soundness and Interpretation.
a) Empirical work: Are the methods appropriate for addressing the research questions and are any interpretations appropriately evidenced?
b) Theoretical work: Does the submission present a logical chain of reasoning, backed up by literature?
c) Systems and tools: Does the paper present either a suitable discussion of the design of the system/tool and/or an evaluation of the system/tool’s effectiveness?
4: The presented work has no issues, at most there are minor suggestions.
3: The presented work has some issues that could be addressed before publication.
2: The presented work has issues that need to be addressed before publication.
1: The presented work has serious issues that cannot be addressed easily.
Comment: For research papers, this is the central aspect. The work presented needs to show the necessary rigor and soundness that distinguishes a research report from other types of publications (e.g.a practical report). For discussion papers, we also require soundness but can keep in mind that the results may be preliminary or that the methods used may also be a part of the discussion. Reviewers have to determine whether they are seeing the presented work as mostly empirical, theoretical or presenting a system or tool – regardless of the distinction between full and discussion paper.
Significance of contribution.
a) Full paper: Does this submission advance knowledge that is of interest to the Koli Calling community, and more generally, to the computing education research community?
b) Discussion paper: Will this submission likely spark discussion or further research or innovation in the community?
4: Very significant
3: Significant
2: Somewhat significant
1: Not significant
Comment: For discussion papers, this aspect is highly relevant, as we aim for interesting discussions at Koli. For research papers, if the other aspects all score highly, the significance of the contribution should be less of a factor to determine the overall rating, but you may justify why this should be different for a particular submission in the overall justification.
Language and expression.
Is the submission written clearly enough to publish? If not, please provide specific areas where clarity could be improved.
4: Submission is ready for publication, there might be minor improvements.
3: Submission needs some improvement before publication.
2: Submission needs major improvements before publication.
1: Submission has serious limitations.
Comment: Please keep in mind that not everyone is a native English speaker, so the focus should less be on perfect language and more on the overall legibility of the language and logical structuring of the submission. If you feel that a lot of editorial work (maybe including “outside help”) is needed, you will have to judge whether this can likely happen before the camera-ready version or not.
Overall evaluation.
Please provide an overall evaluation of the work’s readiness for publication.
2: accept
1: weak accept
-1: weak reject
-2: reject
Comment: The overall recommendation should follow from the prior ratings and be explained by the overall justification. Of course, you will all have your own idea of the weights for combining the single factors and that is perfectly okay – this is why we have a discussion phase and meta-reviewing.
Justification of overall evaluation.
Considering the categories above and the type of the submission, please identify what aspects of the review guided your decision. Please provide enough detail to explain your choice. Indicate if you feel that an acceptance should be conditional and if there are clarifying questions you would want to ask the authors.
Comment: The justification should explain the scores for your ratings, in particular, you should explain why you felt that one of the aspects has not fulfilled the “accept” criteria and what kind of modifications you would expect for a camera ready version (or for a hypothetical future revision). As a suggestion, start with a short summary of the submission and then write one paragraph for each of the aspects. Clarifying questions will be sent to authors after the discussion has started and the answer can be used to modify the review before the reviewing period is over.
Best paper award.
Do you think this submission is a candidate for the best paper award?
Comment: The award will be given solely based on the paper itself. The criteria are the scores of the reviews, reviewers’ indication of possible candidates and meta-reviewers’ indication of possible candidates.