Programming courses suffer from low retention rates, believed to be caused by difficulties in learning programming concepts. Another possibility relates to how programming ability is assessed. Although extensively used, pen-and-paper is arguably not the best way to assess a student's programming ability. Although previous studies have chosen to implement lab exams as a replacement for pen-and-paper examinations, little consideration has been made to evaluate the extent of the exam mode effect in relation to the use of computers during summative assessment in programming courses, and to what degree this could affect the student's performance. This study aims to answer to: Are there any differences in performance between students using a computer, compared to students using pen-and-paper during summative assessment of programming ability among novice programmers? This could aid teachers to better assess students' programming ability in novice courses, which could in turn aid in student retention. An experimental approach has been applied. 20 students participated and were divided in two groups that were giving the same programming problems that was either solved and examined through pen-and-paper assessment or through computerised assessment. While some differences between the groups were noted, the overall results demonstrate no statistical significant difference between the groups in terms of performance.