As been highlighted by many, for instance by PISA, Collaborative Problem Solving (CPS) is a critical and necessary 21st century skill across educational settings. While many initiatives have been launched to investigate the nature of these skills, fewer are the attempts to understand how they should be assessed. However, in 2015, the PISA organization presented a framework for assessing CPS skills. This paper reports on an exploratory study investigating the predictive validity of the PISA assessment framework and if and how modes of communication influence the assessment of 24 students' collaborative problem solving activities when using a computer-based assessment task system. The findings presented demonstrate that the PISA CPS assessment framework have a weak predictive validity, does not count for quality or productivity in communication, and that the mode of communication indeed influence CPS processes and in turn what is possible to assess.