Interaction design is usually described as being concerned with interactions with and through artifacts but independent of a specific implementation. Design work has been characterized as a conversation between the designer and the situation and this conversation poses a particular challenge for interaction design as interactions can be elusive and difficult to describe. Moreover, current trends in interaction design introduce physical materials to a higher degree resulting in even more complex design situations. There is a lack of knowledge about how interaction designers, and especially students, address the very phenomenon of interaction. This study contributes by describing how interaction design students attempt to address aspects of interaction and by presenting an in-depth analysis in the context of an interactionary-type design exercise. The quantitative and qualitative findings showed that (1) the design students brought up aspects of interactivity and dynamics through talk and gestures but (2) a comprehensive design idea about interaction did not guide the design work and they were to a little degree engaged in planning sequences of interactions or interaction on a longer time scale, (3) using physical materials disrupted interaction design, and, (4) there was a lack of continuity when addressing interaction compared to how proposals about artifacts were pursued. As interaction is the core of interaction design, the findings are discussed in terms of how the immaterial design materials may “talk back” to designers. Practical strategies for how the observed phenomena could be constructively addressed within interaction design education are suggested.