In this paper, we develop an analytical tool for the role of the physical environment in mathematics education. We do this by extending the didactical triangle with the physical environment as a fourth actor and test it in a review of literature concerning the physical environment and mathematics education. We find that one role played by the physical environment, in relation to mathematical content, is to portray the content in focus, such as geometry and scale. When focusing on teachers, students, and the interaction between them, the role of the physical environment appears to be a precondition, either positive (enabling) or negative (hindering). Many of the findings are valid for education in general as well, such as the importance of building status.