The Swedish environmental and classification system for pharmaceuticals is a voluntary, industry-owned system with the purpose to provide environmental information about active pharmaceutical ingredients in the Swedish market. In this paper we report the results from a detailed evaluation of the accuracy and consistency of the risk assessments conducted within this system. The evaluation focused on the following three aspects: 1) comparison of the companies' risk assessments with the classification system's own guidance document, 2) how the risk assessments are affected if additional effect data is used, and 3) the consistency of different risk assessments for the same pharmaceutical substance. The analyses show that the system's guidance gives no criteria for when to consider a study "long-term" or "short-term", and that this confusion affected the outcome of the risk assessments in some cases. Furthermore, when the system's guidance document is followed and the risk assessment was supplemented with effect data from the open scientific literature, then the risk classification for a substantial number of the evaluated substances was altered. Our analyses also revealed that in some cases risk assessors disagree on the outcome of the assessment for the same active pharmaceutical ingredient. Finally we list some recommendations to improve the classification system. The recommendations include clarifying concepts and instructions in the guidance document, introduction of a standardized way of reporting data to the website, and promotion of use of non-standard test data when considered the most relevant.