The Common Vulnerability Scoring System (CVSS) is the most widely used standard for quantifying the severity of security vulnerabilities. For instance, all vulnerabilities in the US National Vulnerability Database are scored according to this system. Unfortunately, it is largely unexplored whether or not its scores are accurate. This paper studies this property through a survey with opinions by 384 experts, covering more than 3000 vulnerabilities. The results show that the mean disagreement between the judgments of the experts and the CVSS Base Score is 0.38, with a variance of 4.46 (on a scale from 0 to 10). The direction of this difference depends on the type of vulnerability that is concerned. The experts then suggest a number of possible revisions to the CVSS that could explain this difference.