Assessment of teaching and scientific competence should be done with equal care at appointments. Despite the fact that this has been legislated in Sweden since the 1960s, research shows that there are challenges associated with reviewing teaching portfolios. The difficulty in making qualified judgments does not only apply to hiring and promotion. As systems for awarding pedagogical competence at Swedish higher education institutions have increased, an accentuated need for competent assessors have followed. The need has been met, among other things, through a national training in peer review of pedagogical competence conducted as a collaboration between several Swedish higher education institutions. During three cohorts of the course, between 2017 and 2020, participants were asked to answer questions about perceived challenges when reviewing teaching portfolios. The responses were analyzed using thematic content analysis which resulted in five themes focusing on: organizational legitimacy, reliability, validity, constructivity and professionalism. The results provide a basis for the development of the higher education institutions’ review processes, and can provide support to experts and people who have to make decisions about employment, promotion or educational qualification and also for those who work on compiling a teaching portfolio.