Generative Artificial Intelligence (GenAI) is a subject of intense interest among researchers and practitioners in the public sector, offering the prospect of transforming public administration through automation and improved efficiency. However, there is a lack of comprehensive synthesis of the growing body of literature exploring this technology's multifaceted impact on the sector. To contribute to the growing discourse and bridge the existing literature gap, this research undertook a synthesis of 25 recent studies, analysing the potential benefits, challenges, and strategies for effective GenAI implementation in public organisations. The result of our study indicates that while GenAI offers opportunities to streamline bureaucratic processes, improve service delivery, and enhance decision-making, organisations face various challenges as they implement GenAI. The most common challenges include ethical concerns, regulatory compliance, data privacy, workforce resistance, and transparency issues. These findings call for robust AI governance frameworks, human oversight of GenAI systems, organisational readiness strategies, and continued investment in AI literacy. Future research is necessary to further our understanding of GenAI and how public organisations can realise its benefits. To this end, we propose longitudinal studies evaluating specific GenAI applications and their long-term impact, governance framework development, and methods to enhance GenAI explainability and mitigate bias in public administration contexts.