Non-Technical summary Transdisciplinary sustainability scientists work with many different actors in pursuit of change. In so doing they make choices about why and how to engage with different perspectives in their research. Reflexivity-active individual and collective critical reflection-is considered an important capacity for researchers to address the resulting ethical and practical challenges. We developed a framework for reflexivity as a transformative capacity in sustainability science through a critical systems approach, which helps make any decisions that influence which perspectives are included or excluded in research explicit. We suggest that transdisciplinary sustainability research can become more transformative by nurturing reflexivity. Technical summary Transdisciplinary sustainability science is increasingly applied to study transformative change. Yet, transdisciplinary research involves diverse actors who hold contrasting and sometimes conflicting perspectives and worldviews. Reflexivity is cited as a crucial capacity for navigating the resulting challenges, yet notions of reflexivity are often focused on individual researcher reflections that lack explicit links to the collective transdisciplinary research process and predominant modes of inquiry in the field. This gap presents the risk that reflexivity remains on the periphery of sustainability science and becomes 'unreflexive', as crucial dimensions are left unacknowledged. Our objective was to establish a framework for reflexivity as a transformative capacity in sustainability science through a critical systems approach. We developed and refined the framework through a rapid scoping review of literature on transdisciplinarity, transformation, and reflexivity, and reflection on a scenario study in the Red River Basin (US, Canada). The framework characterizes reflexivity as the capacity to nurture a dynamic, embedded, and collective process of self-scrutiny and mutual learning in service of transformative change, which manifests through interacting boundary processes-boundary delineation, interaction, and transformation. The case study reflection suggests how embedding this framework in research can expose boundary processes that block transformation and nurture more reflexive and transformative research. Social media summary Transdisciplinary sustainability research may become more transformative by nurturing reflexivity as a dynamic, embedded, and collective learning process.