Ambiguity is often recognized as an intrinsic aspect of addressing complex sustainability challenges. Nevertheless, in the practice of transdisciplinary sustainability research, ambiguity is often an ‘elephant in the room’ to be either side-stepped or reduced rather than explicitly mobilized in pursuit of solutions. These responses threaten the salience and legitimacy of sustainability science by masking the pluralism of real-world sustainability challenges and how research renders certain frames visible and invisible. Critical systems thinking (CST) emerged from the efforts of operational researchers to address theoretical and practical aspects of ambiguity. By adapting key concepts, frameworks, and lessons from CST literature and case studies, this paper aims to establish (1) an expansive conceptualization of ambiguity and (2) recommendations for operationalizing ambiguity as a valuable means of addressing sustainability challenges. We conceptualize ambiguity as an emergent feature of the simultaneous and interacting boundary processes associated with being, knowing, and intervening in complex systems, and propose Reflexive Boundary Critique (RBC) as a novel framework to help navigate these boundary processes. Our characterization of ambiguity acknowledges the boundary of a researcher’s subjective orientation and its influence on how ambiguity is exposed and mediated in research (being), characterizes knowledge as produced through the process of making boundary judgments, generating a partial, contextual, and provisional frame (knowing), and situates a researcher as part of the complexity they seek to understand, rendering any boundary process as a form of intervention that reinforces or marginalizes certain frames and, in turn, influences action (intervening). Our recommendations for sustainability scientists to operationalize ambiguity include (1) nurturing the reflexive capacities of transdisciplinary researchers to navigate persistent ambiguity (e.g., using our proposed framework of RBC), and (2) grappling with the potential for and consequences of theoretical incommensurability and discordant pluralism. Our findings can help sustainability scientists give shape to and embrace ambiguity as a fundamental part of rigorous sustainability science.