This study aimed to deepen our understanding of the behavioral and neurocognitive processes involved in human-human and human-robot communication in a more ecologically valid setting compared to the traditional neurolinguistic paradigms. We collected a novel open-source dataset (N=30 for human-human and N=20 for human-robot interactions), that includes fMRI, eye-tracking, segmented audio, video, and behavioral data, resulting in 30 minutes of free conversations per participant. To enable unrestricted, spontaneous robot behavior, we employed a novel VR-mediated teleoperation system. Our mixed design allowed us to compare participants' perception of humans and robots across three within-subject conditions of conversational engagement: Engaged Communicator, Active Listener, and Passive Listener. We provide an open-access dataset, replicable code for the teleoperation system, and an initial analysis of fMRI, behavioral, and speech data. We observed distinct neural profiles: speaking to the human agent recruited more higher-level frontal regions associated with socio-pragmatic processes, while listening to the robot recruited more sensory areas, including auditory and visual regions. Engagement levels and agent types also affected speech and behavioral patterns, offering valuable insights into conversational dynamics in human-human and human-robot interactions.

Pragmatics and Theory of Mind (ToM) are at play during conversational interaction, but the relationship between the two is a matter of debate. Using resting-state fMRI data, we investigate a potential segregation of the two domains by considering functional connectivity within and between the ToM and language networks, and their relation to pragmatic behavioural measures. We also study the connectivity of two cortical clusters, one in the left intraparietal sulcus and one in the bilateral dorsal precuneus. These clusters are located outside both the ToM and language networks and were previously found to covary with individual pragmatic variability. The results show that these two clusters are functionally connected at rest and that their degree of connectivity is related to pragmatic behaviour. On the other hand, there was no relation between pragmatic behaviour and degrees of connectivity involving the ToM and language networks. Furthermore, the two clusters were not connected to either the ToM or language networks. In conclusion, we suggest that the domain of pragmatics is partially segregated from ToM, and provide further support that the two clusters outside the ToM and language networks are pragmatically relevant.

In conversation, the literal form of an utterance often differs from the speaker’s intended meaning. Nonetheless, we typically understand the meaning of such indirect speech acts (ISAs). This ability constitutes one of several pragmatic abilities. fMRI studies in adults show that processing ISAs (compared to direct speech acts) elicits greater activation in language and Theory of Mind (ToM) regions. In our previous investigation, we found that individual differences in pragmatic skill manifested in two clusters outside these networks during ISA processing. However, less is known about the neurodevelopmental trajectory of ISA processing in adolescence. Using an established fMRI paradigm, we investigated ISA processing (1) across adolescence and (2) into adulthood by comparing adolescents to adults with varying pragmatic skill levels. We collected new fMRI data from adolescent participants (n = 51), divided into younger (ages 13—15, n = 24) and older (ages 16—18, n = 27) age groups. Participants listened to short conversations ending with indirect or direct speech acts. We also included our previously collected adult data set (n = 57) in which participants completed the same paradigm. The adults were subdivided into two groups based on performance on two pragmatic behavioral tests: a low-scoring group (LS, n = 28) and a high-scoring group (HS, n = 29). We contrasted activity during ISAs and direct speech acts within each group and tested for differences between groups, using both whole-brain and ROI-based analyses. Our adolescent results replicated prior findings in adults. Activity increased between adolescence and adulthood in the posterior superior medial prefrontal cortex (mPFC). We also found age-related differences in activation in the left posterior intraparietal sulcus (IPS), overlapping with the canonical “frontoparietal network”. Younger adolescents showed greater activation in this cluster when listening to direct speech than ISA, whereas older adolescents showed little difference between conditions. We observed a similar pattern when comparing the adult groups, with the LS and HS groups mirroring the young and old adolescents, respectively. Activity in a cluster within the posterior cingulate cortex also differed between adolescents and adults, and between the LS and HS groups. The increasing activity in the mPFC suggests that this region may be a marker of pragmatic development, possibly reflecting a shift from general ToM to more conversation-specific pragmatic representations. Furthermore, we suggest that the increased activity in the IPS in the younger adolescents and the LS group during the processing of direct speech acts may reflect engagement of additional control functions. These groups may be less certain about whether a direct interpretation is suitable, possibly over-interpreting direct trials. These control functions might involve (1) working memory to recall the previous conversational context and (2) structuring thoughts in an (erroneous) attempt to infer what a speaker may have implicitly meant. The similarity between young adolescents and the LS group indicates that pragmatics is still developing during adolescence, and that a delayed development may lead to persistent difficulties in adulthood. These findings suggest that aspects of pragmatic development depend on cognitive abilities other than language and ToM.

A key question in research on the neurobiology of language is to which extent the language production and comprehension systems share neural infrastructure, but this question has not been addressed in the context of conversation. We utilized a public fMRI dataset where 24 participants engaged in unscripted conversations with a confederate outside the scanner, via an audio-video link. We provide evidence indicating that the two systems share neural infrastructure in the left-lateralized perisylvian language network, but diverge regarding the level of activation in regions within the network. Activity in the left inferior frontal gyrus was stronger in production compared to comprehension, while comprehension showed stronger recruitment of the left anterior middle temporal gyrus and superior temporal sulcus, compared to production. Although our results are reminiscent of the classical Broca–Wernicke model, the anterior (rather than posterior) temporal activation is a notable difference from that model. This is one of the findings that may be a consequence of the conversational setting, another being that conversational production activated what we interpret as higher-level socio-pragmatic processes. In conclusion, we present evidence for partial overlap and functional asymmetry of the neural infrastructure of production and comprehension, in the above-mentioned frontal vs temporal regions during conversation.

Pragmatics and Theory of Mind (ToM) are at play during conversational interaction, but therelationship between the two, in the conversational context, is a matter of debate. Using resting statefMRI-data, we investigate a potential segregation of the two domains by considering functionalconnectivity within the ToM network and between the ToM and language networks and theirrelation to pragmatic measures. We also studied the connectivity of two cortical clusters: a leftsuperior parietal and a bilateral dorsal precuneal cluster. These clusters are located outside both theToM and language networks and were previously found to covary with individual pragmaticvariability. The results show that these two clusters are functionally connected at rest, and that thedegree of connectivity is related to pragmatic behavior. On the other hand, there was no relationbetween pragmatic behavior and the degree of connectivity involving the ToM network.Furthermore, the ToM network and the two clusters were not connected. Approaches such ascognitive spaces/maps could provide new perspectives on pragmatics and related domains. Inconclusion, we suggest that the domain of pragmatics is partially segregated from ToM, and providefurther support that the two clusters outside the ToM and language networks are pragmaticallyrelevant. 

We investigated how verbal communication with a robot differs from talking to a human in terms of brain activity by analysing an open-source fMRI dataset. We focused on modeling conversational dynamics rather than conversation as a whole, by analysing fine-grained events, in particular turn initiation. The results indicate that turn initiation in a conversation with a human involves higher activation in auditory and visual cortex than turn initiation with a robot. Conversely, listening to the robot showed higher engagement of auditory cortex than listening to a human. We suggest that verbal and non-verbal turn-taking cues provided by the human agent engage more cognitive processing for picking up the turn. On the other hand, listening to a robot agent requires more processing than listening to a human. Both findings suggest that the accurate simulation of appropriate turn-taking cues and behaviors will help robots to establish more natural conversation dynamics and that the use of brain imaging can provide valuable objective measurements for assessing user states in human-robot interaction.

Frontal lobe organisation displays a functional gradient, with overarching processing goals located in parts anterior to more subordinate goals, processed more posteriorly. Functional specialisation for syntax and phonology within language relevant areas has been supported by meta-analyses and reviews, but never directly tested experimentally. We tested for organised functional specialisation by manipulating syntactic case and phonotactics, creating violations at the end of otherwise matched and predictable sentences. Both violations led to increased activation in expected language regions. We observe the clearest signs of a functional gradient for language processing in the medial frontal cortex, where syntactic violations activated a more anterior portion compared to the phonotactic violations. A large overlap of syntactic and phonotactic processing in the left inferior frontal gyrus (LIFG) supports the view that general structured sequence processes are located in this area. These findings are relevant for understanding how sentence processing is implemented in hierarchically organised processing steps in the frontal lobe.

Interest is surging around the ”next-word-predictability” task that allowed large language models to reach their current capacity. It is sometimes claimed that prediction is enough to model language production. We set out to study predictability in an interactive setting. The current fMRI study used the information-theoretic measure of surprisal – the negative log-probability of a word occurring given the preceding linguistic context, estimated by a pre-trained language model (GPT-2). Surprisal has been shown to correlate with bottom-up processing located in the bilateral middle and superior temporal gyri (MTG/STG) during narrative comprehension (Willems et al., 2016). Still, surprisal has never been used to investigate conversational comprehension or any kind of language production. We hypothesized that previous results on surprisal in narrative comprehension would be replicated with conversational comprehension and that next-word- predictability would not encompass language production processes. We utilized a publicly available fMRI dataset in which participants (N=24) engaged in unscripted conversations (12 min/participant) via an audio- video link with a confederate outside the scanner. The conversational events Production, Comprehension, and Silence were modeled in a whole-brain analysis. Two parametric modulations of production and comprehension were added: (1) log-transformed context-independent word frequency (control regressor) and (2) surprisal. Production-surprisal and Comprehension-surprisal were respectively contrasted against the implicit baseline. These contrasts were compared with the contrasts Production and Comprehension vs implicit baseline. If surprisal merely indexed part of the activity in the latter, broader contrasts, this provides a handle on production and comprehension processes beyond next-word-predictability. For surprisal in conversational production, we observed statistically signi�cant clusters in the left inferior frontal gyrus (LIFG), the medial frontal gyrus, and the motor cortex. Importantly, Production vs implicit baseline showed bilateral STG activation while STG was not parametrically modulated by surprisal. Moreover, the bilateral MTG/STG were the only clusters active for Comprehension vs implicit baseline and they were also modulated by surprisal. For comprehension, we thus replicated the previous narrative comprehension study (Willems et al.,2016), showing that unpredictable words activate the bilateral MTG/STG also in conversational settings. Next- word-predictability is thus so far a good model for conversational comprehension. For production, however, the next-word-predictability task helped to hone in on what is sometimes considered core production machinery in LIFG. Several functional interpretations of the STG recruitment during production are possible (such as monitoring for speech errors), but the current results point in the direction of two important conclusions: (1) a functional division of the frontal and temporal cortices during production, where the frontal component is prediction-related, and (2) that language processing during production is more than prediction, at least at the word-level. We provide a functional handle on such extra-predictive processes.

Face-to-face communication requires skills that go beyond core language abilities. In dialogue, we routinely make inferences beyond the literal meaning of utterances and distinguish between different speech acts based on, e.g., contextual cues. It is, however, not known whether such communicative skills potentially overlap with core language skills or other capacities, such as theory of mind (ToM). In this functional magnetic resonance imaging (fMRI) study we investigate these questions by capitalizing on individual variation in pragmatic skills in the general population. Based on behavioral data from 199 participants, we selected participants with higher vs. lower pragmatic skills for the fMRI study (N = 57). In the scanner, participants listened to dialogues including a direct or an indirect target utterance. The paradigm allowed participants at the whole group level to (passively) distinguish indirect from direct speech acts, as evidenced by a robust activity difference between these speech acts in an extended language network including ToM areas. Individual differences in pragmatic skills modulated activation in two additional regions outside the core language regions (one cluster in the left lateral parietal cortex and intraparietal sulcus and one in the precuneus). The behavioral results indicate segregation of pragmatic skill from core language and ToM. In conclusion, contextualized and multimodal communication requires a set of interrelated pragmatic processes that are neurocognitively segregated: (1) from core language and (2) partly from ToM.