Aims and objectives: This study investigates the collocational competence of heritage language (HL) speakers and evaluates the feasibility of using automatic assessment to distinguish language proficiency on the collocational levels. We hypothesize that higher collocational scores, associated with the learner texts, indicate more advanced collocational proficiency.

Methodology: The study utilizes t-score metrics to assess the significance of bigrams. The protocol involves calculating the strength of each collocation in a large L1 reference corpus, extracting bigrams from HL data, and comparing them with the L1 data. If a match is found, the HL bigram is assigned the corresponding L1 collocation t-score.

Data and analysis: Research data comprise 118 essays (32,225 tokens) from the National Post-Secondary Russian Essay Contest, USA. The study also utilizes the L1 reference corpus Araneum Russicum and three validation datasets: (a) L1 collocations with the highest t-scores from Araneum Russicum, (b) randomly generated bigrams with low t-scores, and (c) L1 collocations from literary fiction.

Findings: The proposed method is evaluated against three validation datasets and proven to differentiate them adequately. However, the main findings indicate that t-scores do not effectively differentiate proficiency levels among HL learners. The similarity in t-score distributions between HL learner texts and L1 literary texts highlights the challenges of using collocations as a standalone marker of proficiency.

Originality: This study proposes and validates a new method for automatically assessing collocational proficiency. It emphasizes the challenges of using collocations as a definitive marker of HL proficiency but also highlights the benefits of this analysis for L1 language learning and research.

Significance/implications: Automatic collocational analysis shows potential for language proficiency assessment and for developing educational programs to enhance students’ collocational knowledge. By advancing our understanding of collocational patterns and their implications for language use, this study contributes to the development of more effective language assessment tools, fostering greater proficiency and fluency among language learners.