Objective: The objective of this article is to discuss the pedagogical and practical need for automated assessment tools that enable teachers, researchers, and other language practitioners to relatively quickly and automatically assess the general proficiency of second language (L2) speakers according to a number of different linguistic parameters, specifically the use of collocations.

Introduction: The Introduction discusses existing approaches to the concept of collocation and its role in L2 corpus studies.

Methods: In the main part, Toward the automatic detection of collocational strength, we describe an approach to assessing collocations in Russian L2 written production, which is based on the corpus-driven contrastive analysis. In brief, this approach proposes ranking L2 collocations in accordance with the “collocational scores” of bigrams extracted from a reference (L1) corpus, with the assumption that the greater number of native-like bigrams in an L2 text and the higher collocational scores associated with them, the more proficient is the learner who created the text. To test this hypothesis, we conducted a series of experiments based on an L1 reference corpus and split the L2 datasets into four proficiency groups (Beginner, Intermediate, Advanced, and Superior).

Results and evaluation: The findings, presented in the section Results and evaluation of the methods, established an empirical minimum and maximum of mean reciprocal rank (MRR) values, and the variation of the MRR across the four proficiency levels. In the last sections, we discuss shortcomings of the methods and propose further steps for overcoming them. We believe that the proposed method for assessing the collocational skill of Russian L2 writers can be successfully integrated into a bigger system capable of multifaceted automatic assessment of Russian learner texts.