The emergence of pretrained language models has profoundly reshaped natural language processing, serving as foundation models for a wide range of tasks. Over the past decade, pretrained language models have evolved significantly, leading to the development of different types of models and approaches for utilising them. This progression spans from static to contextual models and from smaller models to more powerful, generative large language models. The increasing capabilities of these models have, in turn, led to growing interest in exploring new use cases and applications across various domains, including education, where digitalisation has created opportunities for AI applications that leverage pretrained language models, particularly due to the abundance of text data in educational contexts.

This thesis explores the educational utility of pretrained language models, specifically by investigating how different paradigms of these models can be applied to address tasks in education. These paradigms include various methodologies for leveraging the knowledge embedded in pretrained language models, such as embeddings, fine-tuning, prompt-based learning, and in-context learning. For collaborative learning group formation, a clustering approach based on pretrained embeddings is proposed, enabling the creation of either homogeneous or heterogeneous groups depending on the specific learning situation. For automated essay scoring, a pretrained language model is fine-tuned using both the essay instructions and the essay text as input; the proposed method also highlights key topical sentences that contribute to the predicted essay score. For educational question generation, a method based on prompt-based learning is introduced and shown to be more data-efficient than existing methods. Finally, for educational question answering, certain limitations of the in-context learning (or prompting) paradigm, such as a tendency of large language models to hallucinate or miscalculate, are addressed. Specifically, workflows and prompting strategies based on retrieval-augmented generation and tool-augmented generation are proposed, allowing large language models to ground answers in specific learning materials and to leverage external tools, such as calculators and knowledge bases, within chain-of-thought reasoning processes. These strategies are shown to produce more reliable and transparent answers to complex questions.

Through five empirical studies, methodological innovations within each paradigm of pretrained language models are proposed and evaluated for specific educational use cases. In addition to contributing methodologically to natural language processing, the results demonstrate the potential utility of pretrained language models in educational AI applications, thereby advancing the field of technology enhanced learning. The proposed methods not only improve predictive performance on specific tasks but also aim to enhance the transparency of pretrained language models, which is essential for building reliable and trustworthy educational AI applications.

Using AI for automatically linking exercises to curriculum goals can support many educational use cases and facilitate teaching-to-the-curriculum by ensuring that exercises adequately reflect and encompass the curriculum goals, ultimately enabling curriculum-based assessment. Here, we introduce this novel task and create a manually labeled dataset where two types of diagnostic tests are linked to curriculum goals for Biology G7-9 in Sweden. We cast the problem both as an information retrieval task and a multi-class text classification task and explore unsupervised approaches to both, as labeled data for such tasks is typically scarce. For the information retrieval task, we employ SOTA embedding model ADA-002 for semantic textual similarity (STS), while we prompt a large language model in the form of ChatGPT to classify diagnostic tests into curriculum goals. For both task formulations, we investigate different ways of using textbook content as a pivot and provide additional context for linking diagnostic tests to curriculum goals. We show that a combination of the two approaches in a retrieval-augmented generation model, whereby STS is used for retrieving textbook content as context to ChatGPT that then performs zero-shot classification, leads to the best classification accuracy (73.5%), outperforming both STS-based classification (67.5%) and LLM-based classification without context (71.5%). Finally, we showcase how the proposed method could be used in pedagogical practices.

Automated Essay Scoring (AES) automatically allocates scores to essays at scale and may help teachers reduce the heavy burden during grading activities. Recently, researchers have deployed neural-based AES approaches to improve upon the state-of-the-art AES performance. These neural-based AES methods mainly take student essays as the sole input and focus on learning the relationship between student essays and essay scores through deep neural networks. However, their only product, the predicted holistic score, is far from providing adequate pedagogical information, such as automated writing evaluation (AWE). In this work, we propose Topic-aware BERT, a new method of learning relations among scores, student essays, as well as topical information in essay instructions. Beyond improving the AES benchmark performance, Topic-aware BERT can automatically retrieve key topical sentences in student essays by probing self-attention maps in intermediate layers. We evaluate the performance of Topic-aware BERT of different variants to (i) perform AES and (ii) retrieve key topical sentences using the open dataset Automated Student Assessment Prize and a manually annotated dataset. Our experiments show that Topic-aware BERT achieves a strong AES performance compared with the previous best neural-based AES methods and demonstrates effectiveness in identifying key topical sentences in argumentative essays.

As an integral part of AI-enhanced learning, a content recommender automatically filters and recommends relevant learning materials to the learner or the instructor in a learning system. It can effectively help instructors in pedagogical practices and support students in self-regulated learning. Content recommendation technologies and applications have been studied extensively, however, the SOTA technologies have not adequately adapted to the education domain and there is very limited research on how different models and solutions can be applied in the Swedish context and for multiple subjects. In this paper, we develop a text similarity-based content recommender system. Specifically, given a quiz, we automatically recommend supportive learning resources as a reference to the answer and link back to the textbook sections where the examined knowledge points reside. We present a generic method for Swedish educational content recommendations using the most representative models, evaluate and analyze in multi-dimensions such as model types, pooling methods, subjects etc. The best results are obtained by Sentence-BERT (SBERT) with max paragraph-level pooling, outperforming traditional Natural Language Processing (NLP) models and knowledge graph-based models, obtaining on average 95% in Recall@3 and 82% in MRR, and outstanding in dealing with texts containing symbols, equations or calculations. This research provides empirical evidence and analysis, and can be used as a guidance when building a Swedish educational content recommender.

Practice and exam-style questions, as essential educational tools, contribute to educators’ effective teaching. Automatic question generation (QG) is a promising technique that can eliminate the manual effort of constructing questions and boost technology-enhanced education systems. Recently, deep neural network-based question-generation approaches have significantly improved upon state-of-the-art of question generation. Nevertheless, these approaches are often developed based on huge and non-educational datasets consisting of over 100,000 examples, which negatively affect the scalability and reliability of the educational QG systems. This study proposes a prompt-based learning QG approach that could generate questions in a data-effective way. The proposed prompt-based learning QG approach is trained and evaluated on a general dataset SQuAD, and an educational dataset SciQ. Experiment results demonstrate that our approach outperforms existing best QG models by a vast margin in data-effective scenarios and could generate high-quality educational questions with as few as 1,000 training examples.

Large language models (LLMs), such as ChatGPT, have shown remarkable performance on various natural language processing (NLP) tasks, including educational question answering (EQA). However, LLMs generate text entirely based on knowledge obtained during pre-training, which means they struggle with recent information or domain-specific knowledge bases. Moreover, only providing answers to questions posed to LLMs without any grounding materials makes it difficult for students to judge their validity.

We therefore propose a method for integrating information retrieval systems with LLMs when developing EQA systems, which in addition to improving EQA performance grounds the answers in the educational context. Our experiments show that the proposed system outperforms vanilla ChatGPT with a vast margin of 110.9%, 67.8%, and 43.3% on BLEU, ROUGE, and METEOR scores. In addition, we argue that the use of the retrieved educational context enhances the transparency and reliability of the EQA process, making it easier to determine the correctness of the answers.

Educational concepts are the core of teaching and learning. From the perspective of educational technology, concepts are essential meta-data, represen- tative terms that can connect different learning materials, and are the foundation for many downstream tasks. Some studies on automatic concept extraction have been conducted, but there are no studies looking at the K-12 level and focused on the Swedish language. In this paper, we use a state-of-the-art Swedish BERT model to build an automatic concept extractor for the Biology subject using fine- annotated digital textbook data that cover all content for K-12. The model gives a recall measure of 72% and has the potential to be used in real-world settings for use cases that require high recall. Meanwhile, we investigate how input data fea- tures influence model performance and provide guidance on how to effectively use text data to achieve the optimal results when building a named entity recognition (NER) model.

Automated Essay Scoring (AES) automatically assigns scores to essays at scale and may help to support teachers' grading activities. Recently, AES methods based on deep neural networks (DNN) have significantly improved upon the state-of-the-art performance by learning relations between holistic essay scores and student essays. However, DNN-based AES methods function like black-box, negatively affecting the ability to provide automated writing evaluation (AWE). In this work, we proposed a new method, topic-aware BERT, based on fine-tuning the pre-trained language model to learn relations between essay scores and text representations of student essays as well as topical information in essay writing instructions. Moreover, we propose an approach to automatically retrieve key topical sentences in student essays by probing self-attention maps in intermediate layers of topic-aware BERT. We evaluate the performance of topic-aware BERT to (i) perform AES and (ii) retrieve key topical sentences using the open dataset Automated Student Assessment Prize and a manually annotated dataset, respectively. Our model achieves a strong AES performance compared with previous state-of-the-art DNN-based methods and shows effectiveness in identifying key topical sentences in argumentative essays.

Today, collaborative learning has become quite central as a method for learning, and over the past decades, a large number of studies have demonstrated the benefits from various theoretical and methodological perspectives. This study proposes a novel approach that utilises Natural Language Processing(NLP) methods, particularly pre-trained word embeddings, to automatically create homogeneous or heterogeneous groups of students in terms of knowledge and knowledge gaps expressed in assessments. The two different ways of creating groups serve two different pedagogical purposes: (1) homogeneous group formation based on students’ knowledge can support and make teachers’ pedagogical activities such as feedback provision more time efficient, and (2) the heterogeneous groups can support and enhance collaborative learning. We evaluate the performance of the proposed approach through experiments with a dataset from a university course in programming didactics.

Knowledge tracing—where a machine models the students’ knowledge as they interact with coursework—is a well-established area in the field of Artificial Intelligence in Education. In this paper, an ensemble approach is proposed that addresses existing limitations in question-centric knowledge tracing and achieves the goal of predicting future question correctness. The proposed approach consists of two models; one is Light Gradient Boosting Machine (LightGBM) built by incorporating all relevant key features engineered from the data. The second model is a Multiheaded-Self-Attention Knowledge Tracing model (MSAKT) that extracts historical student knowledge of future question by calculating their contextual similarity with previously attempted questions. The proposed model’s effectiveness is evaluated by conducting experiments on a big Kaggle dataset achieving an Area Under ROC Curve (AUC) score of 0.84 with 84% accuracy using 10fold cross-validation.