Machine learning methods can be used to reduce the manual workload in exam grading, making it possible for teachers to spend more time on other tasks. However, when it comes to grading exams, fully eliminating manual work is not yet possible even with very accurate automated grading, as any grading mistakes could have significant consequences for the students. Here, the evaluation of an automated grading approach is therefore extended from measuring workload in relation to the accuracy of automated grading, to also measuring the overall workload required to correctly grade a full exam, with and without the support of machine learning. The evaluation was performed during an introductory computer science course with over 400 students. The exam consisted of 64 questions with relatively short answers and a two-step approach for automated grading was applied. First, a subset of answers to the exam questions was manually graded and next used as training data for machine learning models classifying the remaining answers. A number of different strategies for how to select which answers to include in the training data were evaluated. The time spent on different grading actions was measured along with the reduction of effort using clustering of answers and automated scoring. Compared to fully manual grading, the overall reduction of workload was substantial-between 64% and 74%-even with a complete manual review of all classifier output to ensure a fair grading.

BackgroundIn patients who underwent colorectal surgery, an existing semi-automated surveillance algorithm based on structured data achieves high sensitivity in detecting deep surgical site infections (SSI), however, generates a significant number of false positives. The inclusion of unstructured, clinical narratives to the algorithm may decrease the number of patients requiring manual chart review. The aim of this study was to investigate the performance of this semi-automated surveillance algorithm augmented with a natural language processing (NLP) component to improve positive predictive value (PPV) and thus workload reduction (WR).MethodsRetrospective, observational cohort study in patients who underwent colorectal surgery from January 1, 2015, through September 30, 2020. NLP was used to detect keyword counts in clinical notes. Several NLP-algorithms were developed with different count input types and classifiers, and added as component to the original semi-automated algorithm. Traditional manual surveillance was compared with the NLP-augmented surveillance algorithms and sensitivity, specificity, PPV and WR were calculated.ResultsFrom the NLP-augmented models, the decision tree models with discretized counts or binary counts had the best performance (sensitivity 95.1% (95%CI 83.5-99.4%), WR 60.9%) and improved PPV and WR by only 2.6% and 3.6%, respectively, compared to the original algorithm.ConclusionsThe addition of an NLP component to the existing algorithm had modest effect on WR (decrease of 1.4-12.5%), at the cost of sensitivity. For future implementation it will be a trade-off between optimal case-finding techniques versus practical considerations such as acceptability and availability of resources.

Today, collaborative learning has become quite central as a method for learning, and over the past decades, a large number of studies have demonstrated the benefits from various theoretical and methodological perspectives. This study proposes a novel approach that utilises Natural Language Processing(NLP) methods, particularly pre-trained word embeddings, to automatically create homogeneous or heterogeneous groups of students in terms of knowledge and knowledge gaps expressed in assessments. The two different ways of creating groups serve two different pedagogical purposes: (1) homogeneous group formation based on students’ knowledge can support and make teachers’ pedagogical activities such as feedback provision more time efficient, and (2) the heterogeneous groups can support and enhance collaborative learning. We evaluate the performance of the proposed approach through experiments with a dataset from a university course in programming didactics.

Knowledge tracing—where a machine models the students’ knowledge as they interact with coursework—is a well-established area in the field of Artificial Intelligence in Education. In this paper, an ensemble approach is proposed that addresses existing limitations in question-centric knowledge tracing and achieves the goal of predicting future question correctness. The proposed approach consists of two models; one is Light Gradient Boosting Machine (LightGBM) built by incorporating all relevant key features engineered from the data. The second model is a Multiheaded-Self-Attention Knowledge Tracing model (MSAKT) that extracts historical student knowledge of future question by calculating their contextual similarity with previously attempted questions. The proposed model’s effectiveness is evaluated by conducting experiments on a big Kaggle dataset achieving an Area Under ROC Curve (AUC) score of 0.84 with 84% accuracy using 10fold cross-validation.

In this paper, we propose a counterfactual explanations-based approach to provide an automatic and intelligent recommendation that supports student's self-regulation of learning in a data-driven manner, aiming to improve their performance in courses. Existing work under the fields of learning analytics and AI in education predict students' performance and use the prediction outcome as feedback without explaining the reasons behind the prediction. Our proposed approach developed an algorithm that explains the root causes behind student's performance decline and generates data-driven recommendations for action. The effectiveness of the proposed predictive model that constitutes the intelligent recommendations is evaluated, with results demonstrating high accuracy.

Collaborative learning has grown more popular as a form of instruction in recent decades, with a significant number of studies demonstrating its benefits from many perspectives of theory and methodology. However, it has also been demonstrated that effective collaborative learning does not occur spontaneously without orchestrating collaborative learning groups according to the provision of favourable group criteria. Researchers have investigated different foundations and strategies to form such groups. However, the group criteria semantic information, which is essential for classifying groups, has not been explored. To capture the group criteria semantic information, we propose a novel Natural Language Processing (NLP) approach, namely using pre-trained word embedding. Through our approach, we could automatically form homogeneous and heterogeneous collaborative learning groups based on student’s knowledge levels expressed in assessments. Experiments utilising a dataset from a university programming course are used to assess the performance of the proposed approach.

Formative feedback has long been recognised as an effective tool for student learning, and researchers have investigated the subject for decades. However, the actual implementation of formative feedback practices is associated with significant challenges because it is highly time-consuming for teachers to analyse students’ behaviours and to formulate and deliver effective feedback and action recommendations to support students’ regulation of learning. This paper proposes a novel approach that employs learning analytics techniques combined with explainable machine learning to provide automatic and intelligent feedback and action recommendations that support student’s self-regulation in a data-driven manner, aiming to improve their performance in courses. Prior studies within the field of learning analytics have predicted students’ performance and have used the prediction status as feedback without explaining the reasons behind the prediction. Our proposed method, which has been developed based on LMS data from a university course, extends this approach by explaining the root causes of the predictions and by automatically providing data-driven intelligent recommendations for action. Based on the proposed explainable machine learning-based approach, a dashboard that provides data-driven feedback and intelligent course action recommendations to students is developed, tested and evaluated. Based on such an evaluation, we identify and discuss the utility and limitations of the developed dashboard. According to the findings of the conducted evaluation, the dashboard improved students’ learning outcomes, assisted them in self-regulation and had a positive effect on their motivation.

This paper proposes a novel approach that employs learning analytics techniques combined with explainable machine learning to provide automatic and intelligent actionable feedback that supports students self-regulation of learning in a data-driven manner. Prior studies within the field of learning analytics predict students’ performance and use the prediction status as feedback without explaining the reasons behind the prediction. Our proposed method, which has been developed based on LMS data from a university course, extends this approach by explaining the root causes of the predictions and automatically provides data-driven recommendations for action. The underlying predictive model effectiveness of the proposed approach is evaluated, with the results demonstrating 90 per cent accuracy.

Health care and clinical practice generate large amounts of text detailing symptoms, test results, diagnoses, treatments, and outcomes for patients. This clinical text, documented in health records, is a potential source of knowledge and an underused resource for improved health care. The focus of this work has been text mining of clinical text in the domain of cancer care, with the aim to develop and evaluate methods for extracting relevant information from such texts. Two different types of clinical documentation have been included: clinical notes from electronic health records in Swedish and Norwegian pathology reports.

Free text, and clinical text in particular, is considered as a kind of unstructured information, which is difficult to process automatically. Therefore, information extraction can be applied to create a more structured representation of a text, making its content more accessible for machine learning and statistics. To this end, this thesis describes the development of an efficient and accurate tool for information extraction for pathology reports.

Another application for clinical text mining is risk prediction and diagnosis prediction. The goal for such prediction is to create a machine learning model capable of identifying patients at risk of a specific disease or some other adverse outcome. The motivation for cancer diagnosis prediction is that an early diagnosis can be beneficial for the outcome of treatment. Here, a disease prediction model was developed and evaluated for prediction of cervical cancer. To create this model, health records of patients diagnosed with cervical cancer were processed in two steps. First, clinical events were extracted from free text clinical notes through the use of named entity recognition. The extracted events were next combined with other event types, such as diagnosis codes and drug codes from the same health records. Finally, machine learning models were trained for predicting cervical cancer, and evaluation showed that events extracted from the free text records were the most informative event type for the diagnosis prediction.

Electronic health records (EHRs) contain rich documentation regarding disease symptoms and progression, but EHR data is challenging to use for diagnosis prediction due to its high dimensionality, relative scarcity, and substantial level of noise. We investigated how to best represent EHR data for predicting cervical cancer, a serious disease where early detection is beneficial for the outcome of treatment. A case group of 1321 patients with cervical cancer were matched to ten times as many controls, and for both groups several types of events were extracted from their EHRs. These events included clinical codes, lab results, and contents of free text notes retrieved using a LSTM neural network. Clinical events are described with great variation in EHR texts, leading to a very large feature space. Therefore, an event hierarchy inferred from the textual events was created to represent the clinical texts. Overall, the events extracted from free text notes contributed the most to the final prediction, and the hierarchy of textual events further improved performance. Four classifiers were evaluated for predicting a future cancer diagnosis where Random Forest achieved the best results with an AUC of 0.70 from a year before diagnosis up to 0.97 one day before diagnosis. We conclude that our approach is sound and had excellent discrimination at diagnosis, but only modest discrimination capacity before this point. Since our study objective was earlier disease prediction than such, we propose further work should consider extending patient histories through e.g. the integration of primary health records preceding referral to hospital.