Reinforcement learning algorithms have a dependency on observing rewards for actions taken. The relaxed setting of having fully observable rewards, however, can be infeasible in certain scenarios, due to either cost or the nature of the problem. Of specific interest here is the challenge of learning a policy when rewards are delayed, aggregated, and anonymous (DAAF). A problem which has been addressed in bandits literature and, to the best of our knowledge, to a lesser extent in the more general reinforcement learning (RL) setting. We introduce a novel formulation that mirrors scenarios encountered in real-world applications, characterized by intermittent and aggregated reward observations. To address these constraints, we develop four new algorithms: one employs least squares for true reward estimation; two and three adapt Q-learning and SARSA, to deal with our unique setting; and the fourth leverages a policy with options framework. Through a thorough and methodical experimental analysis, we compare these methodologies, demonstrating that three of them can approximate policies nearly as effectively as those derived from complete information scenarios, albeit with minimal performance degradation due to informational constraints. Our findings pave the way for more robust RL applications in environments with limited reward feedback.

Addressing complexity in transportation in cases such as disruptive trends or disaggregated management strategies has become increasingly important. This in turn is resulting in the rising adoption of Agent-Based and Activity-Based modeling. Still, a broad adoption is hindered by the high complexity and computational needs. For example, hundreds of parameters are involved in the calibration of Activity-Based models focused on behavioral theory, to properly frame the required detailed socio-economical characteristics. To address this challenge, this paper presents a novel Bayesian Optimization approach that incorporates a surrogate model defined as an improved Random Forest to automate the calibration process of the behavioral parameters. The presented solution calibrates the largest set of parameters yet, according to the literature, by combining state-of-the-art methods. To the best of the authors' knowledge, this is the first work in which such a high dimensionality is tackled in sequential model-based algorithm configuration theory. The proposed method is tested in the city of Tallinn, Estonia, for which the calibration of 477 behavioral parameters is carried out. The calibration process results in a satisfactory performance for all the major indicators, the OD matrix average mismatch is equal to 15.92 vehicles per day while the error for the overall number of trips is equal to 4%.

COVID-19 is a viral infectious disease that has created a global pandemic, resulting in millions of deaths and disrupting the world order. Different machine learning and deep learning approaches were considered to detect it utilizing different medical data. Thermal imaging is a promising option for detecting COVID-19 as it is low-cost, non-invasive, and can be maintained remotely. This work explores the COVID-19 detection issue using the thermal image and associated tabular medical data obtained from a publicly available dataset. We incorporate a multi-modal machine learning approach where we investigate the different combinations of medical and data type modalities to get an improved result. We use different machine learning and deep learning methods, namely random forests, Extreme Gradient Boosting (XGBoost), Multilayer Perceptron (MLP), and Convolutional Neural Network (CNN). Overall multi-modal results outperform any single modalities, and it is observed that the thermal image is a crucial factor in achieving it. XGBoost provided the best result with the area under the receiver operating characteristic curve (AUROC) score of 0.91 and the area under the precision-recall curve (AUPRC) score of 0.81. We also report the average of leave-one-positive-instance-out cross- validation evaluation scores. This average score is consistent with the test evaluation score for random forests and XGBoost methods. Our results suggest that utilizing thermal image with associated tabular medical data could be a viable option to detect COVID-19, and it should be explored further to create and test a real-time, secure, private, and remote COVID-19 detection application in the future.

Despite the increased attention to explainable AI, explainability methods for understanding reinforcement learning (RL) agents have not been extensively studied. Failing to understand the agent’s behavior may cause reduced productivity in human-agent collaborations, or mistrust in automated RL systems. RL agents are trained to optimize a long term cumulative reward, and in this work we formulate a novel problem on how to generate explanations on when an agent could have taken another action to optimize an alternative reward. More concretely, we aim at answering the question: What does an RL agent need to do differently to achieve an alternative target outcome? We introduce the concept of a counterfactual policy, as a policy trained to explain in which states a black box agent could have taken an alternative action to achieve another desired outcome. The usefulness of counterfactual policies is demonstrated in two experiments with different use-cases, and the results suggest that our solution can provide interpretable explanations.

The interpretability of deep neural networks has become a subject of great interest within the medical and healthcare domain. This attention stems from concerns regarding transparency, legal and ethical considerations, and the medical significance of predictions generated by these deep neural networks in clinical decision support systems. To address this matter, our study delves into the application of four well-established interpretability methods: Local Interpretable Model-agnostic Explanations (LIME), Shapley Additive exPlanations (SHAP), Gradient-weighted Class Activation Mapping (Grad-CAM), and Layer-wise Relevance Propagation (LRP). Leveraging the approach of transfer learning with a multi-label-multi-class chest radiography dataset, we aim to interpret predictions pertaining to specific pathology classes. Our analysis encompasses both single-label and multi-label predictions, providing a comprehensive and unbiased assessment through quantitative and qualitative investigations, which are compared against human expert annotation. Notably, Grad-CAM demonstrates the most favorable performance in quantitative evaluation, while the LIME heatmap score segmentation visualization exhibits the highest level of medical significance. Our research underscores both the outcomes and the challenges faced in the holistic approach adopted for assessing these interpretability methods and suggests that a multimodal-based approach, incorporating diverse sources of information beyond chest radiography images, could offer additional insights for enhancing interpretability in the medical domain.

Although Machine Learning can be seen as a promising tool to improve clinical decision-making, it remains limited by access to healthcare data. Healthcare data is sensitive, requiring strict privacy practices, and typically stored in data silos, making traditional Machine Learning challenging. Federated Learning can counteract those limitations by training Machine Learning models over data silos while keeping the sensitive data localized. This study proposes a Federated Learning workflow for Intensive Care Unit mortality prediction. Hereby, the applicability of Federated Learning as an alternative to Centralized Machine Learning and Local Machine Learning is investigated by introducing Federated Learning to the binary classification problem of predicting Intensive Care Unit mortality. We extract multivariate time series data from the MIMIC-III database (lab values and vital signs), and benchmark the predictive performance of four deep sequential classifiers (FRNN, LSTM, GRU, and 1DCNN) varying the patient history window lengths (8h, 16h, 24h, and 48h) and the number of Federated Learning clients (2, 4, and 8). The experiments demonstrate that both Centralized Machine Learning and Federated Learning are comparable in terms of AUPRC and F1-score. Furthermore, the federated approach shows superior performance over Local Machine Learning. Thus, Federated Learning can be seen as a valid and privacy-preserving alternative to Centralized Machine Learning for classifying Intensive Care Unit mortality when the sharing of sensitive patient data between hospitals is not possible.

In reinforcement learning, an agent makes decisions to maximize rewards in an environment. Rewards are an integral part of the reinforcement learning as they guide the agent towards its learning objective. However, having consistent rewards can be infeasible in certain scenarios, due to either cost, the nature of the problem or other constraints. In this paper, we investigate the problem of delayed, aggregated, and anonymous rewards. We propose and analyze two strategies for conducting policy evaluation under cumulative periodic rewards, and study them by making use of simulation environments. Our findings indicate that both strategies can achieve similar sample efficiency as when we have consistent rewards.

Visualization is a key point in data exploration. In this paper we have emphasis in adding dynamic features by constructing exploration animations. We use Principal Component Analysis (PCA) in dimensionality reduction and K-means clustering algorithm in defining states. In predicting state transitions, we use Hidden Markov Model (HMM). Analyzed physical data is got from self-healing autonomous data centers. Our research methodology is to animate state transitions for data exploration in modern computerized environment. We use Jupyter tool and Python 3 programming language in our experimental realization. As results we get PCA animations for exploration purposes. Our approach is based on state discovery, where it is possible to find some physical interpretations for the defined states and state transitions. State structure and behaviour depend strongly on analyzed data.

Calibration of simulation models and hyperparameter optimisation of machine learning and deep learning methods are computationally demanding optimisation problems, for which many state-of-the-art optimisation methods are adopted and applied in various studies. However, their performances come to a test when the parameter optimisation problems exhibit high-dimensional spaces and expensive evaluation of models’ or methods’ settings. Population-based (evolutionary) methods work well for the former but not suitable for expensive evaluation functions. On the opposite, Bayesian optimisation eliminates the necessity of frequent simulations to find the global optima. However, the computational demand rises significantly as the number of parameters increases. Bayesian optimisation with random forests has overcome issues of its state-of-the-art counterparts. Still, due to the non-parametric output, it fails to utilise the capabilities of available acquisition functions. We propose a semi-parametric approach to overcome such limitations to random forests by identifying a mixture of parametric components in their outcomes. The proposed approach is evaluated empirically on four optimisation benchmark functions with varying dimensionality, confirming the improvement in guiding the search process. Finally, in terms of running time, it scales linearly with respect to the dimensionality of the search space.

Activity-based transportation models simulate demand and supply as a complex system and therefore large set of parameters need to be adjusted. One such model is Preday activity-based model that requires adjusting a large set of parameters for its calibration on new urban environments. Hence, the calibration process is time demanding, and due to costly simulations, various optimisation methods with dimensionality reduction and stochastic approximation are adopted. This study adopts Bayesian Optimisation for Likelihood-free Inference (BOLFI) method for calibrating the Preday activity-based model to a new urban area. Unlike the traditional variant of the method that uses Gaussian Process as a surrogate model for approximating the likelihood function through modelling discrepancy, we apply a composite surrogate model that encompasses Random Forest surrogate model for modelling the discrepancy and Gaussian Mixture Model for estimating the its density. The results show that the proposed method benefits the extension and improves the general applicability to high-dimensional settings without losing the efficiency of the Bayesian Optimisation in sampling new samples towards the global optima.