Drug-drug interactions (DDIs) can lead to serious adverse effects and compromised treatment efficacy. As artificial intelligence models gain traction in DDI prediction, the interpretability of these models becomes crucial. Despite advancements in predictive performance, most DDI models lack adequate explainability, limiting their clinical utility. This study aims to enhance the interpretability of a deep learning DDI prediction model by applying a post-hoc Explainable Artificial Intelligence (XAI) method and evaluating the model’s reasoning in the context of domain knowledge. The analysis focused on a publicly available DDI model based on autoencoders and a deep neural network. Kernel SHAP was applied to the model using the original input space, which included feature vectors derived from structural, target gene, and gene ontology similarity matrices. DrugBank served as the reference for validating pharmacological relevance. Explanations were evaluated based on domain knowledge and visualized using SHAP tools. Several features highlighted by Kernel SHAP appeared to align with the underlying mechanisms of DDIs. Structural similarity features appeared frequently among the top contributors to the prediction. Often, high importance was assigned to identity-like structural similarity features, which raises concerns about potential shortcut learning. Target gene and gene ontology features were less consistent, sometimes reflecting mechanisms of interaction and other times appearing misleading due to limitations in similarity calculations. The study demonstrates that XAI methods, when supported by domain-informed interpretation, can uncover valuable insights into a model’s internal reasoning. It also raises important concerns regarding feature design and the necessity for more rigorous validation.

Several interpretability methods for convolutional network-based classifiers exist. Most of these methods focus on extracting saliency maps for a given sample, providing a local explanation that highlights the main regions for the classification. However, some of these methods lack detailed explanations in the input space due to upscaling issues or may require random perturbations to extract the explanations. We propose Convolutional Rectifier for Interpretable Time Series Classification, or CRITS, as an interpretable model for time series classification that is designed to intrinsically extract local explanations. The proposed method uses a layer of convolutional kernels, a max-pooling layer and a fully-connected rectifier network (a network with only rectified linear unit activations). The rectified linear unit activation allows the extraction of the feature weights for the given sample, eliminating the need to calculate gradients, use random perturbations and the upscale of the saliency maps to the initial input space. We evaluate CRITS on a set of datasets, and study its classification performance and its explanation alignment, sensitivity and understandability.

Sepsis mortality risk prediction has been explored in the machine learning (ML) community as a use case that could benefit from data-driven decision support systems in intensive care units (ICU). However, most of the work focuses on training models to reach a high classification performance, and there are limited efforts on technical artifacts that explore the practical utility of such predictive models. Therefore, we present the tool SepsisVision, an example of a web-based explainable user interface (XUI) to support sepsis mortality risk through static explanatory and interactive exploratory interfaces to navigate the behavior of an ML model through different kinds of explanations: SHAP-based local and global feature attributions, distribution comparison, and counterfactual analysis. Tool available at: https://sepsisvision.streamlit.app/.

Depression remains a prevalent condition in primary care due to its heterogeneous symptoms. To support early screening, this study presents DepreScan, an interactive web-based clinical decision support system powered by machine learning and explainable artificial intelligence. Trained on a representative health survey dataset, DepreScan uses interpretable models and multiple explanation techniques—including SHAP plots and simplified decision trees—to assist healthcare practitioners in screening depression risk. A mixed-methods user study with 16 clinicians assessed the system’s trustworthiness, usability, and perceived clinical utility. Results indicate moderate to high acceptance and trust, particularly for SHAP-based global feature explanations, with differences in measured trust between two related questionnaires. The study highlights the importance of aligning ML explainability with healthcare professionals’ mental models and the findings inform the design of user-centered XAI tools for mental health decision support in primary care.

CounterFair is a group counterfactual search algorithm that detects and minimizes biases among sensitive groups and identifies relevant subgroups inside these sensitive groups based on shared counterfactual instances. We investigate the latter capability, analyzing the found subgroups from the perspective of fairness based on counterfactual reasoning, in order to evaluate whether they present different biases with respect to each other and to the sensitive feature groups they belong to. We perform these measurements on the subgroups extracted by CounterFair over six binary classification datasets, providing figures and their respective analysis on the presence of bias.

Counterfactual explanations can be used as a means to explain a models decision process and to provide recommendations to users on how to improve their current status. The difficulty to apply these counterfactual recommendations from the users perspective, also known as burden, may be used to assess the models algorithmic fairness and to provide fair recommendations among different sensitive feature groups. We propose a novel model-agnostic, mathematical programming-based, group counterfactual algorithm that can: (1) detect biases via group counterfactual burden, (2) produce fair recommendations among sensitive groups and (3) identify relevant subgroups of instances through shared counterfactuals. We analyze these capabilities from the perspective of recourse fairness, and empirically compare our proposed method with the state-of-the-art algorithms for group counterfactual generation in order to assess the bias identification and the capabilities in group counterfactual effectiveness and burden minimization.

Counterfactual explanations modify the feature values of an instance in order to alter its prediction from an undesired to a desired label. As such, they are highly useful for providing trustworthy interpretations of decision-making in domains where complex and opaque machine learning algorithms are utilized. To guarantee their quality and promote user trust, they need to satisfy the faithfulness desideratum, when supported by the data distribution. We hereby propose a counterfactual generation algorithm for mixed-feature spaces that prioritizes faithfulness through k-justification, a novel counterfactual property introduced in this paper. The proposed algorithm employs a graph representation of the search space and provides counterfactuals by solving an integer program. In addition, the algorithm is classifier-agnostic and is not dependent on the order in which the feature space is explored. In our empirical evaluation, we demonstrate that it guarantees k-justification while showing comparable performance to state-of-the-art methods in feasibility, sparsity, and proximity.

Healthcare faces significant global challenges due to an aging population and the surge of chronic health conditions. Artificial Intelligence (AI) has emerged as a promising tool to address these issues, but the lack of transparency hampers clinicians’ trust. Explainable AI (XAI) is a method to explain the AI’s logic behind its predictions, thereby improving transparency. However, very little research has been done on XAI from the clinicians’ perspective. Our research aims to understand how clinicians want the XAI explanations of medical AI predictions to be presented. A total of 30 physicians from 9 medical specialties evaluated two of the most used XAI methods on AI predictions on patients: Local Interpretable Model-agnostic Explanations (LIME) and Diverse Counterfactual Explanations (DiCE). A mixed method approach was used with web-based questionnaires, and the results were analyzed with statistical and thematic methods. We found a significant disparity: the explanations generated by XAI methods often fail to align with physicians’ expectations and requirements for reducing uncertainty and providing enough depth of explanation in AI predictions. Moreover, there is a paradox where the areas rated most important by the physicians performed the poorest, whereas XAI excels in areas not rated equally essential. The physicians also highlighted additional areas that need to be addressed, including general upskilling of clinical staff in AI and XAI, ensuring the AI and XAI tools are integrated into the normal healthcare processes, and the ability to personalize the XAI explainability presentations.

In the current state of AI development, it is reasonable to think that AI will continue to expand and be increasingly utilized across different fields, highly impacting every aspect of humanity's welfare and livelihood. However, different AI researchers and institutions agree that AI has the potential to be extremely beneficial but also may pose existential threats to humanity. It is therefore necessary to develop tools to open the so-called black-box AI algorithms and increase their understandability and trustworthiness, in order to avoid conceivably harmful future scenarios.

The lack of interpretability of AI is a challenge to its own development: it is an obstacle equivalent to those that triggered previous AI winters, such as hardware or technological constraints or public over-expectation. In other words, research in interpretability and model understanding, both from theoretical and pragmatic perspectives, will help avoid a third AI winter, which could be devastating for the current world economy.

Specifically, from the theoretical perspective, the subfields of local explainability and algorithmic fairness require some improvements in order to enhance the explanation output. Local explainability refers to the algorithms that attempt to extract useful explanations for the output of machine learning models for individual instances, while algorithmic fairness refers to the study of biases or fairness issues among different groups of people, whenever the datasets refer to humans. Providing a higher level of explanation accuracy, explanation fidelity and explanation support for the observations of each dataset would help improve the overall level of trustworthiness and the understandability of the explanations. The explainability methods should also be applied to practical scenarios. In the area of autonomous driving, for example, providing confidence intervals on the positioning estimates and positioning errors is important for vehicle operations, and machine learning models coupled with conformal prediction may provide a solution that focuses on the confidence of these estimates, prioritizing safety.  

This thesis contributes to research in the field of AI interpretability, focusing mainly on the algorithms related to local explainability, algorithmic fairness and conformal prediction. Specifically, the thesis targets the improvement of counterfactual and local surrogate explanation algorithms. These explainability methods may also reveal the existence of biases, and therefore the study of algorithmic fairness is a relevant part of interpretability. This thesis focuses on the topic of machine learning fairness assessment through the use of local explainability methods, proposing two novel elements: a single accuracy-based and counterfactual-based bias detection measure and a counterfactual generation method for groups intended for bias detection and fair recommendations across groups. Finally, the idea behind interpretability is to be able to eventually implement such methods in real-world applications. This thesis presents an application of the conformal prediction framework to a regression problem related to autonomous vehicle localization systems. In this application, the framework is able to output the predicted positioning error of a vehicle and its confidence interval with some level of significance.

In this paper, we use counterfactual explanations to offer a new perspective on fairness, that, besides accuracy, accounts also for the difficulty or burden to achieve fairness. We first gather a set of fairness-related datasets and implement a classifier to extract the set of false negative test instances to generate different counterfactual explanations on them. We subsequently calculate two measures: the false negative ratio of the set of test instances, and the distance (also called burden) from these instances to their corresponding counterfactuals, aggregated by sensitive feature groups. The first measure is an accuracy-based estimation of the classifier biases against sensitive groups, whilst the second is a counterfactual-based assessment of the difficulty each of these groups has of reaching their corresponding desired ground truth label. We promote the idea that a counterfactual and an accuracy-based fairness measure may assess fairness in a more holistic manner, whilst also providing interpretability. We then propose and evaluate, on these datasets, a measure called Normalized Accuracy Weighted Burden, which is more consistent than only its accuracy or its counterfactual components alone, considering both false negative ratios and counterfactual distance per sensitive feature. We believe this measure would be more adequate to assess classifier fairness and promote the design of better performing algorithms.