A Prescriptive Learning Analytics Framework: Beyond Predictive Modelling and onto Explainable AI with Prescriptive Analytics

With the increased embedding of complex predictive models into contexts that were previously dominated by human decision-making, the need has arisen for predictive models to display a greater degree of transparency behind their mechanisms of reasoning. This is the case not least for establishing trust in them, but also from the perspective of compliance. It is increasingly becoming a legal requirement across international jurisdictions²²2GDPR [30] is one example. that the mechanisms behind any automated decisions affecting humans be clearly explained to those affected by them. Wachter et al. [40] also point out the necessity of being able to contest decisions made by automated systems by those concerned and to also have the opportunity to be informed with respect to the current decision-making model as to what it would need to see change in their data inputs to produce an alternative or a more desirable decision.

It is for these reasons that a relatively new research field of eXplainable Artificial Intelligence (XAI) (sometimes referred to as Interpretable Machine Learning) has emerged. Some of the key aspirations of this field are focused on developing techniques that address the high-level interpretability of opaque predictive models as well as on devising tools that enable them to interrogate and extract explanations of how the models arrive at given conclusions [24]. Meanwhile, there currently exists a gathering the research interest exploring prescriptive analytics where methods are being developed that leverage data-driven approaches to generate evidence-based actionable insights [20].

From a technical point of view, model interpretability tends to pertain to the tasks of making sense of a model’s internals generated post-training by a machine learning algorithm. It usually concerns a level of clarity into the mechanics of a model at a global-level, while explainability of models refers to extracting the reasoning behind the model’s prediction for a specific learner in this context, which is referred to as local-level explainability. Both perspectives on model behaviour are important and serve multiple overlapping purposes. Model interpretability affords an institution the ability to communicate to all concerned stakeholders how a predictive model works in general terms using broad brush strokes. While local-level explainability of models enables the validation of specific predictions to take place by student support teams before interventions are initiated, it also enables clear responses to be given to affected students as to exactly how and why they have been identified as being at-risk. However, clues can also be gleaned from the local-level explanations into possible remedial actions that can be suggested to the learners.

Tools and techniques supporting the goals of model transparency are reaching maturity. SHAP [22] is a visualisation technique that is currently recognised as being state-of-the-art in the field of XAI [15] for realising both global and local-level model transparency. The Anchors [32] technique has also been developed recently as a tool that imparts a high degree of local-level explainability. The benefit of Anchors is that it produces human-readable rule-based models which are succinct approximations of the behaviour of the underlying complex models. However, a more advanced approach using Counterfactuals [40] enables the analytics processes to surpass predictive capabilities, and enter into the prescriptive. This technique provides precise suggestions to the learners regarding the smallest set of adjustments they need to make in their learning behaviour for an alternative prediction to be generated. This study demonstrates the applicability of all three of these technologies for different steps of the proposed prescriptive analytics framework.

This study presents a novel prescriptive analytics framework for supporting LA aims of identifying and initiating both timely and effective interventions with at-risk students. The proposed framework demonstrates how both predictive and prescriptive analytics can be more fully leveraged than has previously been done. This study illustrates how effective models predicting qualification completion outcomes can be developed using machine learning and made transparent at both global and local levels to meet the needs of all stakeholders. Furthermore, this work goes on to illustrate through case study examples how prescriptive analytics tools can subsequently be utilised to automatically generate specific prescriptive feedback, that is evidence-based as well as actionable, and how the output of these methods can be converted into human-readable text.

An in-depth SLR was conducted by Namoun and Alshanqiti [26], covering a total of 62 relevant studies within predictive LA, in which they focused on three areas: (1) ways in which academic performance was measured using learning outcomes and subsequently predicted, (2) the types of algorithms used to forecast student learning outcomes, and (3) which features are most impactful for predicting student outcomes. The authors found that 90% of the studies predicted course-level outcomes, while only three studies considered predicting programme/qualification-level outcomes. As part of their findings, they urged the research community to conduct further work in predictive modelling, and more specifically, to do so at the programme level which they described presently to be in its infancy. Meanwhile, the authors called researchers to rise above simply predicting outcomes, and to also incorporate model interpretability and explainability into their studies.

Albreiki et al. [2] likewise conducted an SLR consisting of 78 studies of relevant EDM literature from 2009 to 2021 concerning predicting at-risk learners of non-completion. The review indicated that only a handful of studies proposed means for generating remedial solutions consisting of feedback that learners and educators can use to address the underlying obstacles. The authors noted that future research will place greater emphasis on devising machine learning methods to predict students’ performance in general and will also augment this with automatically generated remedial actions to assist learners as early as possible.

Another up-to-date survey by Xiao et al. [43] examined almost 80 studies using EDM to predict students’ performance and provided insights in line with previous studies. The authors note that one of the deficiencies in this field is that very few studies have attended to explore model interpretability and have thus neglected explanations of the mechanics of predictions and the role that different features play in the predicted outputs. Indeed, the authors stress that the use of model interpretability tools ought to be one of the chief pursuits and the direction of future research in this field of study.

There were several other recently conducted reviews into LA and EDM studies. Interestingly, these did not address the issue of model interpretability and explainability, nor was there coverage of the use of prescriptive analytics tools indicating that these approaches are not yet in wide use. Fahd et al. [13] carried out a broad meta-analysis of literature of 89 studies from 2010 to 2020. Their survey analysed the application of machine learning approaches in predicting student academic performance. Their primary focus was on considering what types of models were being used in research at HEI, and their identification of trends centred around specific matters of how to achieve better predictive accuracies using machine learning. Similarly, Batool et al. [4] conducted a LA survey of some 260 studies over the last 20 years on the topic of student outcome prediction. Similar to previous reviews, they focused on highlighting the most effective features for this task and the types of algorithms and techniques used, as well as data mining tools that are most frequently applied. However, the important issues of model interpretability and prescriptive analytics were not raised.

Likewise, Shafiq et al. [34] conducted an SLR covering 100 papers. Their focus was again on predictive analytics, looking mostly at what was being predicted, what kind of data was used, which sets of features were effective or otherwise, and what types of algorithms were explored. The focus was essentially on ways to enhance the accuracy of predictive models but there was no coverage of techniques that extend beyond mere predictions Instead, the recommendations emphasised that more ensemble-based and clustering methods should be explored to predict the performance of students and enhance the prediction accuracy. In the same vein, Tjandra et al. [37] conducted a comprehensive review of recent studies based on student performance prediction. The review considered features and algorithms used, accuracies attained, as well as commonly used tools. The authors concluded that there is still limited use of personal characteristics data such as psychological and social/behavioural features for developing student performance predictions and that future research should focus on including these to address dropout rates. Finally, Hernández-de Menéndez et al. [17] investigated the practices of 16 HEIs that have deployed LA projects. The authors found that they have mostly used LA technologies for student retention. These tools largely supported strategies for identifying at-risk students through predictive analytics, which served as a springboard for initiating various types of interventions.

A vast majority of the predictive models identifying at-risk students have focused on course-level outcomes rather than on programme-level completions [26]. Since course-level predictive models have tended to only be deployed across subsets of all courses on offer, the likelihood of at-risk students evading detection is therefore high. Thus, there is high utility in pursuing programme-level completion predictions. Though prior works in the prediction of programme completions are very rare, those that exist have generally followed an outcomes-based approach, which breaks down all constituent parts of programme-level outcome requirements into course outcomes first, and subsequently proceeds to map them all to programmes. These types of bottom-up predictive models then operate on the level of course-level outcomes and these predictions are combined to generate programme-level outcome predictions. Examples of this approach are [9], as well as [5] and [6] who likewise developed programme-completion prediction models and claimed to achieve accuracies in the upper ranges between 90% and 95% using various techniques from the WEKA data mining toolkit. More recently, Gupta and Ghosal [16] followed a similar approach in first determining course-level outcome predictions and mapping these to programmes, before determining overall outcomes. Such systems embody a great deal of complexity and appear to have been mostly research prototypes since productionisation and long-term maintenance of these approaches in a dynamic setting are questionable with respect to sustainability.

Model interpretability is not only important for building trust, meeting compliance and extracting maximal value from predictive systems, but it is also vital for developing accurate predictive models. Feature engineering and selection tasks are more critical to the success of machine learning models than the choice of algorithm or the size of the datasets used [11]. While a wide range of features for predicting student performance have been used in literature, there is still a lack of clarity on which specific features are most effective and how they interact to influence the attainment of course and programme outcomes [26]. Therefore, tools and approaches that specifically illuminate the influence and the behaviour of the models in terms of the underlying features, are important. In one of the earliest studies into methods of illuminating the predictions of black-box models for learners, Villagrá-Arnedo et al. [39] developed a set of proprietary graphical tools to exploit the output information and provide a meaningful guide to both learners and instructors. Dass et al. [10] develop dropout prediction models for students in a MOOC course and did strive to gain a deeper understanding of the effectiveness of the various features to support accurate predictions, as well as to predict the point in time when the students were likely to drop out. Similarly, Jang et al. [18] demonstrate the usage of XAI techniques to assist in interpreting the classification results of the models which were designed to identify at-risk students. The study paid particular attention to selecting features that were relevant to stakeholders.

Jenhani et al. [19] developed one of the earliest automated remedial prescriptive systems which leveraged machine learning. The system was designed in such a way that a separate classification model was trained on a remedial action dataset describing historical data based on experts’ and instructors’ actions to improve the low learning outcomes. The types of predictive outputs that the system made as suggestions for at-risk students were: revise a concept, attempt extra quizzes, solve specific practice examples, take extra assignments, etc. All these prescriptions were at a generic level. Elhassan et al. [12] extended this work further to enable the system to recommend a set of remedial actions to address specific shortcomings rather than a single action as previously.

Albreiki et al. [1] developed predictive models for at-risk students which they combined with a customised system that enabled the instructor to set various thresholds and weights alongside the predictive models. Based on the configuration and the outputs of the predictive models, the system would then select from a predefined list of remedial actions the most suitable one.

Most recently, Susnjak et al. [36] proposed a student-oriented LA dashboard which apart from using descriptive and predictive analytics, also demonstrated a prototype of how to integrate interpretability and explainability aspects of the predictive models, as well as the automatic generation of feedback based on prescriptive analytics.

From this literature review, several conclusions can be drawn. The literature indicates that the use of predictive analytics within the LA field is widespread and on the rise. The literature points to an acute gap in the current research concerning the use of predictive analytics tools which enable interpretability and explainability of the models. The bulk of the research is still largely focused on devising means of more effectively predicting various outcomes. There is a general absence of XAI tools in usage. These tools have only just started to emerge in LA research even though these tools are not esoteric. Much less is there any evidence that there exists the use of data-driven prescriptive analytics which provide automated and tangible remedial advice to at-risk students. Thus the conversion of predictions into actionable insights is broadly missing from the literature at this point.

There is therefore a need and an emerging requirement to begin to implement predictive analytics systems with responsible and accountable characteristics. These LA systems ought to have embedded transparency that is enabled by an expressive set of technologies, while venturing beyond the predictive realm and ideally being augmented by prescriptive tools that assist learners in addressing their challenges. Unless an effective predictive system can be matched with the reasoning behind its decisions surrounding the underlying variables, and suggestions about concrete pathways for moving forward, these systems will only ever have constrained capabilities and limited uptake.

To that end, this work proposes a prescriptive learning analytics framework that attempts to address the outlined gaps. This work defines a step-by-step process for building predictive models and highlights ways for enhancing them with various levels of interpretability, while ultimately showing how prescriptive analytics can complement the entire process. This work demonstrates the proposed framework using programme/qualification completion predictive models and maps each step of the framework with specific technologies while showing the practicality of the framework with use case scenarios.

SHAP (SHapley Additive exPlanation) which is based on Shapley values [35] drawn from game theory literature, was used to generate global interpretability defined in Step (4), as well as local model explainability required for Step (5), addressing both the "how" and "why" questions around model behaviour respectively. In addition, the Anchors tool was also used in Step (5) alongside SHAP. Both techniques approach the generation of new surrogate models approximating the behaviour of the original "black-box" models differently and offer complementary insights. The advantage of SHAP lies in its detailed quantification of effects that each feature and their values exert on the final prediction, while Anchors reduces a black-box model into a human-readable set of predicates resembling a degenerate decision tree.

For the prescriptive analytics phase, the Diverse Counterfactual Explanations (DiCE) [25] technique was used in Step (6) to simulate what-if scenarios and generate candidate prescriptive feedback suggestions. DiCE can generate counterfactuals for many machine learning models, approaching the task as an optimisation problem. However, counterfactual modelling is a challenge and needs to be configured in such a way that, firstly, the features used are relevant for a learner and represent learning behaviours that can be adjusted, and secondly, the suggested action must be feasible and practical [27]. To that end, a new underlying machine learning model was generated using an alternative set of features comprising predominately mutable features which would support the development of actionable feedback. These features can be seen in Table 1 under the Prescriptive Modelling column. However, the counterfactual modelling was further configured to exclusively rely on features upon which prescriptive advice was to be based. Table 1 therefore shows features that were used for creating a new prescriptive model as denoted by $◯$, and features that were used for both modelling and for generating counterfactuals $\oplus$. Secondly, the features used for counterfactual modelling have been constrained to fall within feasible and realistic ranges. Finally, once the candidate counterfactuals were generated, these were then converted into human-readable text as indicated in Step (7).

The quantitative component of the framework concerning predictive analytics is evaluated through empirical experiments that demonstrate the accuracy of the generated models, which also validate the efficacy of the engineered features (Steps 2 - 3).

The remaining parts of the framework are qualitative. Steps 4 and 5 are evaluated by assessing the suitability of the tools to visually convey the mechanics of the underlying model and to establish its reasonableness. A case study using two hypothetical students is used to evaluate in detail the model behaviour in explaining its predictions for Step 5. The same case study examples are then carried through to the prescriptive analytics component (Steps 6 - 7), where several prescriptive feedback suggestions are generated for each student, and subsequently converted into human-readable text for assessment of their feasibility.

Table 3 summarises the generalisation performances of all candidate models across several evaluation measures. The results indicate that overall, the models have achieved a high level of efficacy for predicting whether a learner would eventually complete or abandon their studies. The preferred F1-measure on the unbalanced dataset indicates that the values range from  92% to  95% and represent a significant improvement over the baseline models. The table highlights the importance of reporting the accuracies of baseline models so that the genuine value of the predictive models with respect to various forms of random guessing can be quantified. On average, the best-performing algorithm for this dataset is CatBoost, which is slightly better than Random Forest. Some of the performance advantages of CB can arguably be attributed to its inherent ability to handle both missing values and categorical data directly in contrast to the other algorithms in these experiments.

Having determined the efficacy of the model to identify students at risk of programme non-completion, the global-level interpretability of the best-performing CatBoost model is next examined. Figure 2 depicts SHAP’s perspective of the model’s dynamics. Two key components are shown. The first lists features in their order of importance, from highest to lowest in terms of the impact they exert on the eventual prediction. It can be seen that the learners’ current full-time status, their prior activity with respect to the current academic year, the mean grade mark, together with the number of failed papers are the most impactful features.

The second component in the figure offers an additional dimension concerning the interpretability of the model. Here we observe how an increase or decrease in feature values affects the final prediction. The colour gradients represent increasing (red) and decreasing (blue) feature values, while grey represents categorical values. The x-axis depicts SHAP values. Data points with a positive SHAP value (appearing to the right of the vertical zero line) have a positive impact on the predictions, in other words, they contribute towards driving the prediction towards programme completion predictions. Conversely, the points with a negative SHAP value (to the left of the vertical zero line) influence the prediction towards programme non-completion. The extent of the points from the vertical line signifies the magnitude of the effect that they contribute to the final prediction.

In the figure, it can be observed that the full-time study status (value 1) has a positive effect on completion predictions, while part-time (value 0), has the opposite. Learners’ mean grade has an unsurprising effect; however, a nuanced interpretation can be extracted from the plot where one can deduce that lower grade averages have more of a negative predictive outcome than high grade-averages have on positive outcomes. A similar interpretation can be made regarding the number of failed papers. A large number of failed papers has more of a negative predictive effect than having no failed papers has on positive predictive outcomes.

In general, elevated assignment scores and submission counts are predictive of positive outcomes. Learner engagement with the VLE conveys a more mixed picture. VLE pages viewed and the number of online quizzes taken have a positive effect; however, forum post creation counts are ambivalent, while the reverse holds for the number of forum posts read. As the number of qualification credits increases, the effect is stronger for negative outcomes, while the reverse holds for the learner’s age. Possessing previous studies (value 1) indeed has a more positive effect than otherwise (value 0), while the on-campus study mode (value 1) is associated with more negative qualification outcomes than studying online (value 0).

By considering both the feature importance ranks and how the feature values affect the final prediction, it is possible to validate the model against an expected behaviour and communicate its mechanics in a simplified form to all stakeholders. The behaviour of the examined model confirms that it is reasonable and thus valid.

Having achieved interpretation and validation of the predictive model, the next step is to examine the model behaviour at an individual (or local) prediction level. SHAP as well as Anchors⁶⁶6An alternative technology for this is LIME which has some additional advantages. are used in this step. To demonstrate this, two hypothetical students are used - Student A and Student B. Both students have been predicted by the CatBoost model with non-completion outcomes, with the probability of 97% and 90% respectively.

Figure (a)a(a) shows the top nine features and their values for Student A on the y-axis, rank-ordered by influence on the final prediction. Informally, the figure can be viewed and interpreted as a tug-of-war. The mid-line represented with the value of 1.823 is the expected or the average SHAP value of all the predicted data points. A final SHAP value to the left of this line represents a non-completion prediction and alternatively, the values on the right side denote positive outcome predictions. Blue bars represent the forcing effects towards negative predictions, while red the opposite. The size of the bars represents the magnitude of the forcing of the corresponding features and their values. These graphs are best interpreted from the bottom up. The topmost feature represents the final SHAP value that includes all feature contributions.

Figure (a)a(a) shows that the least significant 11 features collectively have an influence tending towards negative outcomes to the left of the 1.823 value mid-line. This increases with each of the features except for the student’s learning behaviour for on-time assignment submissions, which has a positive effect.

Model reasoning is depicted in Figure (a)a(b) for Student B. Similar patterns are observed with the exception that this student is a returning learner (prior activity is ’university student’) and that the student’s mean grade is 66%, which have a strong positive influence on the final prediction. However, in totality, the majority of the features are forcing the prediction towards negative outcomes, and this is where they ultimately settle. The utility of the SHAP tool to visually explain its reasoning to academic advisors pre-intervention as well as to other relevant stakeholders is demonstrable through these examples.

While SHAP provides a detailed visual perspective into the model mechanics, some cccasions will require that a more succinct and simplified explanation of a prediction is communicated. This can be achieved using Anchors. Figures 4(a) and 4(b) show how the complex predictive model can be reduced to the most essential and explained in a more simplified manner for both Student A and B respectively. The figures show that the prediction model has been re-cast as a rule-based decision tree consisting of only three conditions, which result in a non-completion prediction if they all hold true. Student A and B’s actual values for the three features are shown in the first column with the conditional statements and their thresholds for each feature shown in the second column.

In both cases, the Anchor surrogate model has identified and used the number of papers failed as one of the reasons for a negative prediction. For Student A, the fact that the student has a low average grade of 53% while undertaking a commitment to a programme that is higher than the standard bachelor’s qualification (360 credits), has been used as further conditions for the classification as non-completion. In the case of Student B, the fact that the student has already withdrawn from papers in the current academic year and is presently a part-time student, has created conditions for a non-completion predicted outcome. These forms of model simplifications serve as effective and suitable tools for communicating to affected learners how exactly they have come to be identified as being at-risk, thus meeting the requirements of transparency and responsible use of predictive analytics.

The generation of surrogate models as demonstrated above already hints at possible prescriptive suggestions which can be constructed from them. Indeed, there is a potential to do this, however, a more data-driven and robust method ought to be pursued which considers the interaction of the features and their effects in a more principled approach. This is where prescriptive analytics tools make their contribution.

Counterfactual modelling is used in this step to generate a set of possible pathways for a specific learner that would lead them to a positive outcome prediction. In more precise terms, here we are looking at several possible sets of minimal adjustments to selected feature values which would result in an alternate outcome for a student who is predicted to be on track for non-completion. This type of what-if modelling is demonstrated for Student A and B in Figure (a)a.

Figure (a)a depicts both the most concise set of features and the smallest required adjustments in their values which would be needed for the selected students to toggle their predicted outcome to a successful completion. The first column lists the selected features by the counterfactual model, and the next column shows the actual values for each of the hypothetical students, followed by three sets of counterfactuals from which automated and data-driven Prescriptive Feedback (PF) advice can be generated. In each PF set, only three feature values have been varied. The dash represents no required changes to the actual values.

In Figure (a)a(a), it can be seen that modest increases to the grade average have been identified as a pathway to completion for Student A, together with a switch to a smaller programme of study, as well as a mixture of adjustments to the online learning engagement behaviours. In the case of Student B, Figure (a)a(b) shows multiple pathways towards completion if the study mode is changed to full-time as well as to online mode on another occasion. Pathways exist through some modifications in online learning behaviours, while no adjustment is needed to be made for the grade average of 66%. Interestingly, the qualification percent completion feature has been identified in the case of this student as being potentially helpful. In all three PFs, it is observable that if the student succeeds in completing 8.2% of their programme, up from the current 4.1%, then this also suggests that positive outcomes become more likely.

From this, it becomes immediately apparent that z-score values and percentages of completion carry with them very little meaning and actionable potential on behalf of the target learners as well as for the academic advisors. It is for this reason, that the engineered features need to be converted back into original raw values which will then make them practical.

The final step in the proposed framework performs two types of conversions. The first converts the engineered features chosen by the counterfactuals into raw values, while the second converts the candidate PFs into a natural language form which can then be dispensed to students. The example results of both types of conversions of the counterfactuals into final text-based PFs are shown in Figure (a)a for Student A and B.

Figure (a)a shows that the proposed format of the final PFs to be used for remedial interventions comprises two parts. The first informs learners of their current status concerning the particular features that are to be used for PFs. This provides a context for the second part, which consists of sets of possible feedback suggestions that advisors can provide to the students. The figure shows that both the z-scores and the programme completion percentages have been converted into meaningful values which are now actionable and measurable. This underscores the dependence between Step 7 and Step 1 in the proposed framework for the conversion of engineered values to the raw values. The conversion of the counterfactuals into text-based PFs can either be automated through text-based templates or using Natural Language processing techniques. Once converted, interventions can be initiated as outlined in Step (8).