Student Retention Early Warning

A distance-learning provider reviewed student outcomes after each presentation ended. The review was thorough and completely useless for the students in it: by the time results were finalised, the people who withdrew had already gone. Tutors often knew which students were struggling, but that knowledge arrived too late, lived in individual inboxes and was not connected to any support capacity.

The retention decision

Each week, the student support team had to decide whom to contact, through which channel and with what offer — a call, an extra tutorial, a deadline extension or a referral to student services. The useful output was not a risk score. It was a short, explained worklist sized to tutor capacity, produced early enough for the intervention to change the outcome.

What we worked with

• Student registrations: 32,593 student-module records across seven modules and four presentations, with demographics, prior attempts, credits and disability status.
• Virtual learning environment: 10,655,280 daily click records across 6,364 resources, linked to module, site and day of study.
• Assessments: 173,912 student-assessment records and 206 assessment definitions, including due dates, weights and scores where submitted.
• Outcome: final result — pass, distinction, fail or withdrawal — recorded at the end of the presentation.

The data described several different things at once: how a module was designed, when assessments fell due and how an individual behaved. Separating those effects was the first analytical problem.

The traps we designed around

• Outcome leakage. Final results and late assessment submissions are recorded after the fact. A model that uses them describes the past perfectly and cannot support a decision in week four.
• Withdrawal is not one event. Some students withdraw before the course starts; others stop engaging and are later marked withdrawn. Both end in the same column, but only the second offers a useful intervention window.
• Clicks are not engagement. Click volume depends on module design, resource structure and assessment timing. A different module can produce twice the clicks with no difference in learning. Activity patterns and consistency carried more signal than raw counts.
• Demographics can become destiny. Region, education level and deprivation band predict outcomes in the data, partly because they proxy for circumstances the institution cannot change. Using them to rank students risks penalising people for who they are. They were used for fairness monitoring, not for targeting.
• Capacity and contact fatigue. A list of 3,000 at-risk students is not an intervention. Thresholds were set against the number of students the tutors could genuinely support each week.

How the early warning was built

1. Fix the cutoff. We agreed a weekly decision point and a target outcome: withdrawal or failure before the next assessment cycle. Only data available before that point was allowed into the features.
2. Audit the label. Pre-start withdrawals were separated from mid-course disengagement, and module presentations were analysed separately where their structures differed.
3. Engineer engagement features. Distinct active days, recency of activity, click mix by resource type, before-deadline versus after-deadline behaviour and assessment submission history replaced raw click totals.
4. Compare a simple and a flexible model. A regularised logistic model provided interpretable coefficients and a sanity check; a tree-based model captured interactions. Both were calibrated and evaluated on a later presentation, not a random row split.
5. Evaluate at tutor capacity. The operating point was chosen by how many at-risk students could be contacted each week, using recall within that list, lift and calibration. Accuracy was irrelevant with withdrawal and failure rates in the data.
6. Check fairness explicitly. Recall and false-negative rates were compared across demographic groups. Where base rates differed, thresholds were reviewed per module and the trade-off documented rather than hidden.
7. Design the intervention and the test. Tutors received a script, a small menu of support actions and a way to record what happened. A holdout group made it possible to measure whether contact changed outcomes rather than merely identifying students who would have struggled anyway.

What this changes

• Support starts during the course. Tutors work a weekly list while there is still time to act, instead of reviewing outcomes after they are fixed.
• Contact is explained. Each entry shows the behaviour behind the flag — for example, a drop in active days or a missed assessment — so the conversation starts with evidence.
• Capacity is explicit. The team knows how many students it can support and what happens to the rest; the list is not allowed to grow beyond that number.
• Fairness is monitored. Group-level recall, false negatives and intervention uptake are reviewed as part of the system, not as a one-off audit.

What we delivered

• A calibrated early-warning score with module-level reason codes and a weekly worklist sized to tutor capacity.
• A documented label and leakage audit covering pre-start withdrawals and late submissions.
• A fairness monitoring view across demographic groups and module presentations.
• A tutor playbook with support options, contact scripts and a record of actions taken.
• A holdout evaluation plan and a monitoring dashboard for drift between presentations.

Where this applies

This fits universities, colleges, online learning providers and professional training programmes with a learning management system and reliable outcome records. The minimum inputs are a stable student-module identifier, a defined withdrawal or failure outcome, engagement logs and a support team that can act weekly. The first deliverable is usually the label audit and the capacity agreement — because an early-warning system that identifies more students than anyone can help is simply a more efficient way to feel concerned.

This is an illustrative case study built on an anonymized dataset; it does not report results from a specific client engagement. Student data is sensitive by default: minimum necessary fields, role-based access, retention limits and fairness review were treated as part of the method.