Auto Insurance Claims Fraud Screening

A claims file with an outcome flag looks like a ready-made training set. Every row has a label, the fraud rate is known and the fields are structured. The temptation is to fit a classifier and report its accuracy. The actual decision is narrower: which claims should a limited investigations team review next week, why were they chosen, and how many genuine cases will that queue contain?

The decision

The insurer needed a repeatable screening queue: claims ranked by expected loss, a reason code attached to each, a cut-off tied to actual review capacity, and a validation plan that would still mean something a year later. Accuracy was not the metric that mattered — the number of investigated claims that turn out to be fraudulent was.

What we worked with

17,998 claim records with 25 fields, dated January 2015 to September 2016:

• Claim identity: claim number and claim date.
• Claimant profile: driver age, gender, marital status, education indicator, annual income, address change indicator, living status and postal code.
• Claim context: accident site, day of week, prior claims, witness indicator, liability percentage, reporting channel and whether a police report was filed.
• Vehicle: age, category, price, colour and weight.
• Outcome: a fraud flag, present on 15.65% of records.

The traps we designed around

• 15.65% is a sample rate, not a live base rate. The file is a curated extract. The share of fraudulent claims arriving in next month's queue will be different, and any threshold has to be set against the queue the team actually sees.
• The witness signal points the wrong way. A witness was recorded on 16.0% of fraudulent claims but 24.6% of non-fraudulent ones. The intuition that a witness is a fraud signal is wrong here; a one-variable rule built on it would screen in the wrong direction. The relationship may reverse within particular accident sites, but that has to be tested, not assumed.
• Claim size barely separates the classes. Mean estimated payout was 4,890 for fraudulent claims and 4,992 for non-fraudulent ones. Ranking by payout alone is not screening, and a model that mostly reproduces payout will look busy while adding little.
• Liability allocation is almost identical. Mean liability percentage was 49.5% for fraudulent claims and 49.4% for non-fraudulent ones — no usable separation on its own.
• Some signals are real but modest. A police report was filed on 63.5% of fraudulent claims versus 59.4% of non-fraudulent ones. That is a difference worth using, not a rule worth betting the queue on.
• Dates are text. Claim dates are stored as m/d/Y strings. Sorting them as text, or parsing them day-first, silently swaps days and months. The supplied day-of-week field has to be re-derived from the parsed date rather than trusted separately.
• Money has no stated currency. The payout field is a number without a unit in the file. It can rank claims, but it should not be reported as a currency amount without confirmation.
• Missingness is small but not zero. 132 records lack a witness indicator, 17 lack a payout estimate, 8 lack vehicle age and 5 lack marital status. Dropping them is defensible only if the rule is documented.
• The grain is a claim, not a person. Claim numbers are unique, so claim-level modelling is safe. Investigations, however, are assigned to people and policies, so the queue also needs a claimant-level rollup to avoid sending six separate alerts about one policyholder.
• There is no free text. No adjuster notes and no claim description. The screen can only use structured fields; it cannot read a story.

How the analysis was built

1. Reconcile the grain. Claim number was checked for uniqueness, and the row count was confirmed against the fraud flag totals.
2. Fix the date handling. Dates were parsed with an explicit month-first format, day-of-week was re-derived, and the file was split by time — earlier claims for training, later claims for validation — so the reported performance reflects future claims rather than a random reshuffle.
3. Audit for leakage. Identifiers and any field that could only be known after an investigation were excluded before modelling.
4. Test counterintuitive signals. Witness, police report and accident site were examined together rather than encoded as standalone rules.
5. Rank by expected value. Probability was calibrated and multiplied by estimated payout, so a small certain claim does not outrank a large uncertain one.
6. Set the threshold from capacity. The cut-off was derived from how many claims the investigations team can examine per week, and performance was reported as precision at that operating point, not as overall accuracy.
7. Attach reasons and monitoring. Each queued claim carries the drivers behind its score, and the monitoring plan tracks score distribution, observed fraud rate and feature drift.

What this changes

• The queue is ranked by expected loss, not raw risk. Review time goes to claims where the money at stake justifies it.
• Assumptions are tested before they become rules. The witness finding alone would have sent the team in the wrong direction.
• Precision at the operating point becomes the headline metric. "How many of the claims we reviewed were fraudulent" is actionable; "accuracy" is not.
• The date field stops corrupting seasonality. Month-first text dates are a silent source of wrong trend charts.
• Monitoring is part of the deliverable. A queue without drift checks decays quietly.

What you receive

• A cleaned claim table with a documented grain and exclusion rule.
• A time-based validation report showing performance on later claims.
• A calibrated risk score with reason codes per claim.
• A threshold memo tied to review capacity and expected loss.
• A monitoring sheet covering score drift, fraud rate and data quality.

Where this applies

The method fits insurers, warranty providers, chargeback teams and any operation that must triage a finite review capacity against labelled historical outcomes. The minimum inputs are a claim identifier, an outcome flag, a date, an amount and structured claim and claimant attributes. The first deliverable is usually the operating-point memo — because the useful question is never "how accurate is the model" but "how many claims can we review, and what will that catch".

This is an illustrative case study using an anonymized public insurance claims dataset; it does not report results from a specific client engagement or a validated production model. It reflects our standard method for claims screening.