ML portfolio results

Four small machine learning projects. Each runs on a laptop CPU in under three minutes and has a test suite, and each is built around a decision someone would have to make instead of a leaderboard score.

Synthetic data throughout. Every dataset is generated by code in this repository, so the feature engineering, leakage tests and evaluation can be read and rerun. None of these numbers say anything about a real business.

Fraud detection

Card fraud at a 0.17% base rate.

The alert threshold comes from putting a cost on each type of error instead of maximising a score. Adding scale_pos_weight to handle the imbalance made the ranking worse.

Fraud detection results — Ranking quality and money kept. They're on different scales, so each gets its own panel.

Model	PR-AUC	Precision	Recall	Net savings
Rule baseline	0.301	43.0%	38.1%	46%
Logistic regression	0.506	44.6%	51.5%	53%
LightGBM	0.757	72.7%	66.0%	60%
LightGBM + weight	0.397	54.5%	62.9%	59%

Demand forecasting

28-day demand for 60 SKUs.

Every lag is derived from the forecast horizon, so lag_1 can't be built at a 28-day horizon. Getting this wrong leaks the future into the features.

Demand forecasting results — Error by model across six rolling origins, and error by how many days ahead the forecast is.

Model	WAPE	Worst fold	Value add vs naive
LightGBM (Tweedie)	0.387	0.404	+36%
Lightgbm L2	0.397	0.421	+34%
Moving Average 28D	0.476	0.511	+21%
Last Year	0.580	0.614	+4%
Seasonal naive	0.606	0.623	+0%

Semantic similarity

Duplicate question detection.

The split protocol moves the score more than the model does, and changes which model you'd pick. A random pair split leaks questions between train and test.

Semantic similarity results — The same four models under both split protocols. The gap between the two bars in each pair is the effect of the protocol.

Model	random pair split	cluster disjoint split	Difference
TF-IDF cosine	0.563	0.588	-0.025
Logistic regression	0.655	0.679	-0.024
LightGBM	0.852	0.689	+0.162
LightGBM + question stats	0.861	0.664	+0.197

Recommender system

Implicit-feedback recommendations.

Implicit ALS written from scratch. If you pick a model on accuracy alone, you get one that recommends the same popular items to everyone.

Recommender system results — Recall against catalogue coverage. A model can win on recall while showing a fraction of the catalogue.

Model	Recall@10	nDCG	Catalogue coverage	Novelty
Random	0.004	0.002	100.0%	11.24
Popularity	0.053	0.034	1.7%	8.11
Item k-NN	0.125	0.089	35.0%	9.42
Implicit ALS	0.114	0.075	56.9%	9.64

Generated 25 September 2026 from the metrics each training run wrote, by python -m dashboard.build. The figures are drawn with matplotlib from the same files, so they always match the tables. Source: Wrlog/ml-practice.