Drug X Response Prediction

The charts need JavaScript. The tables below have every number, and the static figures are in the repository.

Best out-of-fold AUROC

0.925

Elastic net on L4, bootstrap 95% CI 0.893 to 0.960

Baseline data only

0.806

best of four algorithms on L1, known on day 0

L3 to L4 gain

+0.087

elastic net, corrected p <0.001; PK/PD refitted with data to day 98

Ceiling

0.981

AUROC of the true simulated probability of being at target

Simulated cohort

189

patients in 3 studies, 72% at target

Findings

Baseline data alone give an AUROC of 0.798 to 0.806 across the four algorithms. The true simulated probabilities reach 0.98, so a lot is left to explain.
The first PK/PD block (data to day 21) adds +0.013 to +0.033. A naive paired t-test calls all four significant (p 0.012 to 0.043); the corrected resampled t-test does not (p 0.21 to 0.33).
Refitting PK/PD with data to day 98 is the big step: +0.043 to +0.087, corrected p below 0.05 for three of the four algorithms. The same day-98 data fed in as raw values (L4r) do about as well, so here the model mostly repackages what the raw data carry.
SHAP from all four refitted models puts the day-98 IC50 estimate first (22% to 27% of the total). Summed SHAP tracks the true mechanism well for baseline level and drug sensitivity and poorly for exposure.

The feature ladder

Each step adds a block of features. The dot marks the day every feature in the step is known; the outcome is read between day 315 and day 399. L4r is a side check outside the ladder.


L1	Baseline clinical	baseline clinical	9	day 0
L2	+ baseline PK/PD	baseline clinical + PK/PD fit to infusion 2	14	day 21
L3	+ clinical to infusion 3	baseline clinical + clinical at infusion 2 + clinical at infusion 3 + PK/PD fit to infusion 2	28	day 42
L4	PK/PD updated at infusion 4	baseline clinical + clinical at infusion 2 + clinical at infusion 3 + PK/PD fit to infusion 4	28	day 98
L5	Everything	baseline clinical + clinical at infusion 2 + clinical at infusion 3 + PK/PD fit to infusion 2 + PK/PD fit to infusion 3 + PK/PD fit to infusion 4	38	day 98
L4r (check)	Raw data to infusion 4	baseline clinical + clinical at infusion 2 + clinical at infusion 3 + clinical at infusion 4 + PK/PD fit to infusion 2	35	day 98

Discrimination

AUROC by feature group and algorithm

Mean out-of-fold AUROC over 3 repeats of 5-fold nested CV. The repeat SD only reflects seed sensitivity; the SD over outer folds (0.05 to 0.09, with 37 to 38 patients per test fold) is the better guide. Click a legend entry to hide an algorithm.

Error bars

Static PNG


L1	0.806 (0.013)	0.805 (0.013)	0.798 (0.013)	0.800 (0.011)
L2	0.818 (0.015)	0.837 (0.009)	0.830 (0.015)	0.831 (0.002)
L3	0.838 (0.010)	0.851 (0.005)	0.839 (0.013)	0.851 (0.010)
L4	0.925 (0.006)	0.897 (0.014)	0.899 (0.010)	0.894 (0.007)
L5	0.925 (0.009)	0.883 (0.001)	0.886 (0.015)	0.894 (0.007)
L4r	0.916 (0.013)	0.915 (0.002)	0.912 (0.004)	0.918 (0.009)

Change at each step

Small points are the 15 paired per-fold differences, large points their mean. p values in the table use the Nadeau and Bengio corrected resampled t-test; the naive paired t-test is shown next to it to show how much it overstates the evidence.

Static PNG


L1 to L2	Elastic net	+0.013	0.29	0.03
L1 to L2	Random forest	+0.032	0.21	0.01
L1 to L2	XGBoost	+0.033	0.33	0.04
L1 to L2	CatBoost	+0.031	0.26	0.02
L2 to L3	Elastic net	+0.020	0.37	0.06
L2 to L3	Random forest	+0.014	0.67	0.36
L2 to L3	XGBoost	+0.009	0.76	0.51
L2 to L3	CatBoost	+0.019	0.40	0.08
L3 to L4	Elastic net	+0.087	<0.001	<0.001
L3 to L4	Random forest	+0.045	0.01	<0.001
L3 to L4	XGBoost	+0.059	0.02	<0.001
L3 to L4	CatBoost	+0.043	0.08	0.001
L4 to L5	Elastic net	−0.001	0.80	0.59
L4 to L5	Random forest	−0.014	0.35	0.05
L4 to L5	XGBoost	−0.013	0.30	0.04
L4 to L5	CatBoost	−0.001	0.96	0.92
L4r to L4	Elastic net	+0.009	0.58	0.24
L4r to L4	Random forest	−0.018	0.46	0.12
L4r to L4	XGBoost	−0.013	0.46	0.12
L4r to L4	CatBoost	−0.024	0.33	0.04

Raw values against the PK/PD model

L4 and L4r use data up to the same day. Tree models do slightly better with the raw values (−0.024 to −0.013 for L4 against L4r) and the elastic net slightly worse (+0.009). None of these is significant (corrected p 0.33 to 0.58).

Checks

Leave one study out

Train on two studies, test on the third. At L4 the mean AUROC in the held-out study is 0.84 to 0.91; study B, with the lightest patients and the most ADA, is the hardest. Columns only one study measured are dropped when that study is held out.

Feature group

All leave-one-study-out results


A	L1	0.770	0.790	0.765	0.742
B	L1	0.670	0.742	0.727	0.658
C	L1	0.850	0.857	0.834	0.863
A	L2	0.807	0.818	0.794	0.831
B	L2	0.711	0.737	0.743	0.718
C	L2	0.767	0.839	0.820	0.834
A	L3	0.845	0.879	0.810	0.876
B	L3	0.738	0.780	0.774	0.773
C	L3	0.833	0.889	0.882	0.866
A	L4	0.887	0.887	0.885	0.922
B	L4	0.871	0.807	0.855	0.837
C	L4	0.939	0.895	0.894	0.911
A	L5	0.888	0.903	0.891	0.896
B	L5	0.867	0.821	0.824	0.846
C	L5	0.942	0.915	0.908	0.923

Missingness and study-only checks

Missingness depends on the study. A model on missing-value indicators alone reaches 0.64 to 0.68, and one on study membership alone 0.68. Error bars are the SD over outer folds.

Missing data by study

Share of patients with each feature missing, for the features that are ever missing. Columns more than 70% missing in a training fold are dropped.

Imputation sensitivity

L5 with five imputers, first repeat only, mean and SD over 5 folds. The spread within an algorithm is at most 0.038, well inside the fold-to-fold SD.

Calibration

Calibration of the best model

Elastic net on L4, out-of-fold predictions averaged over repeats and split into fifths. Slope 1.08, intercept 0.03.

Static PNG

Calibration slope and intercept

From logistic recalibration of the out-of-fold predictions. XGBoost is overconfident (slope 0.78 to 0.88) and random forest underestimates the chance of being at target at L1, L3, L4 and L5 (intercept 0.20 to 0.46).

Show

Table


L1	Elastic net	1.06	+0.02
L1	Random forest	1.06	+0.46
L1	XGBoost	0.88	−0.03
L1	CatBoost	1.08	+0.15
L2	Elastic net	0.99	+0.02
L2	Random forest	1.20	+0.07
L2	XGBoost	0.83	−0.04
L2	CatBoost	1.19	+0.17
L3	Elastic net	1.20	+0.00
L3	Random forest	1.10	+0.32
L3	XGBoost	0.78	−0.09
L3	CatBoost	1.24	+0.15
L4	Elastic net	1.08	+0.03
L4	Random forest	1.29	+0.34
L4	XGBoost	0.86	−0.22
L4	CatBoost	1.11	+0.00
L5	Elastic net	1.02	+0.04
L5	Random forest	1.16	+0.20
L5	XGBoost	0.81	−0.15
L5	CatBoost	1.29	+0.02

Precision and recall

AUPRC is 0.97 at L4 against 0.89 at L1, with a prevalence of 0.72.

Static PNG

Decision curve

The action is dose intensification for a patient predicted to miss target. The L4 model beats intensifying everyone or no one at every threshold from 0.05 to 0.60, for example 0.20 at 0.3 where treating everyone gives −0.03.

Static PNG

SHAP

Global SHAP importance

Top features of the model refitted on the full cohort (L5). The refitted models keep 29 (elastic net), 12 (random forest), 6 (XGBoost) and 6 (CatBoost) features. SHAP is on the log-odds scale except for random forest, so sizes are not comparable across algorithms.

Algorithm

Static PNG

SHAP per patient

Each dot is a patient. Color is the feature value within the cohort, from low (blue) to high (orange). Positive SHAP pushes the prediction towards target; a high IC50 or BASE estimate pushes it away.

Algorithm

Static PNG

Consensus across algorithms

Share of total |SHAP| per feature for the features with the largest mean share. The day-98 IC50 estimate is first for all four; the trees lean more on the raw day-42 biomarker.

Static PNG

Feature selection frequency

Share of the 15 outer folds in which each feature was kept (RFE for the trees, a non-zero coefficient for the elastic net). On L5 the day-98 IC50 estimate is kept in 100% of folds by every algorithm.

Feature group

Static PNG

SHAP against the known mechanism

The true late biomarker is split into exact Shapley parts for exposure, baseline level and drug sensitivity, and SHAP is summed within the feature groups that mostly carry each part. The truth gives exposure 25%; the models give it 0% to 18%. Spearman correlations per patient are 0.80 to 0.92 for baseline level and 0.69 to 0.79 for drug sensitivity.

Static PNG


Elastic net	0.54	0.92	0.79
Random forest	0.41	0.91	0.74
XGBoost	none kept	0.80	0.69
CatBoost	none kept	0.81	0.69

Summed SHAP against the true part, per patient

Pick an algorithm and a mechanism group. XGBoost and CatBoost dropped every exposure feature during feature elimination, so their exposure panels are flat.

AlgorithmGroup

Cohort and PK/PD

Simulated biomarker trajectories

Observed biomarker for every patient, colored by outcome. A patient is at target if the first value in the outcome window is below 200 units. Double-click a legend entry to show one group only.

Study

Static PNG


A	77	87%	13%	65.3	618
B	62	68%	45%	39.9	841
C	50	54%	28%	57.5	788
All	189	72%	28%	54.9	707

MAP estimates against the truth

Individual estimates from the sequential PK-then-PD fit against the simulated true values. IC50 is barely identifiable early (r 0.31 with data to day 21) and much clearer by day 98 (r 0.79). The table gives r on the log scale and shrinkage.

Fit withParameter

Static PNG


data to day 21	0.86 / 0.20	0.90 / 0.53	0.94 / 0.08	0.31 / 0.81
data to day 42	0.94 / -0.03	0.90 / 0.51	0.96 / 0.07	0.41 / 0.48
data to day 98	0.95 / -0.03	0.90 / 0.22	0.96 / 0.06	0.79 / 0.12