MIPD for Drug X: results

Forecast agreement on day 154

0.54

Cohen's kappa using troughs and all biomarker values to day 42 (95% CI 0.45 to 0.62). Covariates only: 0.05.

At target with the 80% rule

88.3%

Standard regimen 79.4%. Same dose rate, 0.62 mg/kg/week.

At target with the point-forecast rule

76.7%

Below the standard regimen. Its picks land just under 200.

Oracle ceiling

98.5%

The same search with the true parameters, at 0.53 mg/kg/week.

What the studies show

The exposure driver can be told apart. On data driven by Cave the Cave model wins by 315 AIC points, and on data driven by the concentration the concentration model wins by 134.
Troughs alone barely help the biomarker forecast (kappa 0.13 against 0.05 with covariates only). The pre-treatment biomarker lifts it to 0.38 and the values through day 42 to 0.54.
The 80% rule gets 88.3% of assessments to target against 79.4% for the standard regimen, using about the same drug. The point-forecast rule does worse than standard (76.7%). The trough rule reaches 92.5% with 2.0 times the drug.
When the true Imax is 0.70 the model still thinks more drug helps. Both MAP rules use 2.4 to 2.8 times the standard dose rate for a gain of 1 to 4 points.

How it works

Drug X is a made-up IV drug with a two-compartment PK model. The biomarker follows an indirect response model, driven by Cave, the average concentration over each dosing interval.
Each patient is fitted in two steps. MAP for CL, V1 and one IOV term per interval from the troughs (BLQ values count as censored), then MAP for BASE and IC50 from the biomarker, with Cave from the PK fit as a time-varying driver.
The next regimen is the cheapest on a grid of 2.5-15 mg/kg every 4-10 weeks that is forecast below 200. The point rule uses the posterior mode; the 80% rule needs 80% of posterior draws below target.
Study sizes: 800 patients for forecasting and diagnostics, 500 new ones for dosing, 150 for the driver comparison. The full run took about 8 minutes on 12 worker processes.

More detail on the model and the estimation is in the README.

Try it on one patient

The dosing app runs the same sequential MAP fit in your browser. Enter covariates, doses, troughs and biomarker values, and it shows the fitted profiles, the forecast and the cheapest regimen that meets the rule. It opens with a patient whose trough is fine while the biomarker is still above target.

Open the dosing app

The first load downloads Python, numpy, scipy, pandas and matplotlib into the browser (Pyodide), so it takes a little while.

Which exposure metric should drive the PD model

I fitted a population PD model to the same 150 patients three times, each with a different driver: Cave per interval, the concentration itself (in 1-day steps) and cumulative AUC. To check that the comparison can tell drivers apart, the biomarker was simulated twice from the same patients, once driven by Cave and once by the concentration. The true driver wins in both. Cumulative AUC is far behind (847 and 1119 AIC points) because it only grows, so the modeled effect can only increase over time.

Data simulated with

Static matplotlib version: drivers.png

Estimates when the fitted driver matches the data

Population estimates next to the values the data were simulated from. SDs are on the log scale. All 6 fits reported convergence.

Fit	BASE (units)	IC50 (mg/L)	kout (/day)	IIV SD, BASE	IIV SD, IC50	Residual SD	Converged
Simulation values	600	3.00	0.0400	0.55	0.77	0.246
Cave data, Cave model	648	3.02	0.0396	0.52	0.72	0.258	yes
Concentration data, concentration model	637	3.00	0.0380	0.52	0.68	0.265	yes

Scenario and forecast day

These two apply to the charts and the table below. The metric chart shows both days; the scatter is for the matched scenario only.

Scenario Forecast day

How much each kind of data adds

Each patient is fitted on data up to and including the day 42 visit, then the biomarker is forecast at the day 154 and day 266 visits under the standard regimen. Below 200 units counts as at target, the positive class. With 82% of patients at target, calling everyone at target already scores well on accuracy, so balanced accuracy and kappa are the better guide. Error bars are 95% CIs (Wilson for proportions, a patient bootstrap for the rest).

Metric

Static matplotlib version: forecast_summary.png

Forecast against the measured value

Each point is one patient in the matched scenario. Orange points were forecast on the wrong side of 200. For the forecasts that come with a 90% prediction interval, the horizontal bars show it. The diagonal is perfect agreement.

Forecast

All forecast metrics

Accuracy, balanced accuracy and kappa with 95% CIs. The last row scores the noise-free true biomarker against the measured one, which is as good as any forecast can get with 25% assay noise. Click a column header to sort.

Forecast	Accuracy (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Balanced accuracy	Kappa	RMSE log10	Within 1.5-fold (%)	90% PI coverage (%)
Covariates only	82 (79 to 85)	99	4	83	50	0.52 (0.50 to 0.54)	0.05 (-0.00 to 0.11)	0.399	35
Troughs only	83 (80 to 86)	99	10	84	65	0.54 (0.52 to 0.57)	0.13 (0.05 to 0.21)	0.384	35
Troughs + baseline biomarker	85 (82 to 88)	96	35	87	67	0.66 (0.61 to 0.70)	0.38 (0.28 to 0.48)	0.329	39
Troughs + all biomarker values	88 (85 to 90)	94	56	91	68	0.75 (0.70 to 0.80)	0.54 (0.45 to 0.62)	0.285	48	86
Last value carried forward	71 (67 to 74)	67	87	96	36	0.77 (0.73 to 0.80)	0.35 (0.28 to 0.41)	0.481	32
True value (noise ceiling)	94 (92 to 96)	98	78	95	89	0.88 (0.84 to 0.92)	0.80 (0.73 to 0.86)	0.111	88

Forecast	Accuracy (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Balanced accuracy	Kappa	RMSE log10	Within 1.5-fold (%)	90% PI coverage (%)
Covariates only	83 (80 to 86)	99	5	83	50	0.52 (0.50 to 0.54)	0.06 (0.00 to 0.12)	0.390	37
Troughs only	84 (81 to 87)	99	12	84	65	0.55 (0.53 to 0.59)	0.16 (0.07 to 0.25)	0.376	38
Troughs + baseline biomarker	87 (84 to 89)	97	38	88	70	0.67 (0.63 to 0.72)	0.42 (0.32 to 0.53)	0.330	41
Troughs + all biomarker values	88 (85 to 90)	95	56	91	69	0.75 (0.70 to 0.80)	0.54 (0.45 to 0.63)	0.286	46	85
Last value carried forward	72 (68 to 75)	69	87	96	36	0.78 (0.74 to 0.81)	0.36 (0.29 to 0.42)	0.473	33
True value (noise ceiling)	95 (93 to 96)	98	80	96	88	0.89 (0.85 to 0.93)	0.81 (0.74 to 0.87)	0.107	90

Forecast	Accuracy (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Balanced accuracy	Kappa	RMSE log10	Within 1.5-fold (%)	90% PI coverage (%)
Covariates only	60 (56 to 63)	99	3	60	70	0.51 (0.50 to 0.52)	0.02 (-0.00 to 0.05)	0.435	30
Troughs only	61 (57 to 64)	99	5	60	82	0.52 (0.51 to 0.54)	0.05 (0.02 to 0.09)	0.433	28
Troughs + baseline biomarker	67 (64 to 71)	99	22	65	95	0.60 (0.58 to 0.63)	0.24 (0.18 to 0.29)	0.387	25
Troughs + all biomarker values	78 (74 to 81)	94	55	75	86	0.74 (0.71 to 0.77)	0.51 (0.45 to 0.58)	0.252	51	88
Last value carried forward	76 (73 to 79)	62	96	96	64	0.79 (0.77 to 0.82)	0.54 (0.48 to 0.60)	0.272	50
True value (noise ceiling)	89 (87 to 91)	92	85	90	88	0.88 (0.86 to 0.91)	0.77 (0.72 to 0.82)	0.111	88

Forecast	Accuracy (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Balanced accuracy	Kappa	RMSE log10	Within 1.5-fold (%)	90% PI coverage (%)
Covariates only	61 (57 to 64)	99	3	60	80	0.51 (0.50 to 0.52)	0.03 (0.01 to 0.06)	0.419	27
Troughs only	62 (58 to 66)	99	7	61	85	0.53 (0.51 to 0.55)	0.07 (0.03 to 0.11)	0.417	29
Troughs + baseline biomarker	69 (65 to 72)	100	23	66	100	0.61 (0.59 to 0.64)	0.26 (0.21 to 0.32)	0.371	27
Troughs + all biomarker values	79 (75 to 82)	95	55	76	87	0.75 (0.71 to 0.78)	0.53 (0.46 to 0.59)	0.244	51	90
Last value carried forward	75 (72 to 79)	62	95	95	63	0.79 (0.76 to 0.81)	0.53 (0.47 to 0.59)	0.268	49
True value (noise ceiling)	90 (87 to 92)	92	86	90	88	0.89 (0.86 to 0.91)	0.78 (0.73 to 0.83)	0.107	90

Forecast	Accuracy (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Balanced accuracy	Kappa	RMSE log10	Within 1.5-fold (%)	90% PI coverage (%)
Covariates only	39 (35 to 43)	99	2	38	80	0.51 (0.50 to 0.51)	0.01 (-0.00 to 0.02)	0.558	12
Troughs only	40 (36 to 44)	99	4	39	88	0.51 (0.50 to 0.53)	0.02 (0.01 to 0.04)	0.559	14
Troughs + baseline biomarker	47 (43 to 51)	100	15	42	98	0.57 (0.55 to 0.59)	0.11 (0.09 to 0.15)	0.525	6
Troughs + all biomarker values	71 (67 to 74)	91	58	57	92	0.75 (0.71 to 0.78)	0.44 (0.38 to 0.50)	0.288	44	81
Last value carried forward	81 (77 to 83)	59	94	85	79	0.77 (0.73 to 0.80)	0.56 (0.49 to 0.63)	0.210	62
True value (noise ceiling)	89 (86 to 91)	86	90	84	91	0.88 (0.85 to 0.91)	0.76 (0.71 to 0.81)	0.111	88

Forecast	Accuracy (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Balanced accuracy	Kappa	RMSE log10	Within 1.5-fold (%)	90% PI coverage (%)
Covariates only	39 (35 to 43)	99	2	39	80	0.51 (0.50 to 0.51)	0.01 (-0.00 to 0.02)	0.541	13
Troughs only	41 (37 to 45)	99	5	39	90	0.52 (0.51 to 0.53)	0.03 (0.01 to 0.05)	0.542	13
Troughs + baseline biomarker	47 (44 to 51)	100	15	42	100	0.57 (0.56 to 0.59)	0.12 (0.09 to 0.15)	0.506	6
Troughs + all biomarker values	72 (68 to 75)	92	59	58	92	0.75 (0.72 to 0.79)	0.46 (0.40 to 0.52)	0.277	49	84
Last value carried forward	79 (76 to 82)	57	94	85	78	0.75 (0.72 to 0.78)	0.54 (0.47 to 0.60)	0.206	62
True value (noise ceiling)	88 (85 to 91)	84	91	85	90	0.87 (0.84 to 0.90)	0.75 (0.69 to 0.80)	0.107	90

Share at target against the drug used

500 new patients start on the standard regimen and get three dosing decisions, the first on day 98. Each decision uses all data so far, and the outcome is the true (noise-free) biomarker at the next assessment. All strategies see the same patients and the same random draws. The point-forecast rule lowers the dose rate in 77% of its decisions, but it picks regimens forecast to land just under 200: 57% of its forecasts were between 150 and 200, and 39% of those ended at 200 or more. The oracle shows how far the grid could go, so most of the gap to it is uncertainty in the individual estimates.

ScenarioOutcome

Static matplotlib version: recommender.png

All strategies

Percent of assessments at target, overall and by decision, the share of patients at target at all three, the mean dose rate (mg/kg per week) and how often the strategy moved away from the standard regimen.

Scenario	Strategy	All assessments	1st	2nd	3rd	All three	Dose rate	Changed (%)
Model matches truth	Standard, 5 mg/kg every 8 weeks	79.4	80.2	79.6	78.4	77.2	0.62	0
Model matches truth	Trough rule (aim for 3 mg/L)	92.5	90.2	93.6	93.8	89.0	1.23	48
Model matches truth	MAP, point forecast below 200	76.7	75.6	77.0	77.4	65.4	0.52	96
Model matches truth	MAP, 80% probability below 200	88.3	88.0	89.2	87.6	78.2	0.62	97
Model matches truth	Oracle (true parameters)	98.5	97.6	99.0	99.0	97.6	0.53	97
True Imax 0.70	Standard, 5 mg/kg every 8 weeks	35.5	36.8	35.2	34.6	34.0	0.62	0
True Imax 0.70	Trough rule (aim for 3 mg/L)	40.2	38.8	39.8	42.0	37.6	1.23	48
True Imax 0.70	MAP, point forecast below 200	36.5	36.2	36.6	36.6	34.0	1.51	97
True Imax 0.70	MAP, 80% probability below 200	39.8	38.6	40.4	40.4	37.8	1.76	97
True Imax 0.70	Oracle (true parameters)	46.3	44.2	46.8	47.8	44.2	2.05	99

Trough looks fine, biomarker does not

Patients whose trough at the first decision was at least 3 mg/L while the biomarker was 200 or more. This is the case the method is meant for: a trough-based rule leaves them alone. In the matched scenario only 8 of 500 patients fit, so the counts are small.

One of those patients

The drug level is fine, but this patient needs more of it. MAP picks up the high IC50 from the biomarker, and the 80% rule raises the dose. Hover over the curves for values; click a legend entry to hide it.

Weight: 42 kg
Trough on day 98: 5.1 mg/L
Biomarker on day 98: 303 units
IC50, MAP estimate: 6.7 mg/L (true 7.0, typical 2.9 for this patient's PLT)
Forecast on day 154, staying on 5 mg/kg every 8 weeks: 334 (true 354)
Forecast on day 154, 15 mg/kg every 8 weeks: 156 (true 170)

Static matplotlib version: case.png

The dosing app opens with a patient like this one, so you can change the inputs and see what the fit does.

Open the dosing app

Every regimen on the grid for this patient

The 80% rule takes the lowest dose rate with at least 80% of posterior draws below 200, preferring the longer interval on a tie.

Show

Forecasting a biomarker and picking the next dose of Drug X