Clean data, no levels given
1.169
Latent ODE median fold error. popPK with covariates 1.171, true model 1.172.
Clean data, 16 levels given
1.081
Latent ODE. popPK MAP gets 1.004, the true model 1.000.
ADA onset before level 24
1.247
Latent ODE, 45 patients. Constant-CL popPK 1.384, time-varying CL 1.161.
Structural checks passed
16 of 17
Failed: clean network, dose proportionality.
Full pipeline
165 s
Laptop CPU. Networks stopped early after 135 to 151 epochs.

What the results say

Forecast error as levels come in

Forecast after each patient's k-th level, scored against the exact concentration at every later sampling time (120 test patients per dataset). Median fold error is 1 for a perfect forecast. The true model is the ceiling: it knows the structural model, the covariate effects and, in the ADA data, each patient's onset time.

All forecast errors

For the dataset picked above. Click a column header to sort.

MethodkMedian fold errorR²Points scored
Latent ODE01.1690.8774,047
popPK, no covariates01.3040.8084,047
popPK with covariates01.1710.8804,047
True model (ceiling)01.1720.8804,047
Latent ODE11.1840.8413,927
popPK, no covariates11.1670.8173,927
popPK with covariates11.1260.8893,927
True model (ceiling)11.1280.8893,927
Latent ODE21.1770.8413,807
popPK, no covariates21.1230.8983,807
popPK with covariates21.0250.9913,807
True model (ceiling)21.0210.9953,807
Latent ODE31.1690.8723,687
popPK, no covariates31.0700.9243,687
popPK with covariates31.0230.9983,687
True model (ceiling)31.0021.0003,687
Latent ODE41.1420.9203,567
popPK, no covariates41.0610.9423,567
popPK with covariates41.0101.0003,567
True model (ceiling)41.0001.0003,567
Latent ODE61.0860.9833,327
popPK, no covariates61.0260.9903,327
popPK with covariates61.0041.0003,327
True model (ceiling)61.0001.0003,327
Latent ODE81.0930.9873,087
popPK, no covariates81.0280.9893,087
popPK with covariates81.0041.0003,087
True model (ceiling)81.0001.0003,087
Latent ODE121.0740.9902,607
popPK, no covariates121.0280.9922,607
popPK with covariates121.0041.0002,607
True model (ceiling)121.0001.0002,607
Latent ODE161.0810.9882,127
popPK, no covariates161.0340.9902,127
popPK with covariates161.0041.0002,127
True model (ceiling)161.0001.0002,127
Latent ODE201.1060.9861,662
popPK, no covariates201.0290.9981,662
popPK with covariates201.0031.0001,662
True model (ceiling)201.0001.0001,662
Latent ODE241.0650.9911,182
popPK, no covariates241.0270.9981,182
popPK with covariates241.0031.0001,182
True model (ceiling)241.0001.0001,182
Latent ODE01.1930.8824,028
popPK, no covariates01.3110.8354,028
popPK with covariates01.1880.8834,028
True model (ceiling)01.1890.8814,028
Latent ODE11.2080.8633,908
popPK, no covariates11.2310.8403,908
popPK with covariates11.1620.8873,908
True model (ceiling)11.1580.8863,908
Latent ODE21.2070.8543,788
popPK, no covariates21.2180.8433,788
popPK with covariates21.1580.8853,788
True model (ceiling)21.1620.8853,788
Latent ODE31.2030.8763,668
popPK, no covariates31.1750.8703,668
popPK with covariates31.1520.9003,668
True model (ceiling)31.1410.8903,668
Latent ODE41.1880.9083,548
popPK, no covariates41.1460.8923,548
popPK with covariates41.1340.9163,548
True model (ceiling)41.1340.9073,548
Latent ODE61.1540.9373,308
popPK, no covariates61.1250.9313,308
popPK with covariates61.1180.9393,308
True model (ceiling)61.1190.9373,308
Latent ODE81.1640.9273,068
popPK, no covariates81.1290.9273,068
popPK with covariates81.1200.9353,068
True model (ceiling)81.1200.9323,068
Latent ODE121.1470.9512,588
popPK, no covariates121.1050.9522,588
popPK with covariates121.0920.9572,588
True model (ceiling)121.0930.9562,588
Latent ODE161.1640.9502,108
popPK, no covariates161.1190.9462,108
popPK with covariates161.1060.9542,108
True model (ceiling)161.1070.9532,108
Latent ODE201.1450.9521,628
popPK, no covariates201.1030.9581,628
popPK with covariates201.0950.9591,628
True model (ceiling)201.0970.9591,628
Latent ODE241.1670.9491,148
popPK, no covariates241.1220.9511,148
popPK with covariates241.1160.9521,148
True model (ceiling)241.1170.9521,148
Latent ODE01.2230.8543,961
popPK, no covariates01.3410.7853,961
popPK with covariates01.2190.8443,961
popPK, time-varying CL01.2190.8443,961
True model (ceiling)01.1970.8913,961
Latent ODE11.2220.8293,841
popPK, no covariates11.2850.7833,841
popPK with covariates11.2040.8423,841
popPK, time-varying CL11.2040.8413,841
True model (ceiling)11.1720.8933,841
Latent ODE21.2310.8193,721
popPK, no covariates21.2620.7853,721
popPK with covariates21.2030.8403,721
popPK, time-varying CL21.2020.8403,721
True model (ceiling)21.1720.8933,721
Latent ODE31.2250.8283,601
popPK, no covariates31.2430.8163,601
popPK with covariates31.1980.8553,601
popPK, time-varying CL31.1940.8543,601
True model (ceiling)31.1750.9063,601
Latent ODE41.2070.8693,481
popPK, no covariates41.1700.8783,481
popPK with covariates41.1630.8913,481
popPK, time-varying CL41.1560.8873,481
True model (ceiling)41.1410.9273,481
Latent ODE61.1840.9163,241
popPK, no covariates61.1480.9003,241
popPK with covariates61.1360.9043,241
popPK, time-varying CL61.1370.8993,241
True model (ceiling)61.1180.9503,241
Latent ODE81.1810.9093,001
popPK, no covariates81.1480.8923,001
popPK with covariates81.1350.8973,001
popPK, time-varying CL81.1360.8863,001
True model (ceiling)81.1190.9473,001
Latent ODE121.1950.9222,521
popPK, no covariates121.1290.9002,521
popPK with covariates121.1290.9012,521
popPK, time-varying CL121.1240.8902,521
True model (ceiling)121.1000.9612,521
Latent ODE161.2130.9082,041
popPK, no covariates161.1620.8852,041
popPK with covariates161.1620.8872,041
popPK, time-varying CL161.1590.8882,041
True model (ceiling)161.1160.9592,041
Latent ODE201.1810.9211,558
popPK, no covariates201.1490.9021,558
popPK with covariates201.1290.9041,558
popPK, time-varying CL201.1130.9121,558
True model (ceiling)201.0950.9691,558
Latent ODE241.1880.9271,078
popPK, no covariates241.1650.8981,078
popPK with covariates241.1460.8991,078
popPK, time-varying CL241.1270.9411,078
True model (ceiling)241.1070.9651,078

Example profiles

Three test patients from the ADA dataset. Pick a patient and how many of their levels the models see. The popPK forecast is MAP with the covariate model. Filled points are the levels given, open ones are held out. Neither model knows about ADA.

Forecast error by ADA status

ADA test patients split by whether onset happened before the last level given. The groups change with k, so the patient counts do too (ADA negative: 68 patients; onset before last level: 11 to 45 patients; onset after last level: 7 to 52 patients). Hover for the count at each k. Nobody can forecast an onset that hasn't happened yet; the ceiling is only low there because it knows the onset time.

Covariate recovery

Every test patient from the dose-level dataset gets each value of one covariate in turn, at 5 mg/kg with no levels given. The curve is the mean concentration over the dosing interval after the third dose, averaged over patients. PLT has no effect in the true model.

Effect size, highest over lowest value

Ratio of the mean concentrations at the two ends of each grid. The network gets the direction of the weight, albumin and INF effects and most of their size, but it also picks up a PLT effect that isn't there.

Covariate rangeTruthpopPK with covariatesLatent ODE
Weight, 25 to 951.521.551.44
Albumin, 3 to 4.81.821.631.55
INF, 1 to 400.630.580.64
PLT, 180 to 5201.000.920.86

Structural checks

Run on each network's test patients. The clean network fails dose proportionality: everyone in that dataset got the same mg/kg dose, and dose-rescaling augmentation alone doesn't fix it. The two networks trained on several dose levels pass. The unseen-interval check gives the dose-level network a maintenance interval it never saw in training.

CheckWhat is measuredCleanDose levels + assay errorADA
Positivitysmallest prediction on a 6-hour grid (mg/L)0.38 mg/L pass0.58 mg/L pass0.57 mg/L pass
Determinismmax difference between repeat calls / batch sizes8.6e-08 pass5.6e-08 pass5.4e-08 pass
One peak per doseshare of patient-intervals with exactly one peak100% pass100% pass100% pass
Dose proportionalitymedian fold deviation of C(2D)/C(D) from 21.105 fail1.029 pass1.040 pass
Grid invariance99th percentile |change in log C| with a 12-hour grid added0.002 pass0.009 pass0.010 pass
Unseen interval (k=0)median fold error 1.224 on q35d vs 1.193 on q56d (ratio of log errors)not run1.146 passnot run
Unseen interval (k=4)median fold error 1.211 on q35d vs 1.188 on q56d (ratio of log errors)not run1.110 passnot run

Training curves

Loss is the negative log-likelihood of the observed log levels plus the KL term, so it can go below zero. Validation error is the mean squared error on log levels from the posterior mean. Hover for the KL weight and learning rate. Training stopped early: clean 148 epochs in 114 s, dose levels + assay error 135 epochs in 105 s, ada 151 epochs in 118 s.

The network

A latent neural ODE with a GRU encoder for the levels seen so far.

  1. The covariates (weight, albumin, INF, PLT) go through a small MLP to a context vector c (8 numbers).
  2. Each patient has a latent vector u (4 numbers), a learned stand-in for the random effects. Its prior depends on c. A GRU reads the first k measured levels and gives the posterior of u. That is the network's version of Bayesian updating; with no levels the posterior is the prior.
  3. A state x (6 numbers) starts at x0(c, u) and follows dx/dt = f(x, u, c), solved with a fixed-step RK4 written in PyTorch.
  4. Each dose is a learned jump in x at the dose time.
  5. A small head maps (x, u) to log concentration, so predictions are always positive.

Trained with Adam (learning rate 0.003, batches of 50), a KL warm-up over 25 epochs, early stopping after 30 epochs without improvement, and dose-rescaling augmentation, which is valid because Drug X has linear PK.

The simulated data

Patient counts per dataset, from the run log.

DatasetTrainValidationTestLevels below LLOQTurned ADA positive
Clean300601200.0%none
Dose levels + assay error300601200.9%none
ADA300601202.6%40%

Clean: one dose level, no assay error, no IOV. Dose levels: several mg/kg dose levels with IOV, assay error and censoring below the LLOQ. ADA: the same, plus anti-drug antibodies that switch on partway through and raise clearance. popPK comparators are two-compartment models fitted by iterative two-stage estimation and forecast by MAP.

popPK estimates

Population estimates from the training patients (CL and Q in L/day, V1 and V2 in L, omegas and sigma as log-scale SDs, covariate effects as power exponents). The true values are in the README.

ParameterpopPK, no covariatespopPK with covariates
CL0.2530.309
V12.5173.223
Q0.3820.485
V21.4211.956
omega_CL0.3720.282
omega_V10.3860.194
sigma0.0560.010
CL~WT0.797
CL~ALB-0.694
CL~INF0.117
CL~PLT0.026
V1~WT0.958
ParameterpopPK, no covariatespopPK with covariates
CL0.2520.297
V12.5563.187
Q0.3730.482
V21.4622.006
omega_CL0.3860.294
omega_V10.3890.196
sigma0.2130.205
CL~WT0.734
CL~ALB-0.816
CL~INF0.114
CL~PLT0.063
V1~WT0.958
ParameterpopPK, no covariatespopPK with covariatespopPK, time-varying CL
CL0.2860.3330.333
V12.4733.1403.140
Q0.4010.5310.531
V21.4592.0262.026
omega_CL0.4290.3450.345
omega_V10.3900.1950.195
sigma0.2900.2860.147
omega_rw0.252
CL~WT0.7190.719
CL~ALB-1.491-1.491
CL~INF0.1020.102
CL~PLT0.0610.061
V1~WT1.0021.002