DPOP London Measles Benchmark

Report generated on: 2026-06-05 00:08:09.668764

This report summarizes a single-unit London measles benchmark for DPOP. The comparison uses the same 100 starting points for all methods, J=5000 particles for search and final pfilter evaluation, and 36 pfilter evaluation replicates.

The benchmark compares:

DPOP uses alpha=0.8, Adam with cosine learning-rate decay, and process weights from the logw state in the 001d London measles model.

Numerical Results

Configuration

Model Warm IF2 DPOP J search J eval Eval reps Adam eta rho eta cosine final
0 IF2-650 650 0 5000 5000 36
1 DPOP-80+80 80 80 5000 5000 36 0.002 0.0005 0.05
2 DPOP-175+175 175 175 5000 5000 36 0.001 0.00025 0.05

Final pfilter summaries

The pre-pruned rows summarize the final pfilter distribution over all 100 search replicates. The post-pruned row is the fresh pfilter evaluation of the single selected replicate.

Model Pre-pruned best Pre-pruned median Pre-pruned mean Post-pruned best n_pre
0 IF2-650 -3798.427 -3802.818 -3804.278 -3799.297 100.000
1 DPOP-80+80 -3798.468 -3805.599 -3809.571 -3798.637 100.000
2 DPOP-175+175 -3796.950 -3802.611 -3805.884 -3797.556 100.000

Runtime

method time Model
0 mif 1020.64 IF2-650
1 pfilter 87.05 IF2-650
2 pfilter 8.12 IF2-650
3 mif 133.61 DPOP-80+80
4 dpop_train 908.38 DPOP-80+80
5 pfilter 87.26 DPOP-80+80
6 pfilter 8.09 DPOP-80+80
7 mif 283.52 DPOP-175+175
8 dpop_train 1975.27 DPOP-175+175
9 pfilter 87.36 DPOP-175+175
10 pfilter 8.15 DPOP-175+175
Model total seconds
0 DPOP-80+80 1137.34
1 DPOP-175+175 2354.29
2 IF2-650 1115.80

Likelihood Distribution

The raincloud plot follows the presentation used for the DMOP Dacca benchmark: the distribution is based on all pre-pruned final pfilter evaluations. The red dotted line marks the best pre-pruned pfilter value observed in this comparison.

Values below -3825.0 are omitted from the raincloud visualization only.

Model omitted
0 DPOP-80+80 5
1 DPOP-175+175 4
2 IF2-650 1

Raincloud plot of final pfilter log-likelihoods.

Optimization Traces

The trace plot uses elapsed time on the x-axis. Each curve is the median log-likelihood over search replicates at that time, and the shaded band runs from the 10th percentile to the maximum. For DPOP, the stored training objective is a negative log-likelihood, so the summary used here is sign-corrected to the log-likelihood direction.

Elapsed-time log-likelihood traces.

Interpretation

The DPOP-80+80 run is close to the runtime of the IF2-650 reference. The DPOP-175+175 run is not runtime matched to IF2-650; it is included as a more conservative DPOP search with a longer IF2 warm start and a smaller learning rate. In these runs, the conservative DPOP schedule gives the best post-pruned pfilter evaluation and a better right tail than the standalone IF2 baseline.

The DPOP training step is currently substantially more expensive than an IF2 iteration for this model. This is consistent with automatic differentiation through the Euler process-ratio correction and the logw process-weight state. Future benchmarks should separate algorithmic performance from implementation cost by including a per-iteration runtime and memory profile.

Process-score coverage

The current 001d London measles implementation should be interpreted as a benchmark of the DPOP training machinery rather than a final accounting of every process-density contribution in the model. The logw state used by dpop_train contains the Euler multinomial transition contribution along the simulated epidemic path. This is the main DPOP-specific process-ratio term and is the part that distinguishes these runs from the measurement-only IFAD/DMOP style correction.

There are several model components whose process-score contribution is not yet fully represented in this benchmark. The Poisson birth mechanism is tied to cohort, so an incomplete birth log probability means the DPOP process score for cohort is only partial. The environmental gamma noise affects sigmaSE, and its density contribution is also not fully represented in the current logw accumulator. Finally, the initial-value parameters S_0, E_0, I_0, and R_0 are sampled through the initial-state construction, but this benchmark does not include an explicit initial-state density or initial-state ratio in the DPOP objective.

For this reason, the likelihood comparisons above are valid comparisons of the completed runs, but parameter-level interpretation for cohort, sigmaSE, and the initial-value parameters should be conservative. A follow-up benchmark should add the missing birth, environmental-noise, and initial-state contributions to logw, then repeat the same IF2/DPOP comparison to isolate the effect of a fully specified process score.

Source Files

The experiment scripts in this directory can be submitted with scripts/run_tests.py from the root of the quant repository. The stored CSV files in plot_exports/ are copied from the completed level-4 Great Lakes runs and allow this report to render without the large pickled result objects.