Show Code
import pandas as pd
import report_utils as ru
from plotnine import (
aes,
coord_flip,
geom_point,
ggplot,
guide_legend,
guides,
labs,
position_dodge,
scale_y_log10,
)
ru.nav_bar("timing")Runtime & Performance Benchmark
Execution speed and throughput: R pomp vs. pypomp, all models
Introduction
This report compares the runtime of IF2 and the particle filter (pfilter) in pypomp (GPU, and CPU where run) and R pomp (CPU) on identical work. For panel measles, R uses panelPomp, and parameter estimation uses marginalized panel iterated filtering (MPIF) in place of IF2.
| Model | Workload |
|---|---|
| SPX | The stochastic volatility model of Sun (2024). Draws few random variates and takes a single rproc step per observation. |
| Dhaka cholera | A lightweight state transition run for 20 Euler steps per observation. Both implementations start from the same parameter vectors (tests/dacca/starting_parameters.csv). |
| Measles | London: 365 Euler steps per year across 14 years of weekly data, drawing 8 random variates per step (1 Poisson, 1 gamma, 6 binomial). Also compares the fast samplers in pypomp.random with stock jax.random samplers on the GPU. |
| Panel measles | London, Halesworth, Hastings and Cardiff, with shared and unit-specific parameters. |
Speedup and throughput use the cold-start particle filter (pfilter_cold), which includes JIT compilation, since that cost is paid in practice. Throughput is relative to a single R CPU core (\(\text{speedup} \times N_{\text{cores}}\)).
Show Code
MODELS = ru.models("timing")
runs = {m: ru.load_runs(m, "timing") for m in MODELS}
tables = {
m: ru.build_timing_comparison_df(runs[m], is_panel=ru.MODELS[m]["panel"]) for m in MODELS
}Show Code
ru.show_missing(runs)Visual review
- Throughput and results by model:
- SPX:
pypomp’s throughput on GPU should be hundreds of times that of Rpompon one core, andpypompon CPU should run at least twice as fast as Rpompon CPU. - Dhaka cholera:
pypomp’s throughput on GPU should be thousands of times that of Rpompon one core, and the total speedup ofpypompon CPU relative to Rpompshould be about 4x. - Measles:
pypomp’s throughput on GPU with the fast samplers should be thousands of times that of Rpompon one core, and the fast-sampler GPU run should be substantially faster than the stock JAX sampler GPU run. We set no expectation for the CPU run. - Panel measles:
pypomp’s throughput on GPU should be thousands of times that of RpanelPompon one core.
- SPX:
Throughput
Show Code
CONFIG = {
"R": "R",
"gpu": "pypomp (GPU)",
"gpu_jax": "pypomp (GPU, JAX samplers)",
"cpu": "pypomp (CPU, 36 cores)",
}
frames = []
for m, df in tables.items():
arms = ru.MODELS[m]["timing"]["arms"]
frames.append(
pd.DataFrame(
{
"Model": ru.MODELS[m]["label"],
"Configuration": df["Configuration"].map(lambda c: CONFIG[arms[c]]),
"throughput": pd.to_numeric(
df["Throughput (vs 1 R CPU core)"].str.rstrip("x"), errors="coerce"
),
}
)
)
tp = pd.concat(frames, ignore_index=True).dropna()
tp["Model"] = pd.Categorical(tp["Model"], categories=tp["Model"].unique()[::-1])
tp["Configuration"] = pd.Categorical(tp["Configuration"], categories=list(CONFIG.values()))
(
ggplot(tp, aes(x="Model", y="throughput", color="Configuration"))
+ geom_point(size=4, position=position_dodge(width=0.5))
+ scale_y_log10()
+ coord_flip()
+ labs(
title="Throughput relative to one R CPU core",
subtitle="Cold-start particle filter; log scale",
x="",
y="Throughput (× one R core)",
color="Configuration",
)
+ guides(color=guide_legend(nrow=2))
+ ru.scale_color_premium()
+ ru.theme_premium
)
Results by model
The first pfilter call in JAX includes one-time JIT compilation. Later calls run without recompiling, as the compilation overhead tables show.
Show Code
for m in MODELS:
print(ru.heading(m))
print(ru.raw_html(tables[m].to_html(classes="table table-striped table-hover", index=False)))
print("\n**JIT compilation overhead**\n")
cold_warm = ru.build_cold_vs_warm_df(runs[m])
print(ru.raw_html(cold_warm.to_html(classes="table table-striped table-hover", index=False)))SPX
| Configuration | IF2 (s) | IF2 Speedup | Pfilter (s) | Pfilter Speedup | Total (s) | Total Speedup | Throughput (vs 1 R CPU core) |
|---|---|---|---|---|---|---|---|
| R pomp (36 cores) | 702.5s (11.71m) | 1.00x | 36.4s (0.61m) | 1.00x | 738.9s (12.32m) | 1.00x | 36.00x |
| pypomp (GPU) | 34.8s (0.58m) | 20.19x | 5.2s (0.09m) | 6.96x | 40.0s (0.67m) | 18.46x | 664.53x |
| pypomp (CPU, 36 cores) | 275.1s (4.58m) | 2.55x | 7.4s (0.12m) | 4.92x | 282.5s (4.71m) | 2.62x | 94.17x |
JIT compilation overhead
| Configuration | Pfilter Cold (s) | Pfilter Warm (s) | Compilation Overhead (s) |
|---|---|---|---|
| R pomp (36 cores) | 36.38s | 36.38s | 0.00s |
| pypomp (GPU) | 5.23s | 3.64s | 1.58s |
| pypomp (CPU, 36 cores) | 7.39s | 6.00s | 1.39s |
Dhaka cholera
| Configuration | IF2 (s) | IF2 Speedup | Pfilter (s) | Pfilter Speedup | Total (s) | Total Speedup | Throughput (vs 1 R CPU core) |
|---|---|---|---|---|---|---|---|
| R pomp (36 cores) | 1608.7s (26.81m) | 1.00x | 379.5s (6.33m) | 1.00x | 1988.2s (33.14m) | 1.00x | 36.00x |
| pypomp (GPU) | 28.3s (0.47m) | 56.79x | 8.2s (0.14m) | 46.39x | 36.5s (0.61m) | 54.46x | 1960.59x |
| pypomp (CPU, 36 cores) | 389.8s (6.50m) | 4.13x | 67.8s (1.13m) | 5.60x | 457.6s (7.63m) | 4.34x | 156.41x |
JIT compilation overhead
| Configuration | Pfilter Cold (s) | Pfilter Warm (s) | Compilation Overhead (s) |
|---|---|---|---|
| R pomp (36 cores) | 379.50s | 379.50s | 0.00s |
| pypomp (GPU) | 8.18s | 6.17s | 2.01s |
| pypomp (CPU, 36 cores) | 67.81s | 69.47s | — |
Measles
| Configuration | IF2 (s) | IF2 Speedup | Pfilter (s) | Pfilter Speedup | Total (s) | Total Speedup | Throughput (vs 1 R CPU core) |
|---|---|---|---|---|---|---|---|
| R pomp (36 cores) | 2922.6s (48.71m) | 1.00x | 895.0s (14.92m) | 1.00x | 3817.6s (63.63m) | 1.00x | 36.00x |
| pypomp (GPU) | 32.2s (0.54m) | 90.67x | 14.5s (0.24m) | 61.75x | 46.7s (0.78m) | 81.70x | 2941.12x |
| pypomp (GPU, JAX samplers) | 1469.8s (24.50m) | 1.99x | 526.4s (8.77m) | 1.70x | 1996.2s (33.27m) | 1.91x | 68.85x |
| pypomp (CPU, 36 cores) | 1070.7s (17.84m) | 2.73x | 302.2s (5.04m) | 2.96x | 1372.9s (22.88m) | 2.78x | 100.11x |
JIT compilation overhead
| Configuration | Pfilter Cold (s) | Pfilter Warm (s) | Compilation Overhead (s) |
|---|---|---|---|
| R pomp (36 cores) | 895.03s | 895.03s | 0.00s |
| pypomp (GPU) | 14.49s | 8.43s | 6.07s |
| pypomp (GPU, JAX samplers) | 526.37s | 519.31s | 7.06s |
| pypomp (CPU, 36 cores) | 302.20s | 297.80s | 4.40s |
Panel measles
| Configuration | MPIF (s) | MPIF Speedup | Pfilter (s) | Pfilter Speedup | Total (s) | Total Speedup | Throughput (vs 1 R CPU core) |
|---|---|---|---|---|---|---|---|
| R panelPomp (36 cores) | 8261.9s (137.70m) | 1.00x | 2251.6s (37.53m) | 1.00x | 10513.5s (175.23m) | 1.00x | 36.00x |
| pypomp (GPU) | 98.2s (1.64m) | 84.17x | 33.7s (0.56m) | 66.79x | 131.9s (2.20m) | 79.73x | 2870.25x |
JIT compilation overhead
| Configuration | Pfilter Cold (s) | Pfilter Warm (s) | Compilation Overhead (s) |
|---|---|---|---|
| R panelPomp (36 cores) | 2251.64s | 2251.64s | 0.00s |
| pypomp (GPU) | 33.71s | 25.02s | 8.69s |
Appendix: settings and environment
Show Code
print(ru.settings_sections(runs))SPX
| Setting / Parameter | R pomp (36 cores) | pypomp (GPU) | pypomp (CPU, 36 cores) |
|---|---|---|---|
| Algorithmic & Workload Settings | |||
| Run Level | 4 | 4 | 4 |
| Starting Searches ($N_{starts}$) | 36 | 36 | 36 |
| IF2 Iterations ($N_{iter}$) | 200 | 200 | 200 |
| IF2 Particles ($N_p$) | 1,000 | 1,000 | 1,000 |
| Pfilter Particles ($N_{p,eval}$) | 1,000 | 1,000 | 1,000 |
| Evaluation Replicates ($N_{reps}$) | 24 | 24 | 24 |
| Random Seed | — | 631409 | 631409 |
| Software & Environment | |||
| Pomp Framework | pomp 6.3 | pypomp 1.0.5 | pypomp 1.0.5 |
| Backend / Engine | R 4.4.0 | JAX 0.11.2 | JAX 0.11.2 |
| Quant Git Commit | 3646a0c |
35b5095 |
35b5095 |
| Run Timestamp | 2026-08-06 16:10:58 | 2026-09-20 16:34:02 | 2026-09-20 16:37:28 |
| Hardware & Compute | |||
| Compute Device | Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz (36 cores) | NVIDIA RTX PRO 6000 Blackwell Server Edition (1 GPU) | Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz (36 cores) |
| Slurm Partition | standard | gpu-rtx6000 | standard |
| Slurm Job ID | 56616028 | 61595110 | 61595111 |
Dhaka cholera
| Setting / Parameter | R pomp (36 cores) | pypomp (GPU) | pypomp (CPU, 36 cores) |
|---|---|---|---|
| Algorithmic & Workload Settings | |||
| Run Level | 4 | 4 | 4 |
| Starting Searches ($N_{starts}$) | 36 | 36 | 36 |
| IF2 Iterations ($N_{iter}$) | 100 | 100 | 100 |
| IF2 Particles ($N_p$) | 5,000 | 5,000 | 5,000 |
| Pfilter Particles ($N_{p,eval}$) | 5,000 | 5,000 | 5,000 |
| Evaluation Replicates ($N_{reps}$) | 36 | 36 | 36 |
| Random Seed | — | 631409 | 631409 |
| Software & Environment | |||
| Pomp Framework | pomp 6.3 | pypomp 1.0.8 | pypomp 1.0.8 |
| Backend / Engine | R 4.4.0 | JAX 0.11.2 | JAX 0.11.2 |
| Quant Git Commit | 9deadf7 |
c0817f5 |
c0817f5 |
| Run Timestamp | 2026-10-07 15:17:07 | 2026-10-07 17:55:58 | 2026-10-07 17:14:08 |
| Hardware & Compute | |||
| Compute Device | Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz (36 cores) | NVIDIA RTX PRO 6000 Blackwell Server Edition (1 GPU) | Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz (36 cores) |
| Slurm Partition | standard | gpu-rtx6000 | standard |
| Slurm Job ID | 63438191 | 63429689 | 63429690 |
Measles
| Setting / Parameter | R pomp (36 cores) | pypomp (GPU) | pypomp (GPU, JAX samplers) | pypomp (CPU, 36 cores) |
|---|---|---|---|---|
| Algorithmic & Workload Settings | ||||
| Run Level | 4 | 4 | 4 | 4 |
| Starting Searches ($N_{starts}$) | 36 | 36 | 36 | 36 |
| IF2 Iterations ($N_{iter}$) | 100 | 100 | 100 | 100 |
| IF2 Particles ($N_p$) | 5,000 | 5,000 | 5,000 | 5,000 |
| Pfilter Particles ($N_{p,eval}$) | 5,000 | 5,000 | 5,000 | 5,000 |
| Evaluation Replicates ($N_{reps}$) | 36 | 36 | 36 | 36 |
| Unit(s) | London | London | London | London |
| Sampler Implementation | Compiled C snippets | fast (pypomp.random) | stock JAX (jax.random) | fast (pypomp.random) |
| Random Seed | — | 594709947 | 594709947 | 594709947 |
| Software & Environment | ||||
| Pomp Framework | pomp 6.3 | pypomp 1.0.6 | pypomp 1.0.6 | pypomp 1.0.6 |
| Backend / Engine | R 4.4.0 | JAX 0.11.2 | JAX 0.11.2 | JAX 0.11.2 |
| Quant Git Commit | ea2b3d1 |
9c649ea |
9c649ea |
9c649ea |
| Run Timestamp | 2026-08-11 12:22:19 | 2026-09-24 21:29:04 | 2026-09-24 22:11:43 | 2026-09-24 20:42:37 |
| Hardware & Compute | ||||
| Compute Device | Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz (36 cores) | NVIDIA RTX PRO 6000 Blackwell Server Edition (1 GPU) | NVIDIA RTX PRO 6000 Blackwell Server Edition (1 GPU) | Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz (36 cores) |
| Slurm Partition | standard | gpu-rtx6000 | gpu-rtx6000 | standard |
| Slurm Job ID | 56947815 | 61826881 | 61826882 | 61826883 |
Panel measles
| Setting / Parameter | R panelPomp (36 cores) | pypomp (GPU) |
|---|---|---|
| Algorithmic & Workload Settings | ||
| Run Level | 4 | 4 |
| Starting Searches ($N_{starts}$) | 36 | 36 |
| MPIF Iterations ($N_{iter}$) | 100 | 100 |
| MPIF Particles / Unit ($N_{p,fit}$) | 5,000 | 5,000 |
| Pfilter Particles / Unit ($N_{p,eval}$) | 5,000 | 5,000 |
| Evaluation Replicates ($N_{reps}$) | 36 | 36 |
| Unit(s) | London, Halesworth, Hastings, Cardiff (4 units) | London, Halesworth, Hastings, Cardiff (4 units) |
| Shared Parameters | — | R0, sigma, gamma, sigmaSE, cohort, amplitude (6 params) |
| Random Seed | — | 594709947 |
| Software & Environment | ||
| Pomp Framework | panelPomp 1.7.0.0 (pomp 6.3) | pypomp 1.0.6 |
| Backend / Engine | R 4.4.0 | JAX 0.11.2 |
| Quant Git Commit | c6ac80c |
9c649ea |
| Run Timestamp | 2026-10-07 16:01:18 | 2026-09-24 22:26:34 |
| Hardware & Compute | ||
| Compute Device | Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz (36 cores) | NVIDIA RTX PRO 6000 Blackwell Server Edition (1 GPU) |
| Slurm Partition | standard | gpu-rtx6000 |
| Slurm Job ID | 63429730 | 61839615 |
References
Sun, Weizhe. 2024. “Model Based Inference of Stochastic Volatility via Iterated Filtering.” Undergraduate Honors Thesis, University of Michigan.