Performance Benchmarks

Last updated: 2026-07-14 Primary benchmark build: main at commit f7af461. Measured over HTTPS (on by default) with an authenticated session, so server times include the TLS handshake and the session check — a few ms above the earlier plain-HTTP figures. USB figures (page loads, CPU, stress) were re-measured on this build, including the stress tables (the load harness now authenticates). NFS figures are carried from the 1.0.0/0.9.0 rounds and marked where used.

Replay Control is designed to run quietly on a Raspberry Pi while still handling large game libraries. The practical result from the 1.0.0 measurements is:

• Normal browsing is fast. Home renders in about 9 ms of server time over HTTPS (TLS + session check included), system and game pages in 5-6 ms, and common searches in about 35 ms (USB) on a Pi 5.
• Idle CPU is negligible: effectively 0% of one core with no requests, and under 1% (USB) while one person browses.
• Idle memory settles to about 99 MB resident (41 MB heap) within ~3 minutes of startup. It is modestly higher than earlier releases, mostly because the on-device game catalog is memory-mapped for fast lookups — shared, file-backed cache the kernel reclaims under pressure, not heap. See Regressions and Analysis.
• Heavy artificial load peaks variably (around 0.7-1.1 GB across runs, as large result sets are built in memory) and the service stays responsive; it settles back to roughly 190-250 MB, though the allocator retains the grown heap rather than returning fully to idle.
• Search cost scales with the number of games, not with where they are stored. The larger (NFS-sourced) test library makes broad search heavier purely because it has more rows. The library database that search reads is on the Pi’s local disk regardless of storage mode; the user database (favorites, recents, custom box art) does live on the ROM storage, but search does not read it.

Two libraries were measured on the same Pi 5: a USB library with 26,777 games and a larger NFS development library with 103,896 games. Page rendering and memory are nearly identical between the two because the catalog and library databases live on the Pi’s local disk; storage mode mostly affects scanning and search-result volume.

Tested Hardware#

Everyday Use#

CPU Use#

CPU is reported as percent of one core. On a Pi 5, 100% means one core is fully busy; 400% would be the whole Pi.

The larger NFS library costs more CPU per request because each search and listing touches more rows.

Memory Use#

Memory is the resident set (VmRSS) for the replay-control process; “heap” is the anonymous portion (RssAnon). The rest is shared, file-backed cache (mostly the memory-mapped catalog), which the kernel reclaims under pressure. Memory is essentially storage-independent — the catalog and library databases sit on the Pi’s local disk regardless of storage mode — so these were measured on the larger NFS library (the worst case for heap and load peaks). Each state is sampled right after the event, then 1, 3, and 5 minutes later; “Peak” is Linux’s high-water mark (VmHWM) since process start.

Idle settles to about 99 MB resident (41 MB heap) within ~3 minutes; the 1-minute bump is startup verification touching catalog pages, which the kernel then reclaims. Heavy broad-search stress peaks variably — between about 730 MB and 1.1 GB across runs — as large result sets are built in memory, then settles to roughly 190-250 MB; the allocator retains the grown heap (~130-190 MB) rather than returning to the idle 41 MB. The peak is run-dependent because it tracks how many concurrent broad searches overlap, which is exactly what the broad-search cap in Regressions and Analysis would bound. The 2 GB Pi stayed up with headroom throughout (≥1.5 GB reported available).

Startup#

A warm restart on an unchanged library settles in a few seconds (the per-system stats stay fresh, so only verification runs). A cold storage switch triggers a rescan: switching to the 26,777-game USB library and reaching “all systems fresh” took under a minute this round. First scans and full rebuilds of large libraries do much more work and are covered in Library Maintenance on NFS.

Page Load Times#

Warm, single-user requests on Pi 5. “Server time” is time-to-first-byte (SSR processing); “Total” includes transfer.

NFS page/search figures are from the 1.0.0 round (2026-06-28).

Download Sizes#

The web app’s static files. WASM is served gzip-compressed by the server.

The WASM bundle includes the wasm-opt -Oz pass. Browsers that accept br download 948 KB — about 498 KB (34%) smaller than gzip; gzip remains the fallback for clients or builds without brotli.

A newer web-framework release (leptos 0.8) was evaluated in 2026-07 but not adopted: it grew the compressed WASM bundle by about a third and intermittently hung page responses under concurrent request load, so the framework stays on 0.7 until a release without those issues.

Stress Tests#

These numbers come from Apache Bench (ab) issuing 50 requests per endpoint at increasing concurrency. This is intentionally heavier than normal use; it is a regression and robustness check. ab’s “Failed” column counts response-size variance (search results differ run to run), not HTTP errors — all requests returned successfully.

USB Library (26,777 games)#

Homepage#

Search#

System and Game Pages#

The c=1 light-page throughput is roughly half the 1.0.0 numbers; the cost is the richer per-game data added since then (now-playing, achievements, series, more metadata per row), measured to be feature work rather than any single regression.

Mixed Concurrent Test#

Four endpoints hit at the same time, each at concurrency 5.

NFS Library (103,896 games)#

Search is much heavier here because the NFS development library is roughly four times larger.

Homepage#

Search#

System and Game Pages#

Mixed Concurrent Test#

Four endpoints hit at the same time, each at concurrency 5.

Library Maintenance on NFS#

NFS is a harder workload because scans must walk a remote ROM tree and rebuilds may stream large files to recompute CRCs. Current builds keep the catalog, library database, and external-metadata database on the Pi, so normal page rendering is much less tied to NFS latency than library maintenance is.

The following are earlier (0.4.0-era) NFS library-maintenance measurements; they were not re-timed in the 0.9.0 round and are kept here as directional references.

Maintenance on a 95,495-ROM development library:

Deferred-identity pipeline on a 99,964-ROM NFS library:

The key result is responsiveness: the foreground library becomes available before the hash tail finishes, and the app stays usable during the remaining NFS reads.

Historical Comparison#

0.9.0 keeps normal browsing in the same fast range as 0.4.0 while the library has grown and gained much richer per-game metadata (achievements, resources, series, expanded catalog). The two visible costs are a modest rise in idle memory (the memory-mapped catalog, plus heap the allocator retains after heavy load) and heavier worst-case search on very large libraries.

* The 0.4.0 “a” figure is from its NFS section (102 K library); the 0.4.0 USB “a” was 158 ms.

Regressions and Analysis#

Two metrics moved the wrong way versus 0.4.0. Both are understood; neither is a correctness bug.

Higher memory use (idle ≈55 MB → ≈99 MB; heap retained after load)#

What it is. Settled idle memory is about 99 MB resident (41 MB heap) — up from 0.4.0’s 45-61 MB, but far less than an early mis-measurement suggested. Resident memory taken right after startup is lower (~60 MB) and briefly bumps to ~113 MB during startup verification before the kernel reclaims it. The more durable effect is after heavy load: resident settles to roughly 190-250 MB and the allocator holds onto ~130-190 MB of heap rather than returning to the idle 41 MB. The transient peak during broad-search stress is run-dependent and ranged ~0.7-1.1 GB across repeats.

Cause. Two things. First, the on-device game catalog is memory-mapped for fast lookups (about 64 MB of mapping against a ~62 MB catalog) and the catalog has grown; those pages are shared, file-backed, and reclaimable, not heap. Second, the allocator (jemalloc) retains freed pages after a memory spike (broad-search stress) instead of returning them to the OS, so heap stays elevated until the next spike reuses it. Available system memory stayed around 1.6 GB throughout on the 2 GB Pi.

Options.

• Accept it — idle stays ~99 MB, well within the 2 GB budget and mostly reclaimable cache.
• Apply the already-investigated allocator tuning (a MALLOC_CONF setting) to return freed pages sooner; it shrinks the retained heap after load at no measured throughput cost.
• Reduce the catalog memory-map size (REPLAY_CATALOG_MMAP_MB, default 64). Measured on the device: capping it at 16 MB cut process resident memory by ~19 MB with no change to warm page times — but it did not free system RAM. MemAvailable was unchanged because the catalog pages simply relocate from the process’s mmap into the kernel page cache. So this lowers the reported process RSS for monitoring, not actual memory pressure. (A value of 0 is ignored — the env reader treats it as unset — so use a small positive value to reduce, not disable.)

Heavier worst-case search on very large libraries#

What it is. Common multi-word searches are on par with 0.4.0, but the pathological single-letter query (“a”) is about 1.7× slower on the larger library (c=1 800 ms → 1,357 ms). The same query on the smaller library is far cheaper (304 ms), and longer queries (e.g. “street fighter”) actually improved at c=1.

This is about library size, not storage. The catalog, library, and metadata databases live on the Pi’s local disk for every storage mode; only the user database (favorites, recents, custom box art) lives on the ROM storage. Search reads just the local library database — not the user database — so it never waits on NFS. Custom box art is a good example: overrides are kept in the user database but are folded into the local library rows during enrichment, so they are not looked up per search result. The USB-vs-NFS labels here are therefore just “small library” vs “large library”; the protocol is irrelevant to search, and the cost tracks the number of matched rows. (User-data-backed features such as favorites, recents, and saving resources are the ones that do feel storage latency, since they read the user database on the ROM share.)

Cause. Text search matches a substring anywhere in each game’s searchable text, then loads every match into memory and ranks it before paginating — there is no early cap. Measured on the device, the database scan itself is cheap (~46 ms even for a query that matches 99,943 of 103,896 rows); the expense is materializing and ranking ~100k full rows in Rust, amplified by how much metadata each result now carries (which has grown since 0.4.0). The ~4× small-vs-large gap (26,777 vs 103,896 games) confirms row count is the dominant factor, not an algorithmic regression.

Options.

• Require a minimum query length — ignore free-text queries shorter than 2 characters after trimming whitespace. This removes the single-character worst case outright and matches common search-box behavior. Keep the threshold at two, not three: two-character board shorthands like “f3” (Taito F3) and “cps” must keep working. Those already resolve through the separate arcade-board recognizer (a board-tag lookup, independent of free-text search), so they return results regardless of this guard or the text index. Cheapest guard, but it only eliminates 1-character queries; 2+ character broad free-text queries still scan and materialize.
• Cap the candidate set with a generous in-database limit before ranking, so even a broad query never loads the entire library into memory. This also lowers the broad-search load-test memory peak (observed ~0.7-1.1 GB), which comes from the same unbounded materialization.
• Add a full-text index for the searchable text. A leading-wildcard substring match cannot use a normal index, so today every text search is a full scan. Measured on the device, an FTS5 index over the search column costs about 6-9 s to build at library-build time and ~20 MB on disk (trigram, which supports substring matching), and turns the per-query lookup from ~46-67 ms into well under 1 ms while returning only matching rows to rank. Trigram matches need ≥3 characters, so two-character free-text queries would keep a LIKE fallback (rare and cheap — few rows match), and board shorthands stay on the recognizer path. It pairs naturally with the minimum-length guard above. This is the deeper fix.

Test Methodology#

• CPU: tools/pi-cpu.sh reads /proc/<pid>/stat and reports CPU relative to one core. --browse simulates one user clicking through home, system pages, search, game detail, and manuals every ~2 s.
• Memory: tools/pi-memory.sh reads /proc/<PID>/status: VmRSS (memory in use), VmHWM (peak since process start), and RssAnon (heap).
• Page and asset benchmarks: tools/bench.sh, with Lighthouse skipped for the recorded release runs.
• Stress tests: tools/load-test.sh, which uses Apache Bench (ab) with 50 requests per endpoint.
• Storage switching: the USB and NFS rounds were measured on the same Pi by switching the active storage and rebooting, then waiting for the rescan to settle (all systems reporting fresh stats) before measuring.
• All measurements were taken on a Pi 5 (4 cores, 2 GB), no game running, default allocator configuration. USB figures: main at commit f7af461 (2026-07-15). NFS figures: the 1.0.0 (433e899) and 0.9.0 (ac518f9) rounds.
• Raw results are stored in tools/bench-results/.