The benchmark has no control, and the load generator has never been checked against another one #116

New issue

Closed

opened 2026-08-13 22:59:44 +02:00 by Claude · 1 comment

Claude commented

2026-08-13 22:59:44 +02:00

Collaborator

Both are named in report.md's own "what is not established", and item 1 of its suggested order is the control — the largest remaining hole in the method rather than in the server.

No control request. Nothing in the timed set does provably identical work on both sides. A password hash at login would: same algorithm, same cost parameters, same input, so a large gap there would indict the measurement rather than the server. The login currently happens once at connect, outside the timed set. Until a control exists, the 5.5–7× and the 43.9 ms against 273.1 ms rest on internal consistency, which is good evidence and is not a control.

The generator has not been cross-checked. It is verified against a test server that returns a fixed response after a fixed delay, so what it ought to report is arithmetic — but it has never been run beside an established generator over the same target. The scheduling property it claims (a request's latency measured from when it was due, not when it was sent) is exactly the property that is easy to get subtly wrong and impossible to notice from the inside.

Neither blocks using the numbers internally. Both block publishing them.

Where to start

crates/jellyfin-bench/ is the generator (load.rs, gate.rs, constant.rs,
stats.rs, report.rs) and crates/jellyfin-bench/README.md says why it can
be believed. docs/benchmarks.md is how a comparison is set up; report.md is
the run these two gaps come out of, and its "what is not established" section is
the specification for this issue.

The control. Something in the timed set that does provably identical work on
both sides. A password hash at login is the candidate named in the report: same
algorithm, same cost parameters, same input, so a large gap there indicts the
measurement rather than the server. It happens once at connect today, outside
the timed set. Getting it into the set is the work — and note the harness's
own rule applies, that a request both servers answer identically for the wrong
reason is a false pass.

The cross-check. Run an established generator (oha, wrk2, k6 — any one
that also schedules by due time rather than send time) against the same target
and compare. The scheduling property is the one that is easy to get subtly wrong
and impossible to see from the inside.

How to know it is done

The control appears in the reported table with a ratio near 1, and a second
generator's numbers for the same target sit inside the spread the report already
records for five repetitions. If either does not, that result is the finding and
it goes in report.md.

House rules

CLAUDE.md is the short version and these are the parts that bite: tests run
under cargo nextest; upstream's source is the checkout in upstream/ at the
pin (v12.0-rc5, 6d501ba4) and not the clone beside the repository;
generated files are regenerated rather than edited; a comment says what the code
does, not how it got there; and Dn references stay out of code and comments.
ARCHITECTURE.md decides which crate a thing goes in and which dependency edges
are illegal, DECISIONS.md says why a rule exists.

The reference pair is cargo xtask reference up (and reference provision,
which is idempotent); a run is
cargo run -p jellyfin-harness -- replay --upstream URL --ours URL --scenario FILE --out DIR,
with diff-db for the two databases. crates/jellyfin-harness/README.md says
what a diff means and what an ignore entry has to carry — every entry needs a
reason and an expiry milestone, and an adds-only history is the smell the risk
table names.

Both are named in `report.md`'s own "what is not established", and item 1 of its suggested order is the control — the largest remaining hole in the method rather than in the server. **No control request.** Nothing in the timed set does provably identical work on both sides. A password hash at login would: same algorithm, same cost parameters, same input, so a large gap there would indict the measurement rather than the server. The login currently happens once at connect, outside the timed set. Until a control exists, the 5.5–7× and the 43.9 ms against 273.1 ms rest on internal consistency, which is good evidence and is not a control. **The generator has not been cross-checked.** It is verified against a test server that returns a fixed response after a fixed delay, so what it ought to report is arithmetic — but it has never been run beside an established generator over the same target. The scheduling property it claims (a request's latency measured from when it was *due*, not when it was sent) is exactly the property that is easy to get subtly wrong and impossible to notice from the inside. Neither blocks using the numbers internally. Both block publishing them. --- ## Where to start `crates/jellyfin-bench/` is the generator (`load.rs`, `gate.rs`, `constant.rs`, `stats.rs`, `report.rs`) and `crates/jellyfin-bench/README.md` says why it can be believed. `docs/benchmarks.md` is how a comparison is set up; `report.md` is the run these two gaps come out of, and its "what is not established" section is the specification for this issue. **The control.** Something in the timed set that does provably identical work on both sides. A password hash at login is the candidate named in the report: same algorithm, same cost parameters, same input, so a large gap there indicts the measurement rather than the server. It happens once at connect today, outside the timed set. Getting it *into* the set is the work — and note the harness's own rule applies, that a request both servers answer identically for the wrong reason is a false pass. **The cross-check.** Run an established generator (`oha`, `wrk2`, `k6` — any one that also schedules by due time rather than send time) against the same target and compare. The scheduling property is the one that is easy to get subtly wrong and impossible to see from the inside. ## How to know it is done The control appears in the reported table with a ratio near 1, and a second generator's numbers for the same target sit inside the spread the report already records for five repetitions. If either does not, that result is the finding and it goes in `report.md`. ### House rules `CLAUDE.md` is the short version and these are the parts that bite: tests run under `cargo nextest`; upstream's source is the checkout in `upstream/` **at the pin** (`v12.0-rc5`, `6d501ba4`) and not the clone beside the repository; generated files are regenerated rather than edited; a comment says what the code does, not how it got there; and `Dn` references stay out of code and comments. `ARCHITECTURE.md` decides which crate a thing goes in and which dependency edges are illegal, `DECISIONS.md` says why a rule exists. The reference pair is `cargo xtask reference up` (and `reference provision`, which is idempotent); a run is `cargo run -p jellyfin-harness -- replay --upstream URL --ours URL --scenario FILE --out DIR`, with `diff-db` for the two databases. `crates/jellyfin-harness/README.md` says what a diff means and what an ignore entry has to carry — every entry needs a reason and an expiry milestone, and an adds-only history is the smell the risk table names.

Claude added the

performance

testing

labels

2026-08-13 23:07:40 +02:00

Claude referenced this issue

2026-08-21 12:28:46 +02:00

The published ffmpeg cannot run on the host, so no benchmark can start #173

Claude commented

2026-08-21 14:18:38 +02:00

Author

Collaborator

Landed in 94e9276.

The control. login is POST /Users/AuthenticateByName — PBKDF2-SHA512 at
the stored hash's iteration count against the same stored hash. Against the
default snapshot at concurrency 1 it reports 68.30 ms upstream against
71.95 ms ours: a ratio of 1.05, with p90 at 1.02, p99 at 0.94 and throughput
at 0.95. So the gap on the read paths is the server rather than the instrument.

It has no committed scenario. An anchor carries the accounts a household had,
and a scenario naming jellyfin would be refused by both servers and agree
about the refusal — a control that measures a 401. The run writes it from its
own --user/--password beside the results, so what was measured is on record
next to the numbers. authenticated: true is on the request not for a token but
because upstream refuses the route without the client and device that header
carries. It writes a device row per request, so it is a write workload.

One ignore-list entry was needed: AccessToken, which each server draws for
itself. Everything else the route answers is diffed — the account, its policy
and configuration, and the session the login opened.

The cross-check. oha 1.15, over ephyra-bench constant --service-ms 20 --body 512 --serialized, offered 100/s for 10s:

	ours	oha
throughput	47.39/s	47.41/s
median from due time	5.558 s	5.556 s
worst	11.00 s (p99)	11.10 s (max)
requests completed	1000	1000
wall clock	21.1 s	21.1 s

Closed loop at one connection: 47.2/s and 21.20 ms against 47.4/s and 21.13 ms.

oha has to be asked for the same run: --latency-correction for the due-time
clock, -w to drain rather than truncate the tail at the deadline, and enough
connections that the rate is never held back by the generator. Ask for less and
the disagreement is the flag rather than the finding — -c 200 without -w
reports a worst case of 5.27 s for the same run, which is the tail it did not
wait for. The flags and the numbers are in crates/ephyra-bench/README.md.

Two small changes made the target reachable by the whole timed path rather than
by load alone: constant answers a JSON string of the length it was asked
for, and run takes an empty --user for a target with no login. The gate also
lists what it dropped before deciding there is nothing to measure — it used to
report a count and nothing to act on.

Landed in 94e9276. **The control.** `login` is `POST /Users/AuthenticateByName` — PBKDF2-SHA512 at the stored hash's iteration count against the same stored hash. Against the `default` snapshot at concurrency 1 it reports 68.30 ms upstream against 71.95 ms ours: a ratio of **1.05**, with p90 at 1.02, p99 at 0.94 and throughput at 0.95. So the gap on the read paths is the server rather than the instrument. It has no committed scenario. An anchor carries the accounts a household had, and a scenario naming `jellyfin` would be refused by both servers and agree about the refusal — a control that measures a 401. The run writes it from its own `--user`/`--password` beside the results, so what was measured is on record next to the numbers. `authenticated: true` is on the request not for a token but because upstream refuses the route without the client and device that header carries. It writes a device row per request, so it is a write workload. One ignore-list entry was needed: `AccessToken`, which each server draws for itself. Everything else the route answers is diffed — the account, its policy and configuration, and the session the login opened. **The cross-check.** `oha` 1.15, over `ephyra-bench constant --service-ms 20 --body 512 --serialized`, offered 100/s for 10s: | | ours | oha | |---|---|---| | throughput | 47.39/s | 47.41/s | | median from due time | 5.558 s | 5.556 s | | worst | 11.00 s (p99) | 11.10 s (max) | | requests completed | 1000 | 1000 | | wall clock | 21.1 s | 21.1 s | Closed loop at one connection: 47.2/s and 21.20 ms against 47.4/s and 21.13 ms. oha has to be asked for the same run: `--latency-correction` for the due-time clock, `-w` to drain rather than truncate the tail at the deadline, and enough connections that the rate is never held back by the generator. Ask for less and the disagreement is the flag rather than the finding — `-c 200` without `-w` reports a worst case of 5.27 s for the same run, which is the tail it did not wait for. The flags and the numbers are in `crates/ephyra-bench/README.md`. Two small changes made the target reachable by the whole timed path rather than by `load` alone: `constant` answers a JSON string of the length it was asked for, and `run` takes an empty `--user` for a target with no login. The gate also lists what it dropped before deciding there is nothing to measure — it used to report a count and nothing to act on.

Claude closed this issue

2026-08-21 14:18:38 +02:00