The bench gate times a request whose whole envelope an ignore entry hides #146

New issue

Closed

opened 2026-08-16 08:42:38 +02:00 by Claude · 2 comments

Claude commented

2026-08-16 08:42:38 +02:00

Collaborator

crates/jellyfin-bench/src/gate.rs keeps a request when the ignore list leaves
nothing behind:

let applied = ignore_list.apply(&request.method, &request.path, recording::compare(upstream, ours));
if applied.kept.is_empty() { … gated.kept.push(request.clone()); }

An entry that hides the whole envelope — status, content type and body — makes a
pair of responses comparable that is not. The gate's own reason for existing is
that timing two servers which answered different things measures the router
rather than the work, and the harness prints exactly that count per run
(report.rs's agreed_without_answering); the gate reads it only for a
shared non-2xx, not for a difference an entry suppressed.

The one entry with that shape today is the idle scheduled-task cancel added in
#125, where upstream throws through its exception middleware and this server
does nothing at all. So the reachable case is benching tasks.json, which
nothing does — this is filed before it is reachable rather than after.

Where to start

gate.rs. The question is whether an entry-suppressed envelope difference
should be a reason to drop a request from the timed set, the way an unanswered
pair already is. A field-scoped entry on a body field is not the same thing: the
two servers still both answered, and timing them is honest.

Applied already carries suppressed with the Rule on each, so the gate can
ask what was hidden without a new mechanism.

How to know it is done

cargo xtask bench over a scenario carrying such a request drops it, says so in
report.md the way it says what else it dropped, and the count is not zero on a
run that includes one.

House rules

Tests run under cargo nextest. crates/jellyfin-harness/README.md says what a
diff means and what an entry must carry; crates/jellyfin-bench is what reads
the same rule for timings.

`crates/jellyfin-bench/src/gate.rs` keeps a request when the ignore list leaves nothing behind: ```rust let applied = ignore_list.apply(&request.method, &request.path, recording::compare(upstream, ours)); if applied.kept.is_empty() { … gated.kept.push(request.clone()); } ``` An entry that hides the whole envelope — status, content type and body — makes a pair of responses comparable that is not. The gate's own reason for existing is that timing two servers which answered different things measures the router rather than the work, and the harness prints exactly that count per run (`report.rs`'s `agreed_without_answering`); the gate reads it only for a *shared* non-`2xx`, not for a difference an entry suppressed. The one entry with that shape today is the idle scheduled-task cancel added in #125, where upstream throws through its exception middleware and this server does nothing at all. So the reachable case is benching `tasks.json`, which nothing does — this is filed before it is reachable rather than after. ## Where to start `gate.rs`. The question is whether an entry-suppressed *envelope* difference should be a reason to drop a request from the timed set, the way an unanswered pair already is. A field-scoped entry on a body field is not the same thing: the two servers still both answered, and timing them is honest. `Applied` already carries `suppressed` with the `Rule` on each, so the gate can ask what was hidden without a new mechanism. ## How to know it is done `cargo xtask bench` over a scenario carrying such a request drops it, says so in `report.md` the way it says what else it dropped, and the count is not zero on a run that includes one. ### House rules Tests run under `cargo nextest`. `crates/jellyfin-harness/README.md` says what a diff means and what an entry must carry; `crates/jellyfin-bench` is what reads the same rule for timings.

Claude added the

testing

labels

2026-08-16 08:58:03 +02:00

Claude commented

2026-08-20 20:12:22 +02:00

Author

Collaborator

The gate now drops a request whose envelope an entry hid, and says how many.
a178908.

What it is

The envelope paths are named beside recording::compare, which writes them, so
the gate and the diff cannot drift apart:

const ENVELOPE: [&str; 4] = ["@status", "@content-type", "@allow", "@body"];
pub fn is_envelope(difference: &Difference) -> bool

@length and @range are deliberately not envelope: both servers sent a body
of the same kind and only its measurements differ, which is the resized-image
entry and not this. gate.rs keeps a request only when applied.kept is empty
and nothing entry-hidden touched the envelope; otherwise it drops it, tags
it hidden_envelope, and lists the suppressed difference as
hidden by an entry — @status: upstream 500, ours 204. Gate::hidden_envelope()
counts them, and Report::render prints the count above the dropped lines.

crates/ephyra-harness/README.md carries the rule beside the one about
requests neither server answered.

Tested

an_entry_over_the_whole_envelope_leaves_nothing_to_time — the #125 entry's
own shape, upstream 500 text/plain against our 204 empty: three differences
suppressed, all three envelope.
an_entry_on_a_body_field_leaves_a_request_to_time — traceId. Both
answered; timing them is honest.
an_entry_on_a_bodys_length_leaves_a_request_to_time — @length on a
resized picture. Same.
a_run_says_how_many_requests_an_entry_took_out_of_the_timed_set — the count
and the line reach render().

Not done

The end-to-end half of "how to know it is done" — cargo xtask bench over a
scenario carrying such a request, with a non-zero count in the report. It is
still unreachable for the reason the issue was filed with: the only entry with
that shape is the idle scheduled-task cancel, and nothing benches tasks.json.
Adding it to the bench manifest is the step that would make the run possible,
and that is a scope decision this issue declined. Leaving open for it.

The gate now drops a request whose envelope an entry hid, and says how many. `a178908`. ## What it is The envelope paths are named beside `recording::compare`, which writes them, so the gate and the diff cannot drift apart: ```rust const ENVELOPE: [&str; 4] = ["@status", "@content-type", "@allow", "@body"]; pub fn is_envelope(difference: &Difference) -> bool ``` `@length` and `@range` are deliberately not envelope: both servers sent a body of the same kind and only its measurements differ, which is the resized-image entry and not this. `gate.rs` keeps a request only when `applied.kept` is empty **and** nothing entry-hidden touched the envelope; otherwise it drops it, tags it `hidden_envelope`, and lists the suppressed difference as `hidden by an entry — @status: upstream 500, ours 204`. `Gate::hidden_envelope()` counts them, and `Report::render` prints the count above the dropped lines. `crates/ephyra-harness/README.md` carries the rule beside the one about requests neither server answered. ## Tested - `an_entry_over_the_whole_envelope_leaves_nothing_to_time` — the #125 entry's own shape, upstream 500 `text/plain` against our 204 empty: three differences suppressed, all three envelope. - `an_entry_on_a_body_field_leaves_a_request_to_time` — `traceId`. Both answered; timing them is honest. - `an_entry_on_a_bodys_length_leaves_a_request_to_time` — `@length` on a resized picture. Same. - `a_run_says_how_many_requests_an_entry_took_out_of_the_timed_set` — the count and the line reach `render()`. ## Not done The end-to-end half of "how to know it is done" — `cargo xtask bench` over a scenario carrying such a request, with a non-zero count in the report. It is still unreachable for the reason the issue was filed with: the only entry with that shape is the idle scheduled-task cancel, and nothing benches `tasks.json`. Adding it to the bench manifest is the step that would make the run possible, and that is a scope decision this issue declined. Leaving open for it.

Claude commented

2026-08-20 22:32:16 +02:00

Author

Collaborator

Measured, and closing.

Benching tasks.json was the wrong way to satisfy this. The gate's job is to
drop that request, so the scenario would time nothing and exist only to make
a counter non-zero. What was actually worth knowing is whether the new rule
removes anything from a set a run already times — and bundle, which is in
DEFAULT_WORKLOADS, is the scenario that can answer it: it is the only one
that asks for /Plugins, /Packages, /Repositories and ExternalIdInfos,
the four routes whose entries are fields: ["**"], scope: "field". A **
pattern matches @status, @content-type and @body, so if the two servers
disagreed on an envelope there, this change would have silently stopped timing
those requests.

cargo xtask bench --workload bundle over the default fixture:

gate: 41 of 43 requests agreed
  dropped GET /Items/…/Ancestors    — [1].Path differs
  dropped GET /System/Info/Storage  — Libraries: upstream 0 entries, ours 4

Both drops are body differences that were already dropping before. Every
hidden_envelope is false, the count line does not print, and all four
wildcard-entry routes were kept. The rule costs the timed set nothing.

Two caveats on the run, neither affecting the count: upstream was started with
a stock ffmpeg rather than the pinned one, because the copy under
reference/published/ffmpeg has an absolute RPATH of
/usr/lib/jellyfin-ffmpeg/lib and cannot load its libraries out of the tree —
bundle asks for no encoding, so nothing it replays reaches an argument
builder. And the two drops above are artifacts of running upstream natively out
of a bench state directory rather than the container's /config.

The rule and its reporting are covered by four tests that fail when the
predicate or the count is wrong:
an_entry_over_the_whole_envelope_leaves_nothing_to_time,
an_entry_on_a_body_field_leaves_a_request_to_time,
an_entry_on_a_bodys_length_leaves_a_request_to_time, and
a_run_says_how_many_requests_an_entry_took_out_of_the_timed_set.

Measured, and closing. Benching `tasks.json` was the wrong way to satisfy this. The gate's job is to *drop* that request, so the scenario would time nothing and exist only to make a counter non-zero. What was actually worth knowing is whether the new rule removes anything from a set a run already times — and `bundle`, which is in `DEFAULT_WORKLOADS`, is the scenario that can answer it: it is the only one that asks for `/Plugins`, `/Packages`, `/Repositories` and `ExternalIdInfos`, the four routes whose entries are `fields: ["**"], scope: "field"`. A `**` pattern matches `@status`, `@content-type` and `@body`, so if the two servers disagreed on an envelope there, this change would have silently stopped timing those requests. `cargo xtask bench --workload bundle` over the `default` fixture: ``` gate: 41 of 43 requests agreed dropped GET /Items/…/Ancestors — [1].Path differs dropped GET /System/Info/Storage — Libraries: upstream 0 entries, ours 4 ``` Both drops are body differences that were already dropping before. Every `hidden_envelope` is false, the count line does not print, and all four wildcard-entry routes were kept. The rule costs the timed set nothing. Two caveats on the run, neither affecting the count: upstream was started with a stock ffmpeg rather than the pinned one, because the copy under `reference/published/ffmpeg` has an absolute `RPATH` of `/usr/lib/jellyfin-ffmpeg/lib` and cannot load its libraries out of the tree — `bundle` asks for no encoding, so nothing it replays reaches an argument builder. And the two drops above are artifacts of running upstream natively out of a bench state directory rather than the container's `/config`. The rule and its reporting are covered by four tests that fail when the predicate or the count is wrong: `an_entry_over_the_whole_envelope_leaves_nothing_to_time`, `an_entry_on_a_body_field_leaves_a_request_to_time`, `an_entry_on_a_bodys_length_leaves_a_request_to_time`, and `a_run_says_how_many_requests_an_entry_took_out_of_the_timed_set`.

Claude closed this issue

2026-08-20 22:32:17 +02:00