alex/ephyra
Watch
1
0
Fork
You've already forked ephyra
 0
Code Issues 2 Pull requests Projects Releases Packages 1 Wiki Activity Actions

Nothing about the write path, artwork or transcoding has been measured on a real library #117

New issue
Closed
opened 2026-08-13 22:59:45 +02:00 by Claude · 8 comments
Claude commented 2026-08-13 22:59:45 +02:00
Collaborator
Copy link

report.md's numbers are one request set, one library, one host, and read-only: the 56,337-item library was copied without its media files, so playback and transcoding were out of scope for the whole run, and the request set names no specific row.

What that leaves unmeasured, in the order it would matter to a running server:

  • Transcoding throughput. How many concurrent sessions this server sustains against upstream, which is the number a person actually feels. Needs media present, and is the reason the hardware-acceleration issue matters before this one is worth running.
  • The write path. Playback reports, user data writes, and a scan running while reads are served — the WAL contention case that a read-only run cannot reach.
  • Artwork. Every image request re-encodes, and report.md notes our responses are 5% larger per request than upstream's, unexplained.
  • Scanning, which #92 measures on its own but never beside upstream doing the same work.

Also still open from the same run: ten requests are dropped before anything is timed, because the two servers disagree about them. Most of the six bugs behind them are closed (#86 and its children); the drop set should be re-derived rather than assumed to be empty.

Where to start

report.md is one request set, one library, one host, and read-only — the
56,337-item library was copied without its media files, so playback and
transcoding were out of scope for the entire run, and the default set names no
specific row.

In the order it would matter to a running server:

  1. Transcoding throughput — concurrent sessions sustained, against upstream.
    Needs media present, and needs #94 first: measuring software-only transcoding
    against an upstream that can use a GPU measures the wrong thing.
  2. The write path — playback reports, user-data writes, and a scan running
    while reads are served. That is the WAL contention case a read-only run
    cannot reach, and the pragmas were chosen for it (M4).
  3. Artwork — every image request re-encodes, and the report notes our
    responses run 5% larger per request than upstream's, unexplained.
  4. Scanning beside upstream — #92 measures ours alone.

crates/jellyfin-bench/ already gates on agreement before timing anything, so
each of these needs a request set and a library arrangement more than it needs
new machinery. docs/benchmarks.md is the setup.

How to know it is done

Each of the four has a number in report.md with the same method the read-path
numbers have — five repetitions, alternating, medians and spread, each server
pinned to its own cores. Also re-derive the ten requests the two servers still
disagree about: most of the six bugs behind them are closed (#86 and its
children), and the drop set should be measured again rather than assumed empty.

House rules

CLAUDE.md is the short version and these are the parts that bite: tests run
under cargo nextest; upstream's source is the checkout in upstream/ at the
pin (v12.0-rc5, 6d501ba4) and not the clone beside the repository;
generated files are regenerated rather than edited; a comment says what the code
does, not how it got there; and Dn references stay out of code and comments.
ARCHITECTURE.md decides which crate a thing goes in and which dependency edges
are illegal, DECISIONS.md says why a rule exists.

The reference pair is cargo xtask reference up (and reference provision,
which is idempotent); a run is
cargo run -p jellyfin-harness -- replay --upstream URL --ours URL --scenario FILE --out DIR,
with diff-db for the two databases. crates/jellyfin-harness/README.md says
what a diff means and what an ignore entry has to carry — every entry needs a
reason and an expiry milestone, and an adds-only history is the smell the risk
table names.

`report.md`'s numbers are one request set, one library, one host, and read-only: the 56,337-item library was copied without its media files, so **playback and transcoding were out of scope for the whole run**, and the request set names no specific row. What that leaves unmeasured, in the order it would matter to a running server: - **Transcoding throughput.** How many concurrent sessions this server sustains against upstream, which is the number a person actually feels. Needs media present, and is the reason the hardware-acceleration issue matters before this one is worth running. - **The write path.** Playback reports, user data writes, and a scan running while reads are served — the WAL contention case that a read-only run cannot reach. - **Artwork.** Every image request re-encodes, and `report.md` notes our responses are 5% larger per request than upstream's, unexplained. - **Scanning**, which #92 measures on its own but never beside upstream doing the same work. Also still open from the same run: **ten requests are dropped before anything is timed**, because the two servers disagree about them. Most of the six bugs behind them are closed (#86 and its children); the drop set should be re-derived rather than assumed to be empty. --- ## Where to start `report.md` is one request set, one library, one host, and read-only — the 56,337-item library was copied **without its media files**, so playback and transcoding were out of scope for the entire run, and the default set names no specific row. In the order it would matter to a running server: 1. **Transcoding throughput** — concurrent sessions sustained, against upstream. Needs media present, and needs #94 first: measuring software-only transcoding against an upstream that can use a GPU measures the wrong thing. 2. **The write path** — playback reports, user-data writes, and a scan running while reads are served. That is the WAL contention case a read-only run cannot reach, and the pragmas were chosen for it (M4). 3. **Artwork** — every image request re-encodes, and the report notes our responses run 5% larger per request than upstream's, unexplained. 4. **Scanning beside upstream** — #92 measures ours alone. `crates/jellyfin-bench/` already gates on agreement before timing anything, so each of these needs a request set and a library arrangement more than it needs new machinery. `docs/benchmarks.md` is the setup. ## How to know it is done Each of the four has a number in `report.md` with the same method the read-path numbers have — five repetitions, alternating, medians and spread, each server pinned to its own cores. Also re-derive the **ten requests the two servers still disagree about**: most of the six bugs behind them are closed (#86 and its children), and the drop set should be measured again rather than assumed empty. ### House rules `CLAUDE.md` is the short version and these are the parts that bite: tests run under `cargo nextest`; upstream's source is the checkout in `upstream/` **at the pin** (`v12.0-rc5`, `6d501ba4`) and not the clone beside the repository; generated files are regenerated rather than edited; a comment says what the code does, not how it got there; and `Dn` references stay out of code and comments. `ARCHITECTURE.md` decides which crate a thing goes in and which dependency edges are illegal, `DECISIONS.md` says why a rule exists. The reference pair is `cargo xtask reference up` (and `reference provision`, which is idempotent); a run is `cargo run -p jellyfin-harness -- replay --upstream URL --ours URL --scenario FILE --out DIR`, with `diff-db` for the two databases. `crates/jellyfin-harness/README.md` says what a diff means and what an ignore entry has to carry — every entry needs a reason and an expiry milestone, and an adds-only history is the smell the risk table names.
Claude commented 2026-08-21 14:19:05 +02:00
Author
Collaborator
Copy link

Partly done in 14c4ec6 and cfb6e1e. Leaving this open — two of the four areas
have numbers, two do not, and one of the two that do turned out to be blocked on
something else.

What was in the way of all of it

A snapshot's rows name the container's paths — /fixtures/media and /config.
Upstream resolves a stored path literally, so on a native run it was answering
404 to every picture and finding no file behind any item, and the gate
dropped the lot. That is almost certainly part of the ten dropped requests this
issue asks about.

A run now moves both prefixes in each copy of the database, and in the library
shortcuts and collection files beside it — upstream reads a folder's physical
locations off those and rescans when they disagree with the rows. It points them
at the library reference up laid under reference/var/media and lays none of
its own: this host's snapshots were cut against a library laid on the 16th and
the image now produces files stamped the 20th, which is the reconciliation
export_media exists to refuse. An anchor is left alone; its paths belong to
the host that wrote it.

Artwork — measured

artwork.json, enriched, three repetitions, ten resize shapes the bundle asks
for. Upstream serves them from cache/images/resized-images, which it writes on
the first request; this server re-encodes on every request.

point upstream ours ours/up
c=1 p50 0.18 ms 59.30 ms 321
c=1 throughput 4183/s 14.3/s 0.003
c=4 throughput 12309/s 52.7/s 0.004
c=16 throughput 13713/s 53.6/s 0.004

~53 responses a second whatever the concurrency. And the 5% larger responses
are gone: bytes per response are 0.98–0.99 of upstream's, so the cost is the
re-encode alone.

Found on the way: the ignore list could not reach an indexed image —
/Items/*/Images/* is a segment short of /Items/{id}/Images/Backdrop/0, and a
* matches exactly one segment — so two resizes were being dropped for the
reason the entry beside them exists to allow.

Transcoding — runs, cannot be timed

transcoding.json gets both servers transcoding natively. The gate refuses to
time any of it, and that is the result:

request upstream ours
stream.mp4 h264 maxWidth=320 vb=200000 61,866 82,599
stream.mp4 hevc vb=200000 71,247 85,218
stream.ts h264 vb=200000 106,596 127,840
stream.mp4 vcodec=copy acodec=mp3 76,594 116,927

One ffmpeg, one host, 20–50% more bytes. Two argument strings that agreed would
not do that, so the throughput number waits on the arguments rather than on the
rig — which is #141's territory, and a good example of the branch a corpus does
not defend. Upstream's command lines are in the run logs; ours logs transcode started without them, so the two cannot be diffed from a run today.

Also uncovered, and not this issue's: /Audio/{id}/universal 500s on a flac
source where upstream answers, in 1–3 ms, so before ffmpeg runs. Both flac
tracks in enriched, container=mp3 and container=aac alike.

Not done

  • The write path. playstate already covers playback reports and user-data
    writes. What is missing is a scan running while reads are served, and that is
    a second thing running during a timed workload rather than another request
    set.
  • Scanning beside upstream. Same: timing a scan on both is not a request
    set.
  • Re-deriving the drop set against an anchor.

One note on the acceptance: it says each of the four should get a number in
report.md with the read-path method. There is no report.md anywhere in the
tree and nothing references one — the numbers it quotes come from a run whose
write-up is not committed. The two above went into docs/benchmarks.md instead.

Partly done in 14c4ec6 and cfb6e1e. Leaving this open — two of the four areas have numbers, two do not, and one of the two that do turned out to be blocked on something else. ## What was in the way of all of it A snapshot's rows name the container's paths — `/fixtures/media` and `/config`. Upstream resolves a stored path literally, so on a native run it was answering `404` to **every** picture and finding no file behind any item, and the gate dropped the lot. That is almost certainly part of the ten dropped requests this issue asks about. A run now moves both prefixes in each copy of the database, and in the library shortcuts and collection files beside it — upstream reads a folder's physical locations off those and rescans when they disagree with the rows. It points them at the library `reference up` laid under `reference/var/media` and lays none of its own: this host's snapshots were cut against a library laid on the 16th and the image now produces files stamped the 20th, which is the reconciliation `export_media` exists to refuse. An anchor is left alone; its paths belong to the host that wrote it. ## Artwork — measured `artwork.json`, `enriched`, three repetitions, ten resize shapes the bundle asks for. Upstream serves them from `cache/images/resized-images`, which it writes on the first request; this server re-encodes on every request. | point | upstream | ours | ours/up | |---|---|---|---| | c=1 p50 | 0.18 ms | 59.30 ms | 321 | | c=1 throughput | 4183/s | 14.3/s | 0.003 | | c=4 throughput | 12309/s | 52.7/s | 0.004 | | c=16 throughput | 13713/s | 53.6/s | 0.004 | ~53 responses a second whatever the concurrency. And the **5% larger responses are gone**: bytes per response are 0.98–0.99 of upstream's, so the cost is the re-encode alone. Found on the way: the ignore list could not reach an indexed image — `/Items/*/Images/*` is a segment short of `/Items/{id}/Images/Backdrop/0`, and a `*` matches exactly one segment — so two resizes were being dropped for the reason the entry beside them exists to allow. ## Transcoding — runs, cannot be timed `transcoding.json` gets both servers transcoding natively. The gate refuses to time any of it, and that is the result: | request | upstream | ours | |---|---|---| | `stream.mp4` h264 maxWidth=320 vb=200000 | 61,866 | 82,599 | | `stream.mp4` hevc vb=200000 | 71,247 | 85,218 | | `stream.ts` h264 vb=200000 | 106,596 | 127,840 | | `stream.mp4` vcodec=copy acodec=mp3 | 76,594 | 116,927 | One ffmpeg, one host, 20–50% more bytes. Two argument strings that agreed would not do that, so the throughput number waits on the arguments rather than on the rig — which is #141's territory, and a good example of the branch a corpus does not defend. Upstream's command lines are in the run logs; ours logs `transcode started` without them, so the two cannot be diffed from a run today. Also uncovered, and not this issue's: `/Audio/{id}/universal` **500s on a flac source** where upstream answers, in 1–3 ms, so before ffmpeg runs. Both flac tracks in `enriched`, `container=mp3` and `container=aac` alike. ## Not done - **The write path.** `playstate` already covers playback reports and user-data writes. What is missing is a scan running while reads are served, and that is a second thing running during a timed workload rather than another request set. - **Scanning beside upstream.** Same: timing a scan on both is not a request set. - **Re-deriving the drop set** against an anchor. One note on the acceptance: it says each of the four should get a number in `report.md` with the read-path method. There is no `report.md` anywhere in the tree and nothing references one — the numbers it quotes come from a run whose write-up is not committed. The two above went into `docs/benchmarks.md` instead.
Claude commented 2026-08-22 19:39:29 +02:00
Author
Collaborator
Copy link

Still open, and not for want of trying — recording where it stands so the next
attempt does not re-derive this.

The transcoding number is still gated on #94. Neither fixture can produce
it: the anchor is a database and a 7.5 GB metadata directory copied without the
media files, and reference/'s library is nineteen items of five seconds each.
"How many concurrent sessions this server sustains" needs media and hardware
parity, and measuring software-only encoding against an upstream that can reach
a GPU measures the wrong thing.

The other three are unblocked and each needs a request set rather than
machinery. cargo xtask bench --fixture ~/.cache/anchor is the run;
docs/benchmarks.md is the method and crates/ephyra-bench/README.md is what
the generator does. What is missing is the load: playback reports and user-data
writes against a scan for the write path, an image set for artwork, and a scan
run beside upstream's.

The drop set has not been re-derived. One data point that bears on it, from
a run today rather than from the report: refusals.json over a live pair
answered 156 responses with 2 differences, and both are one request —
GET /MusicGenres/InstantMix naming no id, which both servers refuse with
400 and which carries a JSON body upstream and a plain-text one here. That is
a much smaller disagreement than the ten the report dropped, but it is one
scenario rather than the bench's request set, so it does not settle the number.

The baseline moved today. #148 rebuilt the reference snapshots, and the
library on the host was re-laid from the current image with it — so every
figure in report.md describes a fixture that is no longer on disk. A re-run
is needed before anything in it is compared against, whatever else changes.

Still open, and not for want of trying — recording where it stands so the next attempt does not re-derive this. **The transcoding number is still gated on #94.** Neither fixture can produce it: the anchor is a database and a 7.5 GB metadata directory copied without the media files, and `reference/`'s library is nineteen items of five seconds each. "How many concurrent sessions this server sustains" needs media and hardware parity, and measuring software-only encoding against an upstream that can reach a GPU measures the wrong thing. **The other three are unblocked and each needs a request set rather than machinery.** `cargo xtask bench --fixture ~/.cache/anchor` is the run; `docs/benchmarks.md` is the method and `crates/ephyra-bench/README.md` is what the generator does. What is missing is the load: playback reports and user-data writes against a scan for the write path, an image set for artwork, and a scan run beside upstream's. **The drop set has not been re-derived.** One data point that bears on it, from a run today rather than from the report: `refusals.json` over a live pair answered 156 responses with **2 differences**, and both are one request — `GET /MusicGenres/InstantMix` naming no id, which both servers refuse with `400` and which carries a JSON body upstream and a plain-text one here. That is a much smaller disagreement than the ten the report dropped, but it is one scenario rather than the bench's request set, so it does not settle the number. **The baseline moved today.** `#148` rebuilt the reference snapshots, and the library on the host was re-laid from the current image with it — so every figure in `report.md` describes a fixture that is no longer on disk. A re-run is needed before anything in it is compared against, whatever else changes.
Claude commented 2026-08-22 19:55:46 +02:00
Author
Collaborator
Copy link

The drop set is re-derived, and it is twenty rather than ten.

cargo xtask bench --fixture ~/.cache/anchor/upstream --user admin over the
committed workloads, against a native upstream from reference publish. The
gate runs before anything is timed, so this needed no new request set:

workload dropped
browsing 3
bundle 3
items 7
portable 7
handshake, login, login-0, playback 0

Three of browsing's are the ones docs/benchmarks.md already says drop on
every native run — upstream derives the root folder's Path from where it was
started. The rest are not, and they fall into six kinds:

  • ParentBackdropImageTags: upstream has 1 entry, ours 2. In browsing,
    items and portable, on every row of the listing. The widest of them.
  • A by-name listing orders two names differently. /Persons puts
    [hoax record awaiting deletion] at index 7 upstream and A Silver Mt. Zion
    there; the same swap shows in /Items sorted by SortName. A collation
    question about how a leading [ ranks against a letter.
  • ImageBlurHashes disagree — we carry a hash keyed by an image upstream
    does not carry one for, and miss one it does.
  • /Trailers answers 500 upstream and 200 here, twice in portable.
  • Bodies run about 1.4% longer. @length: upstream 34549064, ours 35019928
    on the big /Items listings — five of items' seven drops are only this.
    Related to the report's unexplained 5% on artwork, and now measured on rows
    rather than pictures.
  • /System/Info/Storage reports no libraries upstream and three here,
    which is the native run moving upstream's data directory and probably belongs
    with the three browsing drops rather than with the rest.

Each is a difference on a 56k-item library that no scenario over the generated
fixture can reach — which is the argument this issue makes in the first place.
They want their own issues rather than a line here; I have not filed them.

The four numbers this issue is actually about are still not measured, and the
transcoding one is still gated on #94.

**The drop set is re-derived, and it is twenty rather than ten.** `cargo xtask bench --fixture ~/.cache/anchor/upstream --user admin` over the committed workloads, against a native upstream from `reference publish`. The gate runs before anything is timed, so this needed no new request set: | workload | dropped | |---|---| | `browsing` | 3 | | `bundle` | 3 | | `items` | 7 | | `portable` | 7 | | `handshake`, `login`, `login-0`, `playback` | 0 | Three of `browsing`'s are the ones `docs/benchmarks.md` already says drop on every native run — upstream derives the root folder's `Path` from where it was started. The rest are not, and they fall into six kinds: - **`ParentBackdropImageTags`: upstream has 1 entry, ours 2.** In `browsing`, `items` and `portable`, on every row of the listing. The widest of them. - **A by-name listing orders two names differently.** `/Persons` puts `[hoax record awaiting deletion]` at index 7 upstream and `A Silver Mt. Zion` there; the same swap shows in `/Items` sorted by `SortName`. A collation question about how a leading `[` ranks against a letter. - **`ImageBlurHashes` disagree** — we carry a hash keyed by an image upstream does not carry one for, and miss one it does. - **`/Trailers` answers `500` upstream and `200` here**, twice in `portable`. - **Bodies run about 1.4% longer.** `@length: upstream 34549064, ours 35019928` on the big `/Items` listings — five of `items`' seven drops are only this. Related to the report's unexplained 5% on artwork, and now measured on rows rather than pictures. - **`/System/Info/Storage` reports no libraries upstream and three here**, which is the native run moving upstream's data directory and probably belongs with the three `browsing` drops rather than with the rest. Each is a difference on a 56k-item library that no scenario over the generated fixture can reach — which is the argument this issue makes in the first place. They want their own issues rather than a line here; I have not filed them. The four numbers this issue is actually about are still not measured, and the transcoding one is still gated on #94.
Claude commented 2026-08-22 20:36:32 +02:00
Author
Collaborator
Copy link

I tried to measure the artwork path and it cannot be measured with what is
committed. That is a result, so here it is.

cargo xtask bench --fixture ~/.cache/anchor/upstream --workload artwork --reps 5 --duration 10 --warmup 60 runs to completion and prints numbers —
ours at 37,000 rps against upstream's 3,500, better than tenfold at every
concurrency. Every one of those numbers is worthless. The gate says why:

8 requests agreed, 8 of them unanswered by both servers
dropped GET /Items/b463870f…/Images/Primary?fillWidth=100&fillHeight=100&quality=90 — upstream 404 ours 200
dropped GET /Items/04e35a6a…/Images/Primary?maxHeight=300                            — upstream 404 ours 200

artwork.json names five item ids, and they are the reference fixture's. On the
anchor they name nothing, so the run timed 404 routing rather than image
encoding — exactly the fabricated result docs/benchmarks.md warns about, and
the gate caught it rather than the numbers being believed.

The same holds for the write path: playstate.json names eleven ids and
editing.json twelve. portable.json names none, which is why it is the only
one that means anything against an anchor.

So this issue's framing was right and my earlier note above was wrong: the two
"unblocked" measurements are not a matter of pointing the existing harness
at the anchor. Each needs a request set written against the anchor's own rows —
which is the work, and it is not large, but it is work rather than a run.

What is genuinely in hand: the machinery (WRITE_WORKLOADS already gives a
write scenario a fresh fixture copy per repetition and merges the passes), the
gate that refuses to time a disagreement, and a published native upstream.

One more difference for the list, from the two dropped above: we serve an
image where upstream answers 404, for two rows of the anchor.

**I tried to measure the artwork path and it cannot be measured with what is committed. That is a result, so here it is.** `cargo xtask bench --fixture ~/.cache/anchor/upstream --workload artwork --reps 5 --duration 10 --warmup 60` runs to completion and prints numbers — ours at 37,000 rps against upstream's 3,500, better than tenfold at every concurrency. **Every one of those numbers is worthless.** The gate says why: ``` 8 requests agreed, 8 of them unanswered by both servers dropped GET /Items/b463870f…/Images/Primary?fillWidth=100&fillHeight=100&quality=90 — upstream 404 ours 200 dropped GET /Items/04e35a6a…/Images/Primary?maxHeight=300 — upstream 404 ours 200 ``` `artwork.json` names five item ids, and they are the reference fixture's. On the anchor they name nothing, so the run timed 404 routing rather than image encoding — exactly the fabricated result `docs/benchmarks.md` warns about, and the gate caught it rather than the numbers being believed. The same holds for the write path: `playstate.json` names eleven ids and `editing.json` twelve. `portable.json` names none, which is why it is the only one that means anything against an anchor. So this issue's framing was right and my earlier note above was wrong: the two "unblocked" measurements are **not** a matter of pointing the existing harness at the anchor. Each needs a request set written against the anchor's own rows — which is the work, and it is not large, but it is work rather than a run. What is genuinely in hand: the machinery (`WRITE_WORKLOADS` already gives a write scenario a fresh fixture copy per repetition and merges the passes), the gate that refuses to time a disagreement, and a published native upstream. One more difference for the list, from the two dropped above: **we serve an image where upstream answers `404`**, for two rows of the anchor.
Claude commented 2026-08-22 20:55:34 +02:00
Author
Collaborator
Copy link

#94 is closed, so the transcoding number is not blocked on it any more — and
now that it can be attempted, something else blocks it. Correcting my earlier
note, which took this issue's text on trust instead of checking.

Two things had to be fixed before it could even be attempted:

  • transcoding.json named six ids, all absent from the library. Every
    request 404'd on both servers and agreed about it, so the scenario could not
    fail — the same shape as #148 in another corner. Re-pointed in 55c938e.
  • The run has to go somewhere with 2 GiB free: upstream refuses to start
    otherwise, and the scratch tmpfs is not it.

With those fixed the workload is live, both servers really transcode, and the
gate refuses to time it — five of the six disagree:

request upstream ours
stream.mp4 h264, maxWidth=320 61,866 B 82,599 B (+33%)
stream.mp4 hevc 71,247 B 85,218 B (+20%)
stream.ts 2,273,672 B 2,273,484 B (−0.008%)
stream.mp4 videoCodec=copy, audio→mp3 76,594 B 116,927 B (+53%)
/Audio/{flac}/universal?container=mp3 200 audio/mpeg 500 text/plain

So the answer to "how many concurrent sessions does this server sustain" is not
a number yet, and the reason is no longer hardware: we and upstream produce
different output for the same request, and timing that would compare two
different pieces of work. The gate is right to drop them.

The 500 is the sharp end. A flac asked for as mp3 through
/Audio/{id}/universal fails outright where upstream answers with audio, and
the same item asked for as aac does not fail the same way.

That, and the 20–53% size gap on three of four video shapes, want their own
issues; a review is diagnosing the 500 now. Neither is a benchmark problem —
they are correctness, and they are what this issue was going to run into.

**#94 is closed, so the transcoding number is not blocked on it any more — and now that it can be attempted, something else blocks it.** Correcting my earlier note, which took this issue's text on trust instead of checking. Two things had to be fixed before it could even be attempted: - `transcoding.json` named **six ids, all absent** from the library. Every request 404'd on both servers and agreed about it, so the scenario could not fail — the same shape as #148 in another corner. Re-pointed in `55c938e`. - The run has to go somewhere with 2 GiB free: upstream refuses to start otherwise, and the scratch tmpfs is not it. With those fixed the workload is live, both servers really transcode, and the gate refuses to time it — **five of the six disagree**: | request | upstream | ours | |---|---|---| | `stream.mp4` h264, maxWidth=320 | 61,866 B | 82,599 B (+33%) | | `stream.mp4` hevc | 71,247 B | 85,218 B (+20%) | | `stream.ts` | 2,273,672 B | 2,273,484 B (−0.008%) | | `stream.mp4` videoCodec=copy, audio→mp3 | 76,594 B | 116,927 B (+53%) | | `/Audio/{flac}/universal?container=mp3` | `200 audio/mpeg` | **`500 text/plain`** | So the answer to "how many concurrent sessions does this server sustain" is not a number yet, and the reason is no longer hardware: **we and upstream produce different output for the same request**, and timing that would compare two different pieces of work. The gate is right to drop them. The `500` is the sharp end. A flac asked for as mp3 through `/Audio/{id}/universal` fails outright where upstream answers with audio, and the same item asked for as aac does not fail the same way. That, and the 20–53% size gap on three of four video shapes, want their own issues; a review is diagnosing the `500` now. Neither is a benchmark problem — they are correctness, and they are what this issue was going to run into.
Claude commented 2026-08-22 21:10:52 +02:00
Author
Collaborator
Copy link

Correcting the note above, and the numbers with it. A review found the six
ids were not absent — they were the enriched snapshot's, which
docs/benchmarks.md says is this workload's fixture. Running it against
default is what made them look gone. The scenario was not dead; I had the
wrong fixture.

The re-pointed ids resolve in both snapshots, so that is kept and the
document now says so. Three of the six were also measuring something other than
their name, fixed in 2a34f89:

  • The .ts case was the one number that read like a pass — 188 bytes apart
    — and it was not one. Night Signal is ninety seconds, so the rate cap binds
    and both servers converge on it whatever their arguments say. Night Signal
    belongs on video-to-another-codec (it is the fixture's one hevc source);
    a five-second h264 clip takes the container change, which also restores the
    remux coverage the tree had lost.
  • audio-from-flac-to-aac named no transcodingContainer, so upstream
    muxed AAC with the MP3 muxer and died — and ours refuses the route — so the
    two agreed about failing. With the container named it measures.

Corrected, on default, every one of the six disagrees:

request upstream ours
stream.mp4 h264, maxWidth=320 61,866 B 82,599 B (+33%)
stream.mp4 hevc (Night Signal, 90 s) 1,249,636 B 1,481,301 B (+19%)
stream.ts (Red Harbour, 5 s) 106,596 B 127,840 B (+20%)
stream.mp4 copy, audio→mp3 76,594 B 116,927 B (+53%)
/Audio/{flac}/universal → mp3 200 audio/mpeg 500
/Audio/{flac}/universal → aac 200 500

The 500 is not a bug, it is a hole. crates/ephyra-http/src/routes/streaming/audio.rs:92-98
refuses anything the negotiation does not settle as a direct play, with
"transcoding is not implemented". The encoder path it wants already exists —
videos.rs:503-560 builds the request, starts the job and serves the growing
file — it is simply not wired to this route.

The size gap is real and the corpus cannot see it. It is not the bench
truncating: the byte counts come from the gate, which drains each body once per
server before any timing, compares @length exactly, and has no entry hiding
it. Both servers run one ffmpeg. crates/ephyra-encoding/corpus/progressive-video.json
holds 3,380 cases and none of them is this regime — every source is ≥720×576,
≥5 Mbps, stereo or better; the swept -maxrate bottoms out at 800,000 where
these ask 200,000, -ab at 128,000 where they ask 64,000, maxWidth at 640
where one asks 320, and videoCodec=copy with audioCodec=mp3 has no case at
all. A coverage gap, not leniency.

So the transcoding number this issue wants is still not obtainable, and the
reason is now specific: the two servers do different work for all six
requests. Timing that would compare two different things, which is what the
gate is for.

**Correcting the note above, and the numbers with it.** A review found the six ids were not absent — they were the `enriched` snapshot's, which `docs/benchmarks.md` says is this workload's fixture. Running it against `default` is what made them look gone. The scenario was not dead; I had the wrong fixture. The re-pointed ids resolve in *both* snapshots, so that is kept and the document now says so. Three of the six were also measuring something other than their name, fixed in `2a34f89`: - **The `.ts` case was the one number that read like a pass** — 188 bytes apart — and it was not one. Night Signal is ninety seconds, so the rate cap binds and both servers converge on it whatever their arguments say. Night Signal belongs on `video-to-another-codec` (it is the fixture's one hevc source); a five-second h264 clip takes the container change, which also restores the remux coverage the tree had lost. - **`audio-from-flac-to-aac` named no `transcodingContainer`**, so upstream muxed AAC with the MP3 muxer and died — and ours refuses the route — so the two agreed about failing. With the container named it measures. Corrected, on `default`, every one of the six disagrees: | request | upstream | ours | |---|---|---| | `stream.mp4` h264, maxWidth=320 | 61,866 B | 82,599 B (+33%) | | `stream.mp4` hevc (Night Signal, 90 s) | 1,249,636 B | 1,481,301 B (+19%) | | `stream.ts` (Red Harbour, 5 s) | 106,596 B | 127,840 B (+20%) | | `stream.mp4` copy, audio→mp3 | 76,594 B | 116,927 B (+53%) | | `/Audio/{flac}/universal` → mp3 | `200 audio/mpeg` | `500` | | `/Audio/{flac}/universal` → aac | `200` | `500` | **The `500` is not a bug, it is a hole.** `crates/ephyra-http/src/routes/streaming/audio.rs:92-98` refuses anything the negotiation does not settle as a direct play, with "transcoding is not implemented". The encoder path it wants already exists — `videos.rs:503-560` builds the request, starts the job and serves the growing file — it is simply not wired to this route. **The size gap is real and the corpus cannot see it.** It is not the bench truncating: the byte counts come from the gate, which drains each body once per server before any timing, compares `@length` exactly, and has no entry hiding it. Both servers run one ffmpeg. `crates/ephyra-encoding/corpus/progressive-video.json` holds 3,380 cases and none of them is this regime — every source is ≥720×576, ≥5 Mbps, stereo or better; the swept `-maxrate` bottoms out at 800,000 where these ask 200,000, `-ab` at 128,000 where they ask 64,000, `maxWidth` at 640 where one asks 320, and `videoCodec=copy` with `audioCodec=mp3` has no case at all. A coverage gap, not leniency. So the transcoding number this issue wants is still not obtainable, and the reason is now specific: **the two servers do different work for all six requests.** Timing that would compare two different things, which is what the gate is for.
Claude commented 2026-08-22 21:36:25 +02:00
Author
Collaborator
Copy link

The artwork number, measured. One of the four this issue asks for.

The request set had to be written against the anchor's own rows, which is what
this issue says and what I confirmed the hard way — artwork.json names five
enriched ids and answers nothing on an anchor. Ten requests of the same
shapes over a film and a person the anchor holds; both servers native and
pinned; five repetitions; the gate agreed on 8 of the 10 before anything was
timed, so this is a measurement rather than 404 routing.

point   metric                  upstream        ours   ours/up
c=1     throughput_per_second    4750.01       19.99      0.004
c=8     throughput_per_second   18613.10       76.21      0.004
c=128   throughput_per_second   18032.52       75.95      0.004
c=1     p50_ms                      0.16       42.39        265×
c=128   p50_ms                      4.28     1637.16        382×
        bytes_per_response      284618.2    282512.2       0.99

Ours is flat at ~76 rps from c=8 to c=128 with latency growing linearly — a
queue in front of a CPU-bound step — while upstream is flat at ~18,500 rps with
a sub-millisecond median, which is a file being sent. The responses are the
same picture to within a percent.

The cause is the one this issue already suspected in a smaller form: "every
image request re-encodes, and the report notes our responses run 5% larger per
request than upstream's, unexplained." The 5% is not the story. Two hundred
and forty times is. Filed as #185.

The two requests that dropped are a Person image where upstream answers 404
and we answer with a picture — the same difference the drop set turned up.

So of the five things this issue wants: the drop set is re-derived, artwork is
measured, transcoding is blocked on all six of its requests disagreeing (#184
and the size gap), the write path needs the same treatment artwork just got —
a request set over anchor rows, and WRITE_WORKLOADS already gives it a fresh
fixture per repetition — and scanning beside upstream is not an HTTP request
set at all.

The scenario used here is not committed: it names one household's rows, and
reference/snapshots/ shows the shape that problem already has. Where an
anchor-specific request set should live is worth deciding before the write-path
one is written.

**The artwork number, measured.** One of the four this issue asks for. The request set had to be written against the anchor's own rows, which is what this issue says and what I confirmed the hard way — `artwork.json` names five `enriched` ids and answers nothing on an anchor. Ten requests of the same shapes over a film and a person the anchor holds; both servers native and pinned; five repetitions; **the gate agreed on 8 of the 10 before anything was timed**, so this is a measurement rather than 404 routing. ``` point metric upstream ours ours/up c=1 throughput_per_second 4750.01 19.99 0.004 c=8 throughput_per_second 18613.10 76.21 0.004 c=128 throughput_per_second 18032.52 75.95 0.004 c=1 p50_ms 0.16 42.39 265× c=128 p50_ms 4.28 1637.16 382× bytes_per_response 284618.2 282512.2 0.99 ``` Ours is flat at ~76 rps from c=8 to c=128 with latency growing linearly — a queue in front of a CPU-bound step — while upstream is flat at ~18,500 rps with a sub-millisecond median, which is a file being sent. The responses are the same picture to within a percent. The cause is the one this issue already suspected in a smaller form: *"every image request re-encodes, and the report notes our responses run 5% larger per request than upstream's, unexplained."* The 5% is not the story. **Two hundred and forty times** is. Filed as #185. The two requests that dropped are a `Person` image where upstream answers `404` and we answer with a picture — the same difference the drop set turned up. So of the five things this issue wants: the drop set is re-derived, artwork is measured, transcoding is blocked on all six of its requests disagreeing (#184 and the size gap), the write path needs the same treatment artwork just got — a request set over anchor rows, and `WRITE_WORKLOADS` already gives it a fresh fixture per repetition — and scanning beside upstream is not an HTTP request set at all. The scenario used here is not committed: it names one household's rows, and `reference/snapshots/` shows the shape that problem already has. Where an anchor-specific request set should live is worth deciding before the write-path one is written.
Claude commented 2026-08-23 00:24:17 +02:00
Author
Collaborator
Copy link

The write path, measured — and this issue's last piece. Closing it, split into
what is left.

The write path

A request set written against the anchor's own rows, which is what this issue
says and what the artwork run confirmed the hard way: twenty requests, a
playback session reported from start to stop, played/favourite/rating writes and
a posted UserData, with a resume listing and two listings read while the writes
go on. Three repetitions, each from a fresh copy of the fixture, both servers
pinned to their own cores, the generator to its own.

19 of 20 agreed before anything was timed. The one that dropped is
/UserItems/Resume, on ParentBackdropImageTags — the widest of the drop-set
differences already recorded here.

point   metric                  upstream        ours   ours/up
c=1     throughput_per_second     102.75      229.70      2.24
c=8     throughput_per_second     283.87      613.19      2.16
c=32    throughput_per_second     302.87      624.09      2.06
c=128   throughput_per_second     268.66      631.68      2.35
c=1     p50_ms                      6.08        1.76      0.29
c=32    p50_ms                     62.23       49.33      0.79
c=128   p50_ms                    430.01      216.83      0.50
c=128   p99_ms                   1402.26      620.05      0.44
        bytes_per_response       3662.30     3650.60      1.00

Upstream tops out at ~300 writes a second and falls back to 269 at c=128; ours
climbs to ~630 and stays there. At a fixed rate (25/50/100 a second) both serve
every request and ours answers in a third to a half the time.

What the run also recorded, over 134,400 responses ours and 63,849 upstream:

                        upstream        ours
cpu_seconds_per_request  0.01571     0.00792     ours costs half the processor
rss_peak_bytes           671 MB      1135 MB     and holds more of the machine
cold_start_ms              3327         218

So the WAL contention case this issue was written for does not bite: the
pragmas hold under a concurrent write load on a 56k-item library, and the
one thing to watch is peak RSS rather than throughput.

The request set is not committed — it names one household's rows, and a set
that answers nothing on any other fixture is worse in the repository than
beside the library it was written for. b55a583 teaches cargo xtask bench to
take a request set by path (--workload <file.json>, --writes <name>), which
is where that question this issue kept running into gets its answer.

Where the five things stand

  • The drop set — re-derived, twenty rather than ten, recorded above. Staying
    recorded here rather than filed.
  • Artwork — measured, and the cause was #185: 147× the throughput at c=128
    after 365cd2c, and faster than upstream at the tail.
  • The write path — measured, above.
  • Transcoding throughput — still not measurable, and no longer for want of
    hardware: all six requests disagree because the two servers write different
    files for the same request. Filed as #190 (the divergence) and #191 (the
    measurement waiting behind it).
  • Scanning beside upstream — not an HTTP request set, and no fixture here
    can be asked for it: the anchor has no media, the reference library is
    nineteen five-second clips, and no upstream stands on the real one. Filed as
    #192.

Three of the four numbers exist or are answered; the two that do not have issues
of their own now, each with its own blocker. Nothing is left under this heading.

**The write path, measured — and this issue's last piece. Closing it, split into what is left.** ## The write path A request set written against the anchor's own rows, which is what this issue says and what the artwork run confirmed the hard way: twenty requests, a playback session reported from start to stop, played/favourite/rating writes and a posted `UserData`, with a resume listing and two listings read while the writes go on. Three repetitions, each from a fresh copy of the fixture, both servers pinned to their own cores, the generator to its own. **19 of 20 agreed before anything was timed.** The one that dropped is `/UserItems/Resume`, on `ParentBackdropImageTags` — the widest of the drop-set differences already recorded here. ``` point metric upstream ours ours/up c=1 throughput_per_second 102.75 229.70 2.24 c=8 throughput_per_second 283.87 613.19 2.16 c=32 throughput_per_second 302.87 624.09 2.06 c=128 throughput_per_second 268.66 631.68 2.35 c=1 p50_ms 6.08 1.76 0.29 c=32 p50_ms 62.23 49.33 0.79 c=128 p50_ms 430.01 216.83 0.50 c=128 p99_ms 1402.26 620.05 0.44 bytes_per_response 3662.30 3650.60 1.00 ``` Upstream tops out at ~300 writes a second and falls back to 269 at c=128; ours climbs to ~630 and stays there. At a fixed rate (25/50/100 a second) both serve every request and ours answers in a third to a half the time. What the run also recorded, over 134,400 responses ours and 63,849 upstream: ``` upstream ours cpu_seconds_per_request 0.01571 0.00792 ours costs half the processor rss_peak_bytes 671 MB 1135 MB and holds more of the machine cold_start_ms 3327 218 ``` So the WAL contention case this issue was written for does not bite: the pragmas hold under a concurrent write load on a 56k-item library, and the one thing to watch is peak RSS rather than throughput. The request set is not committed — it names one household's rows, and a set that answers nothing on any other fixture is worse in the repository than beside the library it was written for. `b55a583` teaches `cargo xtask bench` to take a request set by path (`--workload <file.json>`, `--writes <name>`), which is where that question this issue kept running into gets its answer. ## Where the five things stand - **The drop set** — re-derived, twenty rather than ten, recorded above. Staying recorded here rather than filed. - **Artwork** — measured, and the cause was #185: 147× the throughput at c=128 after `365cd2c`, and faster than upstream at the tail. - **The write path** — measured, above. - **Transcoding throughput** — still not measurable, and no longer for want of hardware: all six requests disagree because the two servers write different files for the same request. Filed as #190 (the divergence) and #191 (the measurement waiting behind it). - **Scanning beside upstream** — not an HTTP request set, and no fixture here can be asked for it: the anchor has no media, the reference library is nineteen five-second clips, and no upstream stands on the real one. Filed as #192. Three of the four numbers exist or are answered; the two that do not have issues of their own now, each with its own blocker. Nothing is left under this heading.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
correctness
Diverges from upstream

  
encoding
Transcoding, ffmpeg, and the arg strings

  
http
Route behaviour

  
jobs
Scheduled tasks

  
operations
Installing a server and keeping one running

  
p1
Do this first

  
p2
After the p1s

  
p3
Real, and not urgent

  
performance
Speed or resource use

  
providers
Metadata from somewhere other than the file

  
scanner
The library scan

  
security
Who is allowed to see what

  
testing
Test coverage and harness

No labels
correctness
encoding
http
jobs
operations
p1
p2
p3
performance
providers
scanner
security
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
alex/ephyra#117
No description provided.