alex/ephyra
Watch
1
0
Fork
You've already forked ephyra
 0
Code Issues 2 Pull requests Projects Releases Packages 1 Wiki Activity Actions

Hardware transcoding cannot be reached: the encoder is built from a hardcoded default, and three of its filter chains panic #94

New issue
Closed
opened 2026-08-13 22:59:37 +02:00 by Claude · 1 comment
Claude commented 2026-08-13 22:59:37 +02:00
Collaborator
Copy link

crates/jellyfin-server/src/main.rs:160 builds the encoder with EncodingOptions::default(). That default is hardware_acceleration_type: HardwareAcceleration::None (crates/jellyfin-encoding/src/options.rs:292), and nothing reads the value back out of /System/Configuration/encoding — so every transcode this server performs is software, on a machine whose GPU is idle, and no dashboard setting changes that.

This is the single largest performance gap left. Every other number in report.md is a read-path number; transcoding is where a media server actually spends a processor.

Three things have to happen, and they are separable.

The configuration has to reach the encoder. The EncodingOptions fields exist and are corpus-verified; what is missing is the wiring from the configuration section a dashboard posts to the value the arg builder is handed, and a re-read on a configuration change the way the rest of settings.rs already re-reads.

The seams that refuse have to be told apart from the seams that panic. M5's rule is that an unported hardware case refuses rather than answering, so that the corpus cannot pass a software arg string off as a hardware one. Four of them are unimplemented!() rather than a refusal: crates/jellyfin-encoding/src/filter.rs:78 (the filter chain for an unknown backend), :431 (the AMD Vulkan/DRM interop chain), :466 (QSV over d3d11va), :1275 (hardware deinterlace), and crates/jellyfin-encoding/src/quality.rs:83. In a test that is a refusal; in a server that is a panic in a request handler the moment an operator turns VAAPI on.

The arg strings have to be run. VAAPI, QSV and NVENC are ported and agree with the corpus, and have never been executed — M5 named this as the empirical half that no amount of translation settles. The 324 AMF/VideoToolbox/RKMPP cases are not ported at all and are a runner problem rather than a porting one.

Verify by turning VAAPI on and playing a file that must transcode: today there is no way to do the first half of that sentence.

Where to start

Three separable pieces, in this order.

  1. Wiring. crates/jellyfin-server/src/main.rs:160 is the whole of the
    problem's first half — EncodingOptions::default(). The struct and its
    default are crates/jellyfin-encoding/src/options.rs:220-330. The
    configuration section a dashboard posts lands in
    crates/jellyfin-http/src/routes/configuration.rs; crates/jellyfin-app/src/settings.rs
    is the existing example of a value that is re-read while the server runs
    rather than read once at startup (M7 settled that this is a use case holding
    state, not a port).
  2. The seams. crates/jellyfin-encoding/src/filter.rs:78, :431, :466,
    :1275 and crates/jellyfin-encoding/src/quality.rs:83 are unimplemented!().
    Each becomes a typed refusal a route turns into an answer, not a panic in a
    handler. Keep M5's rule while doing it: an unported hardware case must refuse
    rather than return None, because None is what a machine with no hardware
    answers and the corpus would call that a pass.
  3. The hardware. Upstream's is MediaBrowser.MediaEncoding/Encoder/EncodingHelper.cs.
    crates/jellyfin-encoding/README.md says how the arg strings are known to be
    right; the cases are regenerated with cargo xtask reference gen-corpus and
    never hand-edited.

How to know it is done

  • cargo nextest run -p jellyfin-encoding stays green with a hardware backend
    configured — the corpus is the check on what is emitted.
  • Nothing reachable from an HTTP request can panic: grep the crate for
    unimplemented!/panic! and show each one is either unreachable or a refusal.
  • A file that must transcode plays end to end in a browser with the backend on,
    and the spawned command line carries the hwaccel arguments. That is M5's exit
    criterion repeated on hardware, and it needs a machine that has some — say
    which one it was run on.

House rules

CLAUDE.md is the short version and these are the parts that bite: tests run
under cargo nextest; upstream's source is the checkout in upstream/ at the
pin (v12.0-rc5, 6d501ba4) and not the clone beside the repository;
generated files are regenerated rather than edited; a comment says what the code
does, not how it got there; and Dn references stay out of code and comments.
ARCHITECTURE.md decides which crate a thing goes in and which dependency edges
are illegal, DECISIONS.md says why a rule exists.

The reference pair is cargo xtask reference up (and reference provision,
which is idempotent); a run is
cargo run -p jellyfin-harness -- replay --upstream URL --ours URL --scenario FILE --out DIR,
with diff-db for the two databases. crates/jellyfin-harness/README.md says
what a diff means and what an ignore entry has to carry — every entry needs a
reason and an expiry milestone, and an adds-only history is the smell the risk
table names.

`crates/jellyfin-server/src/main.rs:160` builds the encoder with `EncodingOptions::default()`. That default is `hardware_acceleration_type: HardwareAcceleration::None` (`crates/jellyfin-encoding/src/options.rs:292`), and nothing reads the value back out of `/System/Configuration/encoding` — so every transcode this server performs is software, on a machine whose GPU is idle, and no dashboard setting changes that. This is the single largest performance gap left. Every other number in `report.md` is a read-path number; transcoding is where a media server actually spends a processor. Three things have to happen, and they are separable. **The configuration has to reach the encoder.** The `EncodingOptions` fields exist and are corpus-verified; what is missing is the wiring from the configuration section a dashboard posts to the value the arg builder is handed, and a re-read on a configuration change the way the rest of `settings.rs` already re-reads. **The seams that refuse have to be told apart from the seams that panic.** M5's rule is that an unported hardware case refuses rather than answering, so that the corpus cannot pass a software arg string off as a hardware one. Four of them are `unimplemented!()` rather than a refusal: `crates/jellyfin-encoding/src/filter.rs:78` (the filter chain for an unknown backend), `:431` (the AMD Vulkan/DRM interop chain), `:466` (QSV over d3d11va), `:1275` (hardware deinterlace), and `crates/jellyfin-encoding/src/quality.rs:83`. In a test that is a refusal; in a server that is a panic in a request handler the moment an operator turns VAAPI on. **The arg strings have to be run.** VAAPI, QSV and NVENC are ported and agree with the corpus, and have never been executed — M5 named this as the empirical half that no amount of translation settles. The 324 AMF/VideoToolbox/RKMPP cases are not ported at all and are a runner problem rather than a porting one. Verify by turning VAAPI on and playing a file that must transcode: today there is no way to do the first half of that sentence. --- ## Where to start Three separable pieces, in this order. 1. **Wiring.** `crates/jellyfin-server/src/main.rs:160` is the whole of the problem's first half — `EncodingOptions::default()`. The struct and its default are `crates/jellyfin-encoding/src/options.rs:220-330`. The configuration section a dashboard posts lands in `crates/jellyfin-http/src/routes/configuration.rs`; `crates/jellyfin-app/src/settings.rs` is the existing example of a value that is re-read while the server runs rather than read once at startup (M7 settled that this is a use case holding state, not a port). 2. **The seams.** `crates/jellyfin-encoding/src/filter.rs:78`, `:431`, `:466`, `:1275` and `crates/jellyfin-encoding/src/quality.rs:83` are `unimplemented!()`. Each becomes a typed refusal a route turns into an answer, not a panic in a handler. Keep M5's rule while doing it: an unported hardware case must refuse rather than return `None`, because `None` is what a machine with no hardware answers and the corpus would call that a pass. 3. **The hardware.** Upstream's is `MediaBrowser.MediaEncoding/Encoder/EncodingHelper.cs`. `crates/jellyfin-encoding/README.md` says how the arg strings are known to be right; the cases are regenerated with `cargo xtask reference gen-corpus` and never hand-edited. ## How to know it is done - `cargo nextest run -p jellyfin-encoding` stays green with a hardware backend configured — the corpus is the check on what is emitted. - Nothing reachable from an HTTP request can panic: grep the crate for `unimplemented!`/`panic!` and show each one is either unreachable or a refusal. - A file that must transcode plays end to end in a browser with the backend on, and the spawned command line carries the hwaccel arguments. That is M5's exit criterion repeated on hardware, and it needs a machine that has some — say which one it was run on. ### House rules `CLAUDE.md` is the short version and these are the parts that bite: tests run under `cargo nextest`; upstream's source is the checkout in `upstream/` **at the pin** (`v12.0-rc5`, `6d501ba4`) and not the clone beside the repository; generated files are regenerated rather than edited; a comment says what the code does, not how it got there; and `Dn` references stay out of code and comments. `ARCHITECTURE.md` decides which crate a thing goes in and which dependency edges are illegal, `DECISIONS.md` says why a rule exists. The reference pair is `cargo xtask reference up` (and `reference provision`, which is idempotent); a run is `cargo run -p jellyfin-harness -- replay --upstream URL --ours URL --scenario FILE --out DIR`, with `diff-db` for the two databases. `crates/jellyfin-harness/README.md` says what a diff means and what an ignore entry has to carry — every entry needs a reason and an expiry milestone, and an adds-only history is the smell the risk table names.
Claude commented 2026-08-14 06:25:27 +02:00
Author
Collaborator
Copy link

Done, on main — eight commits, e5892eb through 7f0da07.

A transcode actually ran on the GPU

The arg strings have been corpus-verified since M5 and never once executed. They have been now, against the pinned jellyfin-ffmpeg 8.1.2 — pulled out of the image with docker cp, because /usr/bin/ffmpeg here is 9.0 and M5 settled that a stock binary answers differently about itself and earns arg strings no real install gets.

Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
frame= 2160 fps=1134 speed=35.7x
card1 (RX 7900, behind renderD128) max 34%      card0 (Raphael iGPU) max 0%

Reproduced independently by a reviewer, who also confirmed renderD128 → card1 → PCI 1002:744C. Then the live re-read, with no restart: POST /System/Configuration/encoding adding hevc to the decode list → 204, and the next transcode carried -hwaccel vaapi -hwaccel_output_format vaapi and scale_vaapi=format=nv12:extra_hw_frames=24, at 44.4x.

What the issue did not know

There were eleven panicking seams, not five. hw.rs carried six more — the d3d11va child device, the input hwaccel dispatch, the hardware decoder dispatch, the d3d11va hwaccel, the Windows QSV device. grep -rn "unimplemented!\|todo!\|panic!\|unreachable!" crates/jellyfin-encoding/src/ is now empty, and the two remaining .expect("graphical subtitle") calls were checked rather than assumed: each guard directly entails the Some it unwraps.

The refusals are typed (Unported), never Ok(None) — M5's rule, because None is what a machine with no hardware answers and the corpus would then score software arguments as a passing hardware case. tests/args.rs no longer skips unported cases; it asserts they refuse.

A probe was missing, and without it the wiring would have been a broken command line. With vaapi_driver = Unknown every branch of vaapi_hwaccel_args falls through and emits no -init_hw_device at all. So turning VAAPI on would have reached the encoder and produced arguments that cannot work. capabilities::measure now runs upstream's CheckVaapiDeviceByDriverName against the configured node; live here it reads Mesa Gallium driver … navi31 and reports Amd.

The corpus

Untouched — git diff over corpus/ is empty. The run is 5,306 checked / 318 refused of 5,624, where 5,300 was the old checked count.

324 hardware cases exist and only 318 refuse. The six that answer are all -codec:v:0 copy: a remux decodes and encodes nothing, so no backend branch is on the way to its arg string. That exception was the one place the test was taught something new, so the reviewer tested the property across the whole corpus rather than those six — 31 copy groups, zero whose arg string differs between backends, zero mixing copy with non-copy — and structurally input_video_hwaccel_args returns early on is_copy_codec before any backend match. Not a hole.

What the review changed

The save half-worked, which is worse than not working. The first pass kept five fields and answered 204 for the rest: posted tone mapping on, threads 4, CRF 30, preset slow — read back off, -1, 0, auto. An operator ticks Enable tone mapping, is told it saved, and every HDR transcode stays untonemapped. The whole section is persisted now, verified on a server built from this commit:

before  EnableTonemapping False  ThreadCount -1  Throttle 180  H265Crf 28  Preset auto
after   EnableTonemapping True   ThreadCount  4  Throttle 300  H265Crf 30  Preset slow
discarded: none — and the values are on disk, so a restart keeps them
amf → 400   videotoolbox → 400   rkmpp → 400   vaapi → 204

GET also reported DTO zero-defaults for everything it did not map — H265Crf: 0 where upstream reports 28 and our own options held 28. It reports 28 and 23 now.

AMD_DEBUG=noefc was not ported. Upstream sets it unconditionally on the AMD VAAPI branch, before the interop test, with its own comment that Mesa's EFC is unstable — so it applies to the limited chain this server takes. The corpus structurally cannot catch it: it is a process variable, not an arg string. This change is what made that path reachable, so it is exactly the divergence that goes live here.

The Vulkan/interop divergence, stated properly

CheckVulkanDrmDeviceByExtensionName is deliberately not ported. The reviewer ran upstream's probe on this card: all five extensions present. So upstream here sets IsVaapiDeviceSupportVulkanDrmInterop = true and takes the libplacebo chain — which is unported — meaning probing it honestly would make VAAPI refuse every transcode on the only machine that can test VAAPI.

What this server takes instead is the limited AMD chain, which is corpus-verified (ffmpeg-amd exercises it). The first pass documented that as "an HDR source is tonemapped in software", which is wrong: it goes through OpenCL — a round trip out to memory and back, because Mesa has no VAAPI/OpenCL interop — and if OpenCL were absent the chain falls back entirely, so scaling and deinterlacing would not be on the device either. Slower than libplacebo on-device, not wrong. Both READMEs say that now, and crates/jellyfin-encoding/README.md — the document that owns what green does not mean — carries it.

No corpus environment has the extension (vaapiSupportsVulkanDrmInterop: false in all four), so this is a branch nothing sweeps, named rather than left to be found.

DECISIONS.md gains D29 for the refusal at the door: upstream saves any backend it is given, we answer 400 for one whose arguments we cannot build, because the alternative is a setting that saves and then fails every playback with nothing on screen naming the cause.

Numbers

JELLYFIN_RS_REQUIRE_FIXTURES=1 cargo nextest run --workspace on main: 2188 passed, 0 skipped. Clippy clean. cargo fmt --check produces the identical hunk list to the tree before this change — the one hunk the first pass added is gone.

Not established

  • No browser run. Playback stalls in Chrome on this box after PlaybackInfo, with no /Videos request issued at all and the renderer intermittently freezing — distinguishable from #132, which needs a segment request that is never made here. The server answers correctly at the point it stalls (Transcoding: true, SubProto: hls), and following that URL over HTTP transcodes end to end. The dashboard's Playback page does render the configured backend, the device, and the decode checkboxes including the one added over the API minutes earlier.
  • QSV and NVENC remain corpus-verified and unexecuted — no Intel or NVIDIA hardware here. AMF, VideoToolbox and RKMPP are asserted to refuse, not to work. The Windows seams are asserted against a written-out HostOs::Windows, which is the best available and is not running them.
  • No real HDR transcode. The fixture's HEVC file is 10-bit SDR, so the tonemapping paths are argued from code, upstream's source and a live capability probe rather than from a frame.

Two minor and currently unreachable: TranscodeError::Unsupported becomes a 500 on the two playback routes rather than the 400 the config route gives, and start_or_join extracts attachments and subtitles before arguments() refuses. Both are guarded by validation at startup and at the route.

Done, on `main` — eight commits, `e5892eb` through `7f0da07`. ## A transcode actually ran on the GPU The arg strings have been corpus-verified since M5 and never once executed. They have been now, against the **pinned** `jellyfin-ffmpeg 8.1.2` — pulled out of the image with `docker cp`, because `/usr/bin/ffmpeg` here is 9.0 and M5 settled that a stock binary answers differently about itself and earns arg strings no real install gets. ``` Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi)) frame= 2160 fps=1134 speed=35.7x card1 (RX 7900, behind renderD128) max 34% card0 (Raphael iGPU) max 0% ``` Reproduced independently by a reviewer, who also confirmed `renderD128 → card1 → PCI 1002:744C`. Then the live re-read, with no restart: `POST /System/Configuration/encoding` adding `hevc` to the decode list → 204, and the **next** transcode carried `-hwaccel vaapi -hwaccel_output_format vaapi` and `scale_vaapi=format=nv12:extra_hw_frames=24`, at 44.4x. ## What the issue did not know **There were eleven panicking seams, not five.** `hw.rs` carried six more — the d3d11va child device, the input hwaccel dispatch, the hardware decoder dispatch, the d3d11va hwaccel, the Windows QSV device. `grep -rn "unimplemented!\|todo!\|panic!\|unreachable!" crates/jellyfin-encoding/src/` is now empty, and the two remaining `.expect("graphical subtitle")` calls were checked rather than assumed: each guard directly entails the `Some` it unwraps. The refusals are typed (`Unported`), never `Ok(None)` — M5's rule, because `None` is what a machine with no hardware answers and the corpus would then score software arguments as a passing hardware case. `tests/args.rs` no longer *skips* unported cases; it asserts they refuse. **A probe was missing, and without it the wiring would have been a broken command line.** With `vaapi_driver = Unknown` every branch of `vaapi_hwaccel_args` falls through and emits **no `-init_hw_device` at all**. So turning VAAPI on would have reached the encoder and produced arguments that cannot work. `capabilities::measure` now runs upstream's `CheckVaapiDeviceByDriverName` against the configured node; live here it reads `Mesa Gallium driver … navi31` and reports `Amd`. ## The corpus Untouched — `git diff` over `corpus/` is empty. The run is **5,306 checked / 318 refused** of 5,624, where 5,300 was the old checked count. 324 hardware cases exist and only 318 refuse. The six that answer are all `-codec:v:0 copy`: a remux decodes and encodes nothing, so no backend branch is on the way to its arg string. That exception was the one place the test was taught something new, so the reviewer tested the property across the whole corpus rather than those six — **31 copy groups, zero whose arg string differs between backends, zero mixing copy with non-copy** — and structurally `input_video_hwaccel_args` returns early on `is_copy_codec` before any backend match. Not a hole. ## What the review changed **The save half-worked, which is worse than not working.** The first pass kept five fields and answered 204 for the rest: posted tone mapping on, threads 4, CRF 30, preset slow — read back off, -1, 0, auto. An operator ticks *Enable tone mapping*, is told it saved, and every HDR transcode stays untonemapped. The whole section is persisted now, verified on a server built from this commit: ``` before EnableTonemapping False ThreadCount -1 Throttle 180 H265Crf 28 Preset auto after EnableTonemapping True ThreadCount 4 Throttle 300 H265Crf 30 Preset slow discarded: none — and the values are on disk, so a restart keeps them amf → 400 videotoolbox → 400 rkmpp → 400 vaapi → 204 ``` `GET` also reported DTO zero-defaults for everything it did not map — `H265Crf: 0` where upstream reports 28 and our own options held 28. It reports 28 and 23 now. **`AMD_DEBUG=noefc` was not ported.** Upstream sets it unconditionally on the AMD VAAPI branch, *before* the interop test, with its own comment that Mesa's EFC is unstable — so it applies to the limited chain this server takes. The corpus structurally cannot catch it: it is a process variable, not an arg string. This change is what made that path reachable, so it is exactly the divergence that goes live here. ## The Vulkan/interop divergence, stated properly `CheckVulkanDrmDeviceByExtensionName` is deliberately **not** ported. The reviewer ran upstream's probe on this card: all five extensions present. So upstream here sets `IsVaapiDeviceSupportVulkanDrmInterop = true` and takes the libplacebo chain — which is unported — meaning probing it honestly would make VAAPI **refuse every transcode on the only machine that can test VAAPI**. What this server takes instead is the limited AMD chain, which is corpus-verified (`ffmpeg-amd` exercises it). The first pass documented that as "an HDR source is tonemapped in software", which is wrong: it goes through **OpenCL** — a round trip out to memory and back, because Mesa has no VAAPI/OpenCL interop — and if OpenCL were absent the chain falls back entirely, so scaling and deinterlacing would not be on the device either. Slower than libplacebo on-device, not wrong. Both READMEs say that now, and `crates/jellyfin-encoding/README.md` — the document that owns what green does not mean — carries it. No corpus environment has the extension (`vaapiSupportsVulkanDrmInterop: false` in all four), so this is a branch nothing sweeps, named rather than left to be found. `DECISIONS.md` gains **D29** for the refusal at the door: upstream saves any backend it is given, we answer 400 for one whose arguments we cannot build, because the alternative is a setting that saves and then fails every playback with nothing on screen naming the cause. ## Numbers `JELLYFIN_RS_REQUIRE_FIXTURES=1 cargo nextest run --workspace` on `main`: **2188 passed, 0 skipped**. Clippy clean. `cargo fmt --check` produces the identical hunk list to the tree before this change — the one hunk the first pass added is gone. ## Not established - **No browser run.** Playback stalls in Chrome on this box *after* `PlaybackInfo`, with no `/Videos` request issued at all and the renderer intermittently freezing — distinguishable from #132, which needs a segment request that is never made here. The server answers correctly at the point it stalls (`Transcoding: true`, `SubProto: hls`), and following that URL over HTTP transcodes end to end. The dashboard's Playback page does render the configured backend, the device, and the decode checkboxes including the one added over the API minutes earlier. - **QSV and NVENC remain corpus-verified and unexecuted** — no Intel or NVIDIA hardware here. AMF, VideoToolbox and RKMPP are asserted to *refuse*, not to work. The Windows seams are asserted against a written-out `HostOs::Windows`, which is the best available and is not running them. - **No real HDR transcode.** The fixture's HEVC file is 10-bit SDR, so the tonemapping paths are argued from code, upstream's source and a live capability probe rather than from a frame. Two minor and currently unreachable: `TranscodeError::Unsupported` becomes a 500 on the two playback routes rather than the 400 the config route gives, and `start_or_join` extracts attachments and subtitles before `arguments()` refuses. Both are guarded by validation at startup and at the route.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
correctness
Diverges from upstream

  
encoding
Transcoding, ffmpeg, and the arg strings

  
http
Route behaviour

  
jobs
Scheduled tasks

  
operations
Installing a server and keeping one running

  
p1
Do this first

  
p2
After the p1s

  
p3
Real, and not urgent

  
performance
Speed or resource use

  
providers
Metadata from somewhere other than the file

  
scanner
The library scan

  
security
Who is allowed to see what

  
testing
Test coverage and harness

No labels
correctness
encoding
http
jobs
operations
p1
p2
p3
performance
providers
scanner
security
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
alex/ephyra#94
No description provided.