alex/ephyra

Watch

Fork 0

A JPEG encoder that agrees with Skia, byte for byte #140

New issue

Closed

opened 2026-08-14 15:44:45 +02:00 by Claude · 1 comment

Claude commented

2026-08-14 15:44:45 +02:00

Collaborator

TrickplayInfos.Bandwidth is ceil(tile_bytes * 8 / interval_seconds / tiles_wide / tiles_high) — a pure function of the encoded tile's byte length. Measured on the fixture: a 110,967-byte tile gives exactly the 888 upstream stored. So a one-byte difference in the JPEG changes a column the harness diffs.

Upstream encodes through SkiaSharp at quality 90, which is libjpeg-turbo underneath. Matching that means an encoder whose output is byte-identical: the same quantisation tables, the same Huffman tables, the same subsampling, the same restart-marker and progressive decisions.

For scale, crates/jellyfin-images/src/jpeg.rs is the comparable decoder and runs to 1,804 lines. The project has already met this shape once — M10's blur hash needed Skia's resampler reproduced exactly, and the notes there record that the scale factor is applied twice, that the sampling is nearest with a float reciprocal nudged one unit in the last place, and that fancy chroma upsampling is off below a two-pixel block. An encoder is the same kind of problem, larger.

Why it is worth doing at all

Without it, trickplay sheets can be generated and served — a client gets working scrub previews — but one column of one table cannot agree and needs an ignore entry. #99 proposes exactly that as its first step. This issue is what retires the entry.

It would also close the position collage.rs currently records: that a picture this server draws is not upstream's bytes. Genre collages have the same property, so this is not trickplay's alone.

Where to start

crates/jellyfin-images/src/jpeg.rs for the decoder's conventions and the project's existing habits about matching a C++ library exactly. Upstream's side is SkiaSharp's SKImage.Encode(SKEncodedImageFormat.Jpeg, 90).

Settle first, before writing anything: what does Skia actually emit at quality 90 — tables, subsampling, and whether it writes progressive — measured off a tile upstream produced, rather than read out of libjpeg's documentation.

How to know it is done

A tile this server encodes is byte-identical to the one upstream encoded from the same frames, and #99's Bandwidth ignore entry is deleted rather than re-argued.

House rules

Tests run under cargo nextest; upstream's source is the checkout in upstream/ at the pin; generated artifacts are regenerated, never edited. The reference on :8096 is read-only, and Studios=/Artists=/Albums= are writes — upstream resolves them through CreateItemByName. Anything needing a write gets its own container from the pinned image over a cp -a copy of a snapshot. Kill a server by the PID ss -lntpH 'sport = :NNNN' reports; never pkill, killall, or pgrep -f with a pattern your own command line contains.

`TrickplayInfos.Bandwidth` is `ceil(tile_bytes * 8 / interval_seconds / tiles_wide / tiles_high)` — **a pure function of the encoded tile's byte length**. Measured on the fixture: a 110,967-byte tile gives exactly the 888 upstream stored. So a one-byte difference in the JPEG changes a column the harness diffs. Upstream encodes through SkiaSharp at quality 90, which is libjpeg-turbo underneath. Matching that means an encoder whose output is byte-identical: the same quantisation tables, the same Huffman tables, the same subsampling, the same restart-marker and progressive decisions. For scale, `crates/jellyfin-images/src/jpeg.rs` is the comparable **decoder** and runs to 1,804 lines. The project has already met this shape once — M10's blur hash needed Skia's *resampler* reproduced exactly, and the notes there record that the scale factor is applied twice, that the sampling is nearest with a float reciprocal nudged one unit in the last place, and that fancy chroma upsampling is off below a two-pixel block. An encoder is the same kind of problem, larger. ## Why it is worth doing at all Without it, trickplay sheets can be generated and served — a client gets working scrub previews — but one column of one table cannot agree and needs an ignore entry. #99 proposes exactly that as its first step. This issue is what retires the entry. It would also close the position `collage.rs` currently records: that a picture this server *draws* is not upstream's bytes. Genre collages have the same property, so this is not trickplay's alone. ## Where to start `crates/jellyfin-images/src/jpeg.rs` for the decoder's conventions and the project's existing habits about matching a C++ library exactly. Upstream's side is SkiaSharp's `SKImage.Encode(SKEncodedImageFormat.Jpeg, 90)`. Settle first, before writing anything: **what does Skia actually emit at quality 90** — tables, subsampling, and whether it writes progressive — measured off a tile upstream produced, rather than read out of libjpeg's documentation. ## How to know it is done A tile this server encodes is byte-identical to the one upstream encoded from the same frames, and #99's `Bandwidth` ignore entry is deleted rather than re-argued. ### House rules Tests run under `cargo nextest`; upstream's source is the checkout in `upstream/` **at the pin**; generated artifacts are regenerated, never edited. The reference on `:8096` is read-only, and `Studios=`/`Artists=`/`Albums=` are *writes* — upstream resolves them through `CreateItemByName`. Anything needing a write gets its own container from the pinned image over a `cp -a` copy of a snapshot. Kill a server by the PID `ss -lntpH 'sport = :NNNN'` reports; never `pkill`, `killall`, or `pgrep -f` with a pattern your own command line contains.

Claude added the

encoding

labels

2026-08-14 15:44:46 +02:00

Claude referenced this issue

2026-08-16 15:13:37 +02:00

No harness run has ever exercised a trickplay route #148

Claude referenced this issue

2026-08-16 15:13:38 +02:00

The byte-exact JPEG encoder is only used for trickplay #149

Claude referenced this issue

2026-08-16 15:15:37 +02:00

Trickplay sheets are never generated, so scrubbing a video shows nothing #99

Claude commented

2026-08-16 15:15:37 +02:00

Author

Collaborator

Done, on main, and it converged — this was the batch's designated drop and did
not have to be dropped. crates/jellyfin-images/src/jpeg_encode.rs is the
encoder beside the 1,804-line decoder.

What Skia emits at quality 90, measured off upstream's own tile

Baseline sequential (SOF0), not progressive.
JFIF APP0 v1.1, density unit 0, 1×1, no thumbnail. No Adobe APP14, no COM.
4:2:0 chroma (h2 v2 luma, 1 1 chroma), component ids 1/2/3, quantisation
tables 0/1/1.
The spec's K.1 tables scaled by 200 - 2*quality = 20, i.e. (base*20 + 50)/100.
Luma zig-zag reads 3,2,2,3,2,2,3,3,…, chroma 3,4,4,5,4,5,9,5,… — checked
against the DQT markers of upstream's sheet.
Huffman tables optimised per image, not the standard ones
(optimize_coding = TRUE).
No restart markers. One scan, Ss=0 Se=63 Ah=Al=0. Marker order
SOI, APP0, DQT(0), DQT(1), SOF0, DHT(DC0), DHT(AC0), DHT(DC1), DHT(AC1), SOS, …, EOI.

Two rules that are not what libjpeg's documentation says

Both were found by sweeping sizes rather than by reading, and both were then
confirmed against libjpeg's own sources.

A dummy block filling out a partial MCU takes the DC of the previous block
in the MCU, whatever edge it sits on — not the documented "copy the block to
the left, or the block above". Decoding both files' entropy streams and
pulling the DC per block gives -168, -62, -62, -62 where the documented rule
gives -168, -62, -168, -62. Every visible pixel is identical either way; the
DC symbol histogram is not, which changes the optimised table, which changes
the file. 33×17 and 101×7 decoded to a maximum pixel difference of 0 and still
differed by 4 and 9 bytes.
Chroma rows past the bottom of the picture average the rows that exist and
then repeat the last averaged row — not average repeated source rows, which
is a different thing when the height is even. 160×90 was the case that showed
it: 12,426 bytes against 12,384, maximum pixel difference 31.

Verification

The nine frames upstream's own run cut are committed beside the test
(crates/jellyfin-images/tests/sheet.rs); laid out and encoded at 90 they give
110,967 bytes, sha256 0f1d74b1…d9c3d782 — upstream's sheet exactly. End to
end, from the real .mkv through our ffmpeg, the file our server writes is
cmp-identical to upstream's and Bandwidth is 888.

The review round swept 61 images × 13 qualities = 793 encodes against real
libjpeg-turbo 3.2.0 (cjpeg -baseline -optimize -sample 2x2 -dct int): all 793
byte-identical. That covers 1×1, 1×2, 2×1, 100×1, 1×100, odd width and odd
height, grayscale content fed as RGB, flat colour, a black/white checkerboard,
and qualities 1, 2, 5, 10, 25, 49, 50, 51 — both sides of the 5000/q ↔
200-2q branch — 75, 90, 95, 99 and 100. Both quantiser clamps are right,
including that force_baseline caps at 255.

It also read the encoder line by line against jcparam.c, jccolor.c,
jcsample.c, jcprepct.c, jcdctmgr.c, jcdctint.c, jchuff.c and
jcmarker.c: marker order, DQT/DHT sent_table de-duplication, the si-vs-bits
difference in code assignment, the guarded flush_bits, and the FIX()
truncations all agree.

Two things worth writing down

The answer keys are generated with -baseline. cjpeg defaults to
force_baseline=FALSE and emits 16-bit quantisation tables below quality ~24,
which is not what Skia asks for. Regenerating without it will produce a phantom
disagreement. This is in tests/edges.rs's module doc.

Rule (2) had no coverage until the review round. It is a no-op unless the
height is even, not a multiple of 16, and the last two rows differ — and all six
originally committed pictures failed one of those, so mutating the rule left both
answer-key tests green while failing 482 of the 793 sweep. tests/edges/24x18
was added for exactly that shape, and the mutation now goes red.

The `Bandwidth` entry

Opened in 7f31074 and deleted in 3cd112d, rather than argued —
db-ignore-list.json is byte-for-byte unchanged across the batch. Bandwidth
itself is now derived in one place, jellyfin_core::image::sheet_bandwidth.

Filed forward

#149 — the encoder is exact and only trickplay uses it. The served-image path
still encodes through image::codecs::jpeg, so the position collage.rs records
is not yet closed; what stands between them is the resampler, not the encoder.

Not a defect but worth knowing before someone bumps ffmpeg: the host's n9.0.1 and
upstream's jellyfin-ffmpeg 8.1.2-2 produce thumbnails that differ by one byte
each — the COM marker's encoder string — and nothing else. Byte-exactness
here does not depend on having jellyfin's ffmpeg build.

Done, on `main`, and it converged — this was the batch's designated drop and did not have to be dropped. `crates/jellyfin-images/src/jpeg_encode.rs` is the encoder beside the 1,804-line decoder. ## What Skia emits at quality 90, measured off upstream's own tile - Baseline sequential (`SOF0`), **not** progressive. - JFIF `APP0` v1.1, density unit 0, 1×1, no thumbnail. No Adobe `APP14`, no `COM`. - 4:2:0 chroma (`h2 v2` luma, `1 1` chroma), component ids 1/2/3, quantisation tables 0/1/1. - The spec's K.1 tables scaled by `200 - 2*quality` = 20, i.e. `(base*20 + 50)/100`. Luma zig-zag reads `3,2,2,3,2,2,3,3,…`, chroma `3,4,4,5,4,5,9,5,…` — checked against the DQT markers of upstream's sheet. - **Huffman tables optimised per image**, not the standard ones (`optimize_coding = TRUE`). - No restart markers. One scan, `Ss=0 Se=63 Ah=Al=0`. Marker order `SOI, APP0, DQT(0), DQT(1), SOF0, DHT(DC0), DHT(AC0), DHT(DC1), DHT(AC1), SOS, …, EOI`. ## Two rules that are not what libjpeg's documentation says Both were found by sweeping sizes rather than by reading, and both were then confirmed against libjpeg's own sources. 1. **A dummy block filling out a partial MCU takes the DC of the previous block in the MCU**, whatever edge it sits on — not the documented "copy the block to the left, or the block above". Decoding both files' entropy streams and pulling the DC per block gives `-168, -62, -62, -62` where the documented rule gives `-168, -62, -168, -62`. Every visible pixel is identical either way; the DC symbol histogram is not, which changes the optimised table, which changes the file. 33×17 and 101×7 decoded to a maximum pixel difference of 0 and still differed by 4 and 9 bytes. 2. **Chroma rows past the bottom of the picture average the rows that exist and then repeat the last averaged row** — not average repeated source rows, which is a different thing when the height is even. 160×90 was the case that showed it: 12,426 bytes against 12,384, maximum pixel difference 31. ## Verification The nine frames upstream's own run cut are committed beside the test (`crates/jellyfin-images/tests/sheet.rs`); laid out and encoded at 90 they give 110,967 bytes, `sha256 0f1d74b1…d9c3d782` — upstream's sheet exactly. End to end, from the real `.mkv` through our ffmpeg, the file our server writes is `cmp`-identical to upstream's and `Bandwidth` is 888. The review round swept **61 images × 13 qualities = 793 encodes against real libjpeg-turbo 3.2.0** (`cjpeg -baseline -optimize -sample 2x2 -dct int`): all 793 byte-identical. That covers 1×1, 1×2, 2×1, 100×1, 1×100, odd width *and* odd height, grayscale content fed as RGB, flat colour, a black/white checkerboard, and qualities 1, 2, 5, 10, 25, 49, **50, 51** — both sides of the `5000/q` ↔ `200-2q` branch — 75, 90, 95, 99 and **100**. Both quantiser clamps are right, including that `force_baseline` caps at 255. It also read the encoder line by line against `jcparam.c`, `jccolor.c`, `jcsample.c`, `jcprepct.c`, `jcdctmgr.c`, `jcdctint.c`, `jchuff.c` and `jcmarker.c`: marker order, DQT/DHT `sent_table` de-duplication, the `si`-vs-`bits` difference in code assignment, the guarded `flush_bits`, and the `FIX()` truncations all agree. ## Two things worth writing down **The answer keys are generated with `-baseline`.** `cjpeg` defaults to `force_baseline=FALSE` and emits 16-bit quantisation tables below quality ~24, which is not what Skia asks for. Regenerating without it will produce a phantom disagreement. This is in `tests/edges.rs`'s module doc. **Rule (2) had no coverage until the review round.** It is a no-op unless the height is even, not a multiple of 16, and the last two rows differ — and all six originally committed pictures failed one of those, so mutating the rule left both answer-key tests green while failing 482 of the 793 sweep. `tests/edges/24x18` was added for exactly that shape, and the mutation now goes red. ## The `Bandwidth` entry Opened in `7f31074` and deleted in `3cd112d`, rather than argued — `db-ignore-list.json` is byte-for-byte unchanged across the batch. `Bandwidth` itself is now derived in one place, `jellyfin_core::image::sheet_bandwidth`. ## Filed forward #149 — the encoder is exact and only trickplay uses it. The served-image path still encodes through `image::codecs::jpeg`, so the position `collage.rs` records is not yet closed; what stands between them is the resampler, not the encoder. Not a defect but worth knowing before someone bumps ffmpeg: the host's n9.0.1 and upstream's jellyfin-ffmpeg 8.1.2-2 produce thumbnails that differ by **one byte each** — the `COM` marker's encoder string — and nothing else. Byte-exactness here does not depend on having jellyfin's ffmpeg build.