Do the two servers write BaseItems.Data keys in the same order? #150

New issue

Closed

opened 2026-08-16 15:13:38 +02:00 by Claude · 1 comment

Claude commented

2026-08-16 15:13:38 +02:00

Collaborator

Batch D stood a pair up by hand and diff-db reported two differences that have
nothing to do with the pictures it was there to measure. Neither is filed as a
defect, because the way the pair was built could account for both — but if the
first one is real it is much larger than where it was seen.

BaseItems.Data differs by JSON key order on the three library rows. Same
content; ours writes the keys sorted (CollectionType, Height, IsHD, …) and
upstream writes them in insertion order (PhysicalLocationsList, PhysicalFolderIds, CollectionType, …). The existing Data exemption masks only
DateLastSaved, and diff-db compares literals — so if ours sorts everywhere,
this is a difference on every row whose blob carries more than one key, not a
difference on libraries.

Three AncestorIds rows ours files and upstream does not: each library filed
under the user root folder as an ancestor.

Why it is not filed as a defect

The pair was hand-built rather than stood up through cargo xtask reference, and
that upstream received a POST /Library/VirtualFolders/LibraryOptions ours never
saw — so the library rows may have been written by a different upstream path than
a normal bootstrap uses. declare_libraries writes these rows on every start,
which is what makes both look real rather than incidental, but that is an
argument and not a measurement.

How to know it is done

Reproduce through cargo xtask reference — a bootstrap on each side, then
diff-db. Either the difference is there, in which case the key-order half needs
its blast radius established before anything else, or it is not, in which case
this closes with the note that a hand-built pair diffs differently from a
provisioned one, which is itself worth knowing.

House rules

Tests run under cargo nextest; upstream's source is the checkout in
upstream/ at the pin. The reference on :8096 is read-only, and
Studios=, Artists= and Albums= are writes. Anything needing a write gets
its own container from the pinned image over a cp -a copy of a snapshot, on its
own ports. Rebuilding the reference redraws the account and server ids that
scenarios and config.toml name.

Batch D stood a pair up by hand and `diff-db` reported two differences that have nothing to do with the pictures it was there to measure. Neither is filed as a defect, because the way the pair was built could account for both — but if the first one is real it is much larger than where it was seen. **`BaseItems.Data` differs by JSON key order on the three library rows.** Same content; ours writes the keys sorted (`CollectionType, Height, IsHD, …`) and upstream writes them in insertion order (`PhysicalLocationsList, PhysicalFolderIds, CollectionType, …`). The existing `Data` exemption masks only `DateLastSaved`, and `diff-db` compares literals — so if ours sorts everywhere, this is a difference on every row whose blob carries more than one key, not a difference on libraries. **Three `AncestorIds` rows ours files and upstream does not**: each library filed under the user root folder as an ancestor. ## Why it is not filed as a defect The pair was hand-built rather than stood up through `cargo xtask reference`, and that upstream received a `POST /Library/VirtualFolders/LibraryOptions` ours never saw — so the library rows may have been written by a different upstream path than a normal bootstrap uses. `declare_libraries` writes these rows on every start, which is what makes both look real rather than incidental, but that is an argument and not a measurement. ## How to know it is done Reproduce through `cargo xtask reference` — a bootstrap on each side, then `diff-db`. Either the difference is there, in which case the key-order half needs its blast radius established before anything else, or it is not, in which case this closes with the note that a hand-built pair diffs differently from a provisioned one, which is itself worth knowing. ### House rules Tests run under `cargo nextest`; upstream's source is the checkout in `upstream/` **at the pin**. The reference on `:8096` is read-only, and `Studios=`, `Artists=` and `Albums=` are *writes*. Anything needing a write gets its own container from the pinned image over a `cp -a` copy of a snapshot, on its own ports. Rebuilding the reference redraws the account and server ids that scenarios and `config.toml` name.

Claude added the

correctness

labels

2026-08-16 15:13:38 +02:00

Claude commented

2026-08-20 20:11:56 +02:00

Author

Collaborator

Both halves reproduce, neither the way the hand-built pair read them, and both
are fixed. The reference snapshot's own jellyfin.db is the measurement —
reference/snapshots/default/config/data/jellyfin.db is what a provisioned
upstream wrote, so no pair had to be stood up.

Key order: real, and not everywhere

Ours does not sort every blob. Write::derived_data already writes member
by member through item::ordered_object, in upstream's declaration order, and
the module that owns the rule says so:

The order of a stored blob is contract. … A serde_json object sorts its
keys instead, so it can render the values here and never the object around
them.

Exactly two writers took the top-level object from serde_json anyway —
json!({…}).to_string() renders a BTreeMap, so both came out sorted:

scan/library.rs::library_data, the CollectionFolder blob. declare_libraries
writes it on every start, so it is a difference on every library row.
rows/item.rs::Sharing::blob, a playlist's — including the two properties of
each entry of Shares, which sorting inverts.

box_set_data was already using ordered_object. So the blast radius is
library rows plus playlist rows, not every row whose blob carries more than one
key.

What upstream wrote for the reference's movie library:

{"PhysicalLocationsList":["/config/root/default/movies","/fixtures/media/movies"],
 "PhysicalFolderIds":["8155d293ef2e2969deb6937613da502f"],"CollectionType":"movies",
 "IsRoot":false,"IsHD":false,"IsShortcut":false,"Width":0,"Height":0,
 "DateLastSaved":"…","RemoteTrailers":[]}

CollectionFolder declares the first three; Folder declares IsRoot;
BaseItem declares the rest. DateLastSaved sits between Height and
RemoteTrailers and is what the existing Data exemption masks — json_remove
runs over both sides and preserves the order of everything else, which is why
the order difference was not hidden with it.

Ours is now pinned to that text, upstream's own PhysicalFolderIds hash
included, so the test checks what the blob says as well as the order it says it
in.

`AncestorIds`: real, and the other row

The hand-built pair read the three extra rows as each library filed under the
user root folder. A provisioned upstream files exactly that:

CollectionFolder | UserRootFolder | 4
Folder           | AggregateFolder| 4

One AncestorIds row per CollectionFolder, naming the UserRootFolder — and
none naming the root. Ours filed [user_root, root], so the extra row per
library was the one naming the aggregate root, not the user root. A library
hangs from Media Folders, which hangs from nothing, so its chain stops there;
the folder the library points at is the one that hangs from the root, and it
already did.

Nothing could fail on either, so both shapes are now asserted:
a_librarys_blob_is_written_in_the_order_upstream_wrote_it,
a_playlists_blob_is_written_in_the_order_upstream_declares_it,
a_library_is_an_ancestors_chain_of_one_and_its_folder_another.

Left behind

Upstream marks Folder.LinkedChildren [JsonIgnore] at the pin — links live in
the table — and ours still writes "LinkedChildren":[] into a playlist's blob.
That is a difference in what the blob says rather than in the order it says it
in, and no fixture holds a playlist row, so it is not touched here. Worth its
own issue.

0afe805, 9ac0217.

Both halves reproduce, neither the way the hand-built pair read them, and both are fixed. The reference snapshot's own `jellyfin.db` is the measurement — `reference/snapshots/default/config/data/jellyfin.db` is what a provisioned upstream wrote, so no pair had to be stood up. ## Key order: real, and not everywhere Ours does **not** sort every blob. `Write::derived_data` already writes member by member through `item::ordered_object`, in upstream's declaration order, and the module that owns the rule says so: > **The order of a stored blob is contract.** … A `serde_json` object sorts its > keys instead, so it can render the values here and never the object around > them. Exactly two writers took the top-level object from `serde_json` anyway — `json!({…}).to_string()` renders a `BTreeMap`, so both came out sorted: - `scan/library.rs::library_data`, the `CollectionFolder` blob. `declare_libraries` writes it on every start, so it is a difference on every library row. - `rows/item.rs::Sharing::blob`, a playlist's — including the two properties of each entry of `Shares`, which sorting inverts. `box_set_data` was already using `ordered_object`. So the blast radius is library rows plus playlist rows, not every row whose blob carries more than one key. What upstream wrote for the reference's movie library: ``` {"PhysicalLocationsList":["/config/root/default/movies","/fixtures/media/movies"], "PhysicalFolderIds":["8155d293ef2e2969deb6937613da502f"],"CollectionType":"movies", "IsRoot":false,"IsHD":false,"IsShortcut":false,"Width":0,"Height":0, "DateLastSaved":"…","RemoteTrailers":[]} ``` `CollectionFolder` declares the first three; `Folder` declares `IsRoot`; `BaseItem` declares the rest. `DateLastSaved` sits between `Height` and `RemoteTrailers` and is what the existing `Data` exemption masks — `json_remove` runs over both sides and preserves the order of everything else, which is why the order difference was not hidden with it. Ours is now pinned to that text, upstream's own `PhysicalFolderIds` hash included, so the test checks what the blob says as well as the order it says it in. ## `AncestorIds`: real, and the other row The hand-built pair read the three extra rows as each library filed under the user root folder. A provisioned upstream files exactly that: ``` CollectionFolder | UserRootFolder | 4 Folder | AggregateFolder| 4 ``` One `AncestorIds` row per `CollectionFolder`, naming the `UserRootFolder` — and none naming the root. Ours filed `[user_root, root]`, so the extra row per library was the one naming the **aggregate root**, not the user root. A library hangs from *Media Folders*, which hangs from nothing, so its chain stops there; the folder the library points at is the one that hangs from the root, and it already did. Nothing could fail on either, so both shapes are now asserted: `a_librarys_blob_is_written_in_the_order_upstream_wrote_it`, `a_playlists_blob_is_written_in_the_order_upstream_declares_it`, `a_library_is_an_ancestors_chain_of_one_and_its_folder_another`. ## Left behind Upstream marks `Folder.LinkedChildren` `[JsonIgnore]` at the pin — links live in the table — and ours still writes `"LinkedChildren":[]` into a playlist's blob. That is a difference in what the blob *says* rather than in the order it says it in, and no fixture holds a playlist row, so it is not touched here. Worth its own issue. `0afe805`, `9ac0217`.

Claude referenced this issue

2026-08-20 20:12:22 +02:00

An exemption cannot name the rows it covers, only the columns #151

Claude closed this issue

2026-08-20 20:12:26 +02:00

Claude referenced this issue

2026-08-20 20:13:21 +02:00

A playlist's Data blob carries a member upstream ignores and misses one it writes #160