Three things a listing sends that upstream does not #81
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Three things a listing sends that upstream does not.
Measured against a database copied from a running Jellyfin — 55,523 items,
31,206 image rows, 116,813 stream rows — with both servers given their own copy.
Each of these is a field present on our side and absent on upstream's, on
requests both servers otherwise answer identically.
People, wherever it is asked for. Upstream sends an empty array and wesend the cast:
Every item on the page, not a few. Upstream evidently does not populate
Peopleon a list even when the field is requested — plausibly it only fills it for a
single-item read — and we fill it every time. Worth pinning down which, because
the cast is 22 rows of join per item and this is also the read path's cost.
UserDataon by-name items.All fifty rows. A genre has no play state, so upstream omits the object; we
synthesise an empty one.
MediaStreams[].Score.Upstream leaves it null on a listing.
None of the three shows against the committed snapshot: it has one person, no
genres worth paging, and few enough streams that the row never appears.
To reproduce. Any library with cast attached to its items.
These gate three benchmark requests. In bench run
20260805T083531Zthe anchor gate dropped the following, and the differences are exactly the three things this issue names — classified over every difference in the run, not sampled:movies-page-1-fieldsPeople, 13×MediaStreams[].Score, 33×UserData.Keyepisodes-deep-page-fieldsPeople, 9×MediaStreams[].ScoregenresUserDataUserDataon by-name itemsepisodes-deep-page-fieldsandgenresare dropped for these differences alone, so fixing this issue takes both into the timed set.movies-page-1-fieldsalso needs #84.Two things worth carrying back into the fix:
Peoplecase is the expensive one.movies-page-1— the same fifty films with noFields— is in the timed set and agrees, so the entire cost difference between the two is work upstream does not do. Not fillingPeopleon a listing is a correctness fix and a performance fix at once.genresis now a 104.5 ms read, down from 403 ms (#83), and none of that is measured against upstream because of a synthesised emptyUserDataobject on fifty rows.The runs behind this issue gave the two servers a metadata directory neither of them could use, and that is most of what they measured.
A by-name item's id — person, genre, studio, artist — is the MD5 of
<type FullName><metadata path>over UTF-16, lowercased, with the path taken relative to the program data directory when it sits underneath one. A picture's tag is the MD5 of that same path whole. Two things are hashed over one directory, and they answer to different parts of it.cargo xtask benchcopied the anchor's metadata directory tostate/metadata— outside both servers' data directories — deliberately, so that both would name one path and agree on picture tags. That satisfies the second rule and breaks the first. Upstream then derives every by-name id from the absolute path, reaches none of the rows the anchor was written with, and:People: []for every item, on a listing and on a by-id read alike;Personrow for each name it touches, with an id of its own;Ours joins a person by name and keeps answering, so the pair looks like a port that invents a cast, loses a view, and miscounts.
Measured just now against the anchor scenario, same host, same binaries, only the layout changed:
The 3,108 suppressions that stopped firing are the
People[*].PrimaryImageTagand blurhash entries — the ignore list was absorbing a setup fault, which is exactly what it is not for.Fixed in
f064438: one program data directory, named to both servers, with the one metadata copy linked inside it.crates/jellyfin-harness/README.mdnow states both rules.So the numbers in this issue want re-reading, not trusting. What survives the corrected layout is written up per issue.
One of the three was this fixture's own fault; the other two were real, and the first hid a different bug underneath it.
Fixed on
mainin951461f,f462fb3,f26232dand the two commits beside them.People— the reported symptom was the layout, the bug underneath was the lookupUpstream does populate
Peopleon a listing. It answered[]for every item because it could not resolve a single person:AttachPeopleemits a credit only whenLibraryManager.GetPerson(name)finds an item, and that lookup is by an id hashed from the person's metadata path — see the comment above on the fixture layout.Arranged correctly, upstream fills the cast, and what was left was ours sending one credit too many on some items and dropping others' ids. Three causes, all in the same lookup:
PresentationUniqueKey = 'Person-' || Peoples.Name. The anchor holds twoPersonrows for many names — one written under each historical layout — so the join multiplied the credit. Upstream reaches exactly one row, by id.PresentationUniqueKeyis stored with marks stripped (Person-Reiner Schone) whilePeoples.Namekeeps them (Reiner Schöne), so the join missed and we sentId: 00000000…. Upstream resolves it: the id ofmetadata\people\r\reiner schöneisf2dc447c…, which is what it serves.Namecarries trailing whitespace while its id is the hash of the trimmed name. Upstream's dictionary lookup fails and it drops the credit; we listed it.We now derive the id the way upstream derives it and check the item's own name, as upstream's
OrdinalIgnoreCasedictionary does. A credit whose item is unreachable is dropped, as upstream drops it.A second divergence in the same array came out of it:
PersonInfo.IsTypematches a credit's role as well as its stored type, so a credit storedUnknownwithRole='Producer'ranks as a producer. Ours ranked on the type alone.And a third, which the id derivation exposed. Our fold was Rust's
str::to_lowercase, which is full Unicode casing; upstream's isToLowerInvariant, which is length-preserving one-to-one. Fifteen probe names were pushed through a live upstream and read back to settle it rather than reading documentation:str::to_lowercaseΟΔΥΣΣΕΑΣοδυσσεασοδυσσεαςİnciİnci— unchangedi̇nciẞetaßeta𐐀deseret𐐨deseretA Greek name ending in Σ therefore hashed to an id upstream never writes. Harmless while the lookup was by name; once the lookup is by id it costs the credit its listing. The fold is now applied at the four sites where upstream calls
ToLowerInvarianton a key — every by-name id,CleanName,SortName, and a series' presentation key — and pinned by the measured vectors. No row in this library folds differently under the two rules, so this is carried by upstream's own answers rather than by a diff.UserDataon/Genres— realGenresControllerandMusicGenresControllerare the only two controllers inJellyfin.Apithat pass a literalfalseforenableUserData./Studios,/Persons,/Artists,/Yearsand/Genres/{name}all pass the query parameter, which defaults to true — so the asymmetry is exactly two routes wide, and it is now ours too.MediaStreams[].Score— realScoreis written in one place,MediaStreamSelector.SetSubtitleStreamScores, reached only when a media source is built for a user. A request asking forMediaStreamswithoutMediaSourcesre-reads the sources with no user, so nothing is scored. Ours now scores only whenMediaSourceswas asked for.Verified in both directions: over 200 movies with
Fields=MediaStreams,MediaSources, upstream scores a stream on 76 of them and all 200 agree exactly — the gate suppresses only what upstream suppresses.After
Against the id-free 61-request set, fresh pair per run: 5,188 → 4,632 differences, five requests changed, nothing else moved.
Fields=Peopleon moviesOverview,People,MediaStreams/MusicGenres/Genres/Genrespage twoCredit counts went 515/521 → 515/515 and 342/350 → 342/342; items whose
Peoplediffer on name, id, type or role went 13/25 → 0/25 and 8/50 → 0/50.Left standing
IsVisible(user)inAttachPeople; neither side models it. It cannot bite for aPersonon this library, but it is a real gap.ItemValueRow.by_name_idstill resolves through a presentation-key join — the same shape of lookup that was wrong here. It produced no differences in either scenario set; it is the first place to look if a by-name id ever disagrees.