The committed scenarios cannot be replayed against a library they were not written against #82
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The committed scenarios cannot be replayed against a library they were not
written against.
scenarios/browsing.jsoncarries 13 hardcoded 32-hex ids,playback.json11,playstate.json8,items.json3. They are the ids of thedefaultsnapshot,so against any other library those requests address rows that are not there.
Replayed against a database copied from a running Jellyfin (55,523 items), 45 of
browsing.json's 50 requests come back identical from both servers — and 36 ofthose 45 are identical because both servers answer 404. Nine requests
actually return data.
That is not a failing diff; it is worse than one. A replay against an unfamiliar
library reports near-perfect agreement while exercising the router and almost
nothing else, and nothing in the output distinguishes "both servers agree" from
"neither server has this row".
Two things would fix it, and they are separate:
404s. A run whoserequests mostly miss is a run whose result means nothing, and only the replay
can know that.
LimitandStartIndex, the sorts, the by-name tables, search,/UserViews,/Items/Counts— everything the read path does that does not need to know anid in advance. That set is replayable against any library, which is what
makes a second library worth having.
The second is what turns "diff against a bigger library" from a manual exercise
into something repeatable, and a bigger library is where the ordering, counting
and cost problems live — see the issues filed alongside this one, none of which
the 25-item snapshot can show.
Both halves are on
main, and the accounting turned out to want a wider rule than the issue asked for.A run says what it never reached
Every replay now prints, whether the count is zero or the whole run:
and exits
2when that share crosses--max-unanswered— its own code, so a caller can tell asked nothing from the two servers disagree. Differences still outrank it and still exit1. A machine-readablesummary.jsoncarries the same, per request.The issue asked for 404s. Writing it that way left the nearest sibling invisible: two servers failing identically. A 500/500 pair with matching bodies read as a perfect run in every printed number and in the exit code. So the gating rule is neither server answered — neither status is 2xx — with 404-on-both named as the subset it usually is. The three counts (
missing,failed,empty) are disjoint subsets of the agreements, and every one is private behind an accessor sorecordis the only way in: the invariant the whole change is about is now structural rather than remembered.A 404 pair whose bodies differ stays a difference, not a missing row.
An agreement on an empty collection is counted beside them and gates nothing — a query matching no rows is a real, if weak, agreement, and honest on a 25-item library.
Broadening the rule made two honest runs fragile, so a scenario may declare its own bar:
handshake.jsoncarries"max_unanswered": 100, because 11 of its 24 responses are refusals by design. Nothing else declares one.jellyfin-benchuses the same predicate. It had a second, different notion of a vacuous agreement, printed and never acted on; there is now one function, called from both.What the committed scenarios actually were
playback.jsonbrowsing.jsonportable.jsonOn the 25-item snapshot the same sweep reads: smoke 0%, items 8%, browsing 28%, library 37%, handshake 45%, bundle 0%, playback 60%.
A request set that names no row
scenarios/portable.json— 60 requests, no id anywhere, and a test fails the build if 32 hex characters (dashed or not) ever appear in a path. It replacesanchor.json, which was named after one library, and it leadsDEFAULT_WORKLOADSbecause it is the only committed scenario that measures the query path of a fixture it was not written against.It covers
/UserViews,/Items/Counts, listings for every item type, paging including a page past the end, the sorts, every filter, all the by-name tables, search,/Items/Latest,/UserItems/Resume,/Shows/NextUp, the filter endpoints, and theFields=variants. Two deliberate choices:Fields=ProviderIdsis the last request, and a test asserts there is exactly one and it is last. Upstream's user-data cache is filled by the first query to touch an item, so a request asking for external ids earlier changes what every request after it reports.,SortNameas a tiebreak. A set pointed at an unfamiliar library must not report differences that are only about how two servers break a tie.Not in it, and the README says why:
/Items/Root, whosePathand image tag hash the program data directory the server was given and so report where the pair was stood up rather than anything about the port, and/Items/Suggestions, which upstream orders at random.What it found
Six differences on the committed 25-item snapshot, where every existing scenario reports zero:
/Items/Filtersanswers400where upstream answers a body;SearchTerm=aselects 17 rows against upstream's 10;Filters=IsResumablereturns 17 against upstream's 0;/Yearsorders differently; andEnableUserData=false/EnableImages=falseare ignored. Those are not this issue's, and want one of their own.