/Years counts the folders its walk passed through, and answers an unsorted page in a different order #89

New issue

Closed

opened 2026-08-12 08:51:35 +02:00 by Claude · 1 comment

Claude commented

2026-08-12 08:51:35 +02:00

Collaborator

/Years was closed for ordering in #86. Two differences remain, both measured, neither an artefact.

`TotalRecordCount` counts different things

GET /Years?Limit=50&SortBy=SortName
  TotalRecordCount: upstream 18, ours 17     (reference/snapshots/default)
  TotalRecordCount: upstream 21829, ours 21895  (the anchor)

GetYears takes its total from Folder.GetRecursiveChildren(user, query, out totalCount), which is AddChildrenFromCollection: childCount = realChildren.Length at each level, plus the recursive sum over every visible sub-folder. Folders are counted, and there is no de-duplication — the result dictionary de-dups the items, the count does not.

For the snapshot that is the user root's 4 children + movies 3 + music 1 + shows 1 + Playlists 0 + artist 1 + album 2 + series 2 + seasons 4 = 18, confirmed live per folder. Ours reports a flat recursive row count, which is 17.

Reproducing it means modelling upstream's walk, not changing a line.

The unsorted page is in a different order

GET /Years            upstream ["2020","2022","2021","2019"]
                      ours     ["2019","2020","2022","2021"]

Upstream's is the depth-first order AddChildren builds over the user root (movies, shows, music); ours is whatever the SQL returns. SortBy=Default now answers the same page as no sort on each server — that part matches upstream, where no comparer declares ItemSortBy.Default so Sort leaves the walk order alone — but the two servers still disagree on what that walk order is.

portable.json never sees either, because it always sends SortBy=SortName.

Not this, for the record

The 100 remaining anchor differences on this route (50 Items[n].Id, 50 Items[n].UserData.ItemId) are a fixture artefact and not a bug: the anchor carries two rows per year, one written by an earlier run whose metadata directory sat outside its program data directory. Every Name agrees positionally and both servers answer the same set of years, so the page is the right page. The ids agree on a correctly-arranged snapshot pair.

`/Years` was closed for ordering in #86. Two differences remain, both measured, neither an artefact. ## `TotalRecordCount` counts different things GET /Years?Limit=50&SortBy=SortName TotalRecordCount: upstream 18, ours 17 (reference/snapshots/default) TotalRecordCount: upstream 21829, ours 21895 (the anchor) `GetYears` takes its total from `Folder.GetRecursiveChildren(user, query, out totalCount)`, which is `AddChildrenFromCollection`: `childCount = realChildren.Length` at each level, **plus the recursive sum over every visible sub-folder**. Folders are counted, and there is no de-duplication — the `result` dictionary de-dups the *items*, the count does not. For the snapshot that is the user root's 4 children + movies 3 + music 1 + shows 1 + Playlists 0 + artist 1 + album 2 + series 2 + seasons 4 = **18**, confirmed live per folder. Ours reports a flat recursive row count, which is 17. Reproducing it means modelling upstream's walk, not changing a line. ## The unsorted page is in a different order GET /Years upstream ["2020","2022","2021","2019"] ours ["2019","2020","2022","2021"] Upstream's is the depth-first order `AddChildren` builds over the user root (movies, shows, music); ours is whatever the SQL returns. `SortBy=Default` now answers the same page as no sort on each server — that part matches upstream, where no comparer declares `ItemSortBy.Default` so `Sort` leaves the walk order alone — but the two servers still disagree on what that walk order *is*. `portable.json` never sees either, because it always sends `SortBy=SortName`. ## Not this, for the record The 100 remaining anchor differences on this route (50 `Items[n].Id`, 50 `Items[n].UserData.ItemId`) **are** a fixture artefact and not a bug: the anchor carries two rows per year, one written by an earlier run whose metadata directory sat outside its program data directory. Every `Name` agrees positionally and both servers answer the same set of years, so the page is the right page. The ids agree on a correctly-arranged snapshot pair.

Claude commented

2026-08-12 15:26:26 +02:00

Author

Collaborator

Investigated and not fixed, with the reason measured rather than asserted. Both halves need the same thing, and it is not a query change.

Why both halves are the same problem

Upstream's total comes from Folder.GetRecursiveChildren, and its unsorted order is that same walk's order. So reproducing either means reproducing the walk. Three things make that infrastructure rather than SQL:

1. A collection folder's children are linked, not parented. Walking the anchor's BaseItems from the user root through ParentId reaches 3 rows — the three collection folders — and stops. Every one of the 21.9k library items hangs off a physical folder that the collection folder reaches through LinkedChildren, which this port does not model at all. (That same gap was confirmed independently while closing #90: nothing here reads LinkedChildren, which is why upstream's dto.ChildCount ??= folder.LinkedChildren.Length step is unreachable for us.)

2. The walk is type-dependent. MusicAlbum overrides GetEligibleChildrenForRecursiveChildren to return its Tracks rather than its children. Matching the count means porting those overrides, not just descending a parent column.

3. The count is not a count of anything the database can be asked for. AddChildrenFromCollection sums realChildren.Length at every level with no de-duplication — the result dictionary de-dups the items, the count does not. It counts folders it passed through, and it counts a row twice where the walk reaches it twice.

That last point is why no single rule explains both fixtures: on reference/snapshots/default upstream is higher than ours (18 against 17), and on the anchor it is lower (21,829 against 21,895). A rule fitted to one gets the other backwards.

What it would cost

Measured on the anchor pair, GET /Years?Limit=50&SortBy=SortName, warm:


upstream	0.799 s
ours	0.273 s

Upstream is ~3× slower on this route, and the walk is what it is spending the time on. Reproducing the number faithfully means adopting that cost — visiting every item in the library per request where we currently answer with one aggregate query — plus modelling LinkedChildren and the per-type child overrides to get there.

What is not wrong

The remaining 100 /Years differences on the anchor are Items[n].Id and Items[n].UserData.ItemId only. Every Name agrees positionally and both servers answer the same set of years, so the page is the right page; those ids are an artefact of that fixture carrying two rows per year from an earlier run whose metadata directory sat outside its program data directory. The ordering that #86 closed is still closed, and SortBy=Default answers the same page as no sort on both servers.

Recommendation

Leave it. The visible symptom is a total that is out by 0.3% on a 21k library and a page order that only a client sorting by nothing at all would see, and the fix costs a walk of the whole library per request. Worth reopening if LinkedChildren is ever modelled for another reason — at that point both halves become reachable together.

Investigated and **not fixed**, with the reason measured rather than asserted. Both halves need the same thing, and it is not a query change. ## Why both halves are the same problem Upstream's total comes from `Folder.GetRecursiveChildren`, and its unsorted order is that same walk's order. So reproducing either means reproducing the walk. Three things make that infrastructure rather than SQL: **1. A collection folder's children are linked, not parented.** Walking the anchor's `BaseItems` from the user root through `ParentId` reaches **3 rows** — the three collection folders — and stops. Every one of the 21.9k library items hangs off a physical folder that the collection folder reaches through `LinkedChildren`, which this port does not model at all. (That same gap was confirmed independently while closing #90: nothing here reads `LinkedChildren`, which is why upstream's `dto.ChildCount ??= folder.LinkedChildren.Length` step is unreachable for us.) **2. The walk is type-dependent.** `MusicAlbum` overrides `GetEligibleChildrenForRecursiveChildren` to return its `Tracks` rather than its children. Matching the count means porting those overrides, not just descending a parent column. **3. The count is not a count of anything the database can be asked for.** `AddChildrenFromCollection` sums `realChildren.Length` at every level with no de-duplication — the `result` dictionary de-dups the items, the count does not. It counts folders it passed through, and it counts a row twice where the walk reaches it twice. That last point is why no single rule explains both fixtures: on `reference/snapshots/default` upstream is **higher** than ours (18 against 17), and on the anchor it is **lower** (21,829 against 21,895). A rule fitted to one gets the other backwards. ## What it would cost Measured on the anchor pair, `GET /Years?Limit=50&SortBy=SortName`, warm: | | | |---|---| | upstream | **0.799 s** | | ours | **0.273 s** | Upstream is ~3× slower on this route, and the walk is what it is spending the time on. Reproducing the number faithfully means adopting that cost — visiting every item in the library per request where we currently answer with one aggregate query — plus modelling `LinkedChildren` and the per-type child overrides to get there. ## What is not wrong The remaining 100 `/Years` differences on the anchor are `Items[n].Id` and `Items[n].UserData.ItemId` only. Every `Name` agrees positionally and both servers answer the same set of years, so the page is the right page; those ids are an artefact of that fixture carrying two rows per year from an earlier run whose metadata directory sat outside its program data directory. The ordering that #86 closed is still closed, and `SortBy=Default` answers the same page as no sort on both servers. ## Recommendation Leave it. The visible symptom is a total that is out by 0.3% on a 21k library and a page order that only a client sorting by nothing at all would see, and the fix costs a walk of the whole library per request. Worth reopening if `LinkedChildren` is ever modelled for another reason — at that point both halves become reachable together.

Claude closed this issue

2026-08-12 15:26:26 +02:00

Claude referenced this issue

2026-08-12 15:44:29 +02:00

The search candidate cut runs over the whole narrowing, and there is no fallback when the ranking matches nothing #88