A library resolves whole before any of it is filed #32

New issue

Closed

opened 2026-08-03 08:22:36 +02:00 by Claude · 2 comments

Claude commented

2026-08-03 08:22:36 +02:00

Collaborator

A pass hands its rows to a writer a hundred at a time, so what it has read is
bounded. What it is about to read is not: resolving a library materialises a
Write for every item in it before the first batch is filed, and the walk
holds the whole listing beside it.

crates/jellyfin-app/src/scan/run.rs
    let entries = walk(files, &wanted.media_path).await?;      // the whole listing
    let resolved = library_tree(&wanted.media_path, ...);      // a node per entry
    let rows = library(layout, wanted, &resolved.nodes, at)?;  // a Write per node

On a four-thousand-film library the peak is ~35 MB, against ~49 MB before the
batching — so the enriched rows and their streams, chapters and images were
the bulk of it. The skeleton is what is left, and it grows with the library.

Worth measuring on the thirteen-thousand-file library before deciding whether
it is worth streaming the resolver, which is a larger change than the batching
was: library_tree sorts its nodes, and the sort is what makes a folder come
before what is under it.

A pass hands its rows to a writer a hundred at a time, so what it has read is bounded. What it is about to read is not: resolving a library materialises a `Write` for every item in it before the first batch is filed, and the walk holds the whole listing beside it. ``` crates/jellyfin-app/src/scan/run.rs let entries = walk(files, &wanted.media_path).await?; // the whole listing let resolved = library_tree(&wanted.media_path, ...); // a node per entry let rows = library(layout, wanted, &resolved.nodes, at)?; // a Write per node ``` On a four-thousand-film library the peak is ~35 MB, against ~49 MB before the batching — so the enriched rows and their streams, chapters and images were the bulk of it. The skeleton is what is left, and it grows with the library. Worth measuring on the thirteen-thousand-file library before deciding whether it is worth streaming the resolver, which is a larger change than the batching was: `library_tree` sorts its nodes, and the sort is what makes a folder come before what is under it.

Claude referenced this issue

2026-08-03 08:45:59 +02:00

Nothing is written until the whole pass completes #22

Claude commented

2026-08-03 18:07:46 +02:00

Author

Collaborator

Measured before changing anything, on /mnt/media (peak RSS is VmHWM, read
from /proc/<pid>/status at the moment the scanned line is logged).

An instrumented pass over the music library — 5387 rows, nothing else running
— says where the memory is:

RSS 20 MB — a server with an empty library (the floor)
RSS 27 MB — start of pass (Filed: 5387 rows, 5386 chains, 5285 tracks)
RSS 30 MB — the maps derived from Filed, and the skeleton above the libraries
RSS 31 MB — after walk (5410 entries)
RSS 36 MB — after library_tree (5383 nodes)
RSS 40 MB — after library (5385 Writes; size_of::<Write>() is 664)
RSS 41 MB — after plan (5385 planned)
RSS 37 MB — end of pass

So the skeleton is ~11 MB of a 41 MB peak, about 2 KB an item, and the rows
already filed plus the maps built from them are another ~10 MB — the same order,
and not what this issue is about.

Streaming the resolver does not pay for itself. library needs every node
twice — once for the map from a node's path to its id, and again in
series_link, which looks a season's series up among its siblings — so a lazy
library would hold the nodes for as long as the eager one holds the rows.
Nodes measure 5 MB and rows 4 MB: the swap is a wash, for a change that reaches
through library_tree, library and plan.

What was pure duplication has gone instead: the resolved tree now goes out of
scope with the rows it built, and plan is drawn one row at a time rather than
collected into a second vector the size of the library.

scan	before	after
music alone, 5387 rows, rescan	44.3 MB	39.4 MB
whole library, 14578 rows, rescan	73.6 MB	72.1 MB
whole library, 14578 rows, first scan	—	51.9 MB

Two samples each for the music library, three each for the whole one. The
whole-library figure is a mean: those runs spread over 8 MB, which is wider than
the saving, so all it says is that nothing got worse. The music library is the
measurement that resolves — three libraries in a row reuse each other's freed
skeleton, and one does not.

Left open, and bigger than what is left of the skeleton: Filed reads every row,
every chain and every track on record into memory before the pass starts, which
is ~19 MB at 14578 rows and grows with the library exactly as this issue
describes.

Measured before changing anything, on `/mnt/media` (peak RSS is `VmHWM`, read from `/proc/<pid>/status` at the moment the `scanned` line is logged). An instrumented `pass` over the music library — 5387 rows, nothing else running — says where the memory is: ``` RSS 20 MB — a server with an empty library (the floor) RSS 27 MB — start of pass (Filed: 5387 rows, 5386 chains, 5285 tracks) RSS 30 MB — the maps derived from Filed, and the skeleton above the libraries RSS 31 MB — after walk (5410 entries) RSS 36 MB — after library_tree (5383 nodes) RSS 40 MB — after library (5385 Writes; size_of::<Write>() is 664) RSS 41 MB — after plan (5385 planned) RSS 37 MB — end of pass ``` So the skeleton is ~11 MB of a 41 MB peak, about 2 KB an item, and the rows already filed plus the maps built from them are another ~10 MB — the same order, and not what this issue is about. **Streaming the resolver does not pay for itself.** `library` needs every node twice — once for the map from a node's path to its id, and again in `series_link`, which looks a season's series up among its siblings — so a lazy `library` would hold the nodes for as long as the eager one holds the rows. Nodes measure 5 MB and rows 4 MB: the swap is a wash, for a change that reaches through `library_tree`, `library` and `plan`. What was pure duplication has gone instead: the resolved tree now goes out of scope with the rows it built, and `plan` is drawn one row at a time rather than collected into a second vector the size of the library. | scan | before | after | |---|---|---| | music alone, 5387 rows, rescan | 44.3 MB | 39.4 MB | | whole library, 14578 rows, rescan | 73.6 MB | 72.1 MB | | whole library, 14578 rows, first scan | — | 51.9 MB | Two samples each for the music library, three each for the whole one. The whole-library figure is a mean: those runs spread over 8 MB, which is wider than the saving, so all it says is that nothing got worse. The music library is the measurement that resolves — three libraries in a row reuse each other's freed skeleton, and one does not. Left open, and bigger than what is left of the skeleton: `Filed` reads every row, every chain and every track on record into memory before the pass starts, which is ~19 MB at 14578 rows and grows with the library exactly as this issue describes.

Claude commented

2026-08-03 19:48:12 +02:00

Author

Collaborator

Partly fixed and merged, with the premise measured and set aside.

The proposed fix does not pay

Streaming the resolver was measured before being attempted, and it is a wash:
library() needs every node twice — once for the path-to-id map and once for
the sibling lookup a series link does — so a lazy resolver would hold the nodes
exactly as long as the eager one holds the rows it built them into. A change
reaching through three modules for no saving.

What was worth removing was duplication: the resolved tree now goes out of scope
with the rows it built, and a pass draws its plan one row at a time rather than
collecting a second library-sized list of it.

Measured

/mnt/media/music, 5387 rows, release build, VmHWM at the scan's end, three
runs each:

	peak, mean of three
before	54.8 MB
after	50.9 MB

The row order a pass files in is unchanged — the same sequence and the same
25574 ancestor rows — and a rescan still writes nothing.

What is now the larger term

Two things a pass holds that grew since this issue was opened, both worth their
own measurement rather than being folded in here:

Filed reads every row, every chain and every track on record before the pass
starts — around 19 MB at 14578 rows, and it grows with the library.
Where every row hangs is now answered once, after every library has spoken,
so the chains a pass resolved are held until the last library is read. That is
what makes a folder two libraries share hang below both (#33), and it is a
library-sized map.

Partly fixed and merged, with the premise measured and set aside. ## The proposed fix does not pay Streaming the resolver was measured before being attempted, and it is a wash: `library()` needs every node twice — once for the path-to-id map and once for the sibling lookup a series link does — so a lazy resolver would hold the nodes exactly as long as the eager one holds the rows it built them into. A change reaching through three modules for no saving. What was worth removing was duplication: the resolved tree now goes out of scope with the rows it built, and a pass draws its plan one row at a time rather than collecting a second library-sized list of it. ## Measured `/mnt/media/music`, 5387 rows, release build, `VmHWM` at the scan's end, three runs each: | | peak, mean of three | |---|---| | before | 54.8 MB | | after | **50.9 MB** | The row order a pass files in is unchanged — the same sequence and the same 25574 ancestor rows — and a rescan still writes nothing. ## What is now the larger term Two things a pass holds that grew since this issue was opened, both worth their own measurement rather than being folded in here: - `Filed` reads every row, every chain and every track on record before the pass starts — around 19 MB at 14578 rows, and it grows with the library. - Where every row hangs is now answered once, after every library has spoken, so the chains a pass resolved are held until the last library is read. That is what makes a folder two libraries share hang below both (#33), and it is a library-sized map.

Claude closed this issue

2026-08-03 19:48:13 +02:00