Nothing is written until the whole pass completes #22

New issue

Closed

opened 2026-08-02 14:27:25 +02:00 by Claude · 1 comment

Claude commented

2026-08-02 14:27:25 +02:00

Collaborator

Pass accumulates every item, stream and image for all libraries in memory and
the caller writes once at the end. RSS climbed to ~170 MB through the scan and
the database stayed at its 4 seed rows until the final commit.

A crash or a restart part-way through loses everything and starts over, and
nothing is visible until the whole thing lands.

Committing per library — or per N items — bounds memory, makes a crash
resumable, and with a backgrounded scan lets one library be browsable while the
next is still reading.

Found by scanning /mnt/media (285 movies, 289 shows, 74 music artists —
~13 000 media files on a CIFS mount) with both this server and upstream
Jellyfin at the pinned commit, remote metadata and image fetchers disabled on
upstream so both were reading files only. Upstream was given the same mount
path so MD5(UTF-16LE(path)) agrees and rows compare directly.

`Pass` accumulates every item, stream and image for all libraries in memory and the caller writes once at the end. RSS climbed to ~170 MB through the scan and the database stayed at its 4 seed rows until the final commit. A crash or a restart part-way through loses everything and starts over, and nothing is visible until the whole thing lands. Committing per library — or per N items — bounds memory, makes a crash resumable, and with a backgrounded scan lets one library be browsable while the next is still reading. --- Found by scanning `/mnt/media` (285 movies, 289 shows, 74 music artists — ~13 000 media files on a CIFS mount) with both this server and upstream Jellyfin at the pinned commit, remote metadata and image fetchers disabled on upstream so both were reading files only. Upstream was given the same mount path so `MD5(UTF-16LE(path))` agrees and rows compare directly.

Claude added the

scanner

performance

labels

2026-08-02 14:27:25 +02:00

Claude commented

2026-08-03 08:45:59 +02:00

Author

Collaborator

Fixed in c9245f8.

What changed

A pass no longer holds the library it read. ScanWriter is a new port in
jellyfin-core: a pass hands over a batch of a hundred rows — the items, the
ancestor chains they hang from, and what each carries beside its row — and
empties it. The composition root writes each batch in one transaction.
Pass came down to three counts plus the ids a removal is scoped by.

Two things fell out of the port that were worth having anyway:

A row a rescan leaves alone gets its chain filed when it is not the chain on
record. Without that, a renamed library — or a second library added over a
folder already scanned — showed up empty until every file under it happened
to change, because the folder a library is browsed as is derived from its
name. A rescan reads the chains on record once and compares them as sets —
AncestorIds records no order, and upstream writes one a row at a time — so
a rescan that changed nothing still writes nothing.
The sweep of values nothing is listed under moved out of the write and into
one call after the pass — and after the removal, which is what takes the last
mention some of them had. It is an anti-join over the whole table; running it
a hundred and thirty times a scan was work for nothing.

Measured

A synthetic four-thousand-film library on local disk, so the probing is fast
enough to watch:

	before	after
RSS over the scan	15 → 49 MB, climbing	flat at ~35 MB
rows visible during the scan	none until the end	every hundred items
rescan of the unchanged library	—	flat at 31 MB

Both builds produce the same database: BaseItems and AncestorIds digests
match, and the only rows that differ are the library folders' own
DateCreated.

A rescan that changes nothing writes nothing: AncestorIds comes out of it at
the same count and the same maximum rowid it went in with.

Three rounds of review went into this; what they turned up is in the tests —
crates/jellyfin-app/tests/scan_incremental.rs covers a library larger than one
batch, a rename, a rename back to a name once used, two libraries trading
names, a library added over a folder already scanned, a folder two libraries
read as different things, and a chain stored in the order upstream leaves
behind.

On /mnt/media the first hundred rows land 8m33s into the scan at 27 MB, where
before the database stayed empty for the length of it.

What this does not fix

Resolving a library still materialises every Write for it before the first
batch is filed, and the walk holds the whole listing. What the batching takes
off the peak is the enriched rows and their streams, chapters and images —
the bulk of it, but a large library still allocates its own skeleton up front.
Filed separately as #32.

Two things the review turned up that are older than this branch:

#33 — a folder two libraries share ends up listed under whichever named it
last, and a renamed library leaves its old collection folder behind.
#34 — an item left alone keeps a ParentId naming a folder that has stopped
being a row, and goes with it for one scan.

Both want an upstream comparison before anything is changed; the tests assert
what the port does today either way.

Fixed in c9245f8. ## What changed A pass no longer holds the library it read. `ScanWriter` is a new port in `jellyfin-core`: a pass hands over a batch of a hundred rows — the items, the ancestor chains they hang from, and what each carries beside its row — and empties it. The composition root writes each batch in one transaction. `Pass` came down to three counts plus the ids a removal is scoped by. Two things fell out of the port that were worth having anyway: - A row a rescan leaves alone gets its chain filed when it is not the chain on record. Without that, a renamed library — or a second library added over a folder already scanned — showed up empty until every file under it happened to change, because the folder a library is browsed as is derived from its name. A rescan reads the chains on record once and compares them as sets — `AncestorIds` records no order, and upstream writes one a row at a time — so a rescan that changed nothing still writes nothing. - The sweep of values nothing is listed under moved out of the write and into one call after the pass — and after the removal, which is what takes the last mention some of them had. It is an anti-join over the whole table; running it a hundred and thirty times a scan was work for nothing. ## Measured A synthetic four-thousand-film library on local disk, so the probing is fast enough to watch: | | before | after | |---|---|---| | RSS over the scan | 15 → 49 MB, climbing | flat at ~35 MB | | rows visible during the scan | none until the end | every hundred items | | rescan of the unchanged library | — | flat at 31 MB | Both builds produce the same database: `BaseItems` and `AncestorIds` digests match, and the only rows that differ are the library folders' own `DateCreated`. A rescan that changes nothing writes nothing: `AncestorIds` comes out of it at the same count and the same maximum rowid it went in with. Three rounds of review went into this; what they turned up is in the tests — `crates/jellyfin-app/tests/scan_incremental.rs` covers a library larger than one batch, a rename, a rename back to a name once used, two libraries trading names, a library added over a folder already scanned, a folder two libraries read as different things, and a chain stored in the order upstream leaves behind. On `/mnt/media` the first hundred rows land 8m33s into the scan at 27 MB, where before the database stayed empty for the length of it. ## What this does not fix Resolving a library still materialises every `Write` for it before the first batch is filed, and the walk holds the whole listing. What the batching takes off the peak is the enriched rows and their streams, chapters and images — the bulk of it, but a large library still allocates its own skeleton up front. Filed separately as #32. Two things the review turned up that are older than this branch: - #33 — a folder two libraries share ends up listed under whichever named it last, and a renamed library leaves its old collection folder behind. - #34 — an item left alone keeps a `ParentId` naming a folder that has stopped being a row, and goes with it for one scan. Both want an upstream comparison before anything is changed; the tests assert what the port does today either way.

Claude closed this issue

2026-08-03 08:46:13 +02:00

Claude referenced this issue

2026-08-03 10:52:50 +02:00

A pass reads one file at a time where upstream reads thirteen #36