pass() does three linear scans over the library per item #20

New issue

Closed

opened 2026-08-02 14:27:25 +02:00 by Claude · 1 comment

Claude commented

2026-08-02 14:27:25 +02:00

Collaborator

Per item, pass() scans collections the size of the library:

call	site
`entry_for(&entries, &item)`	`scan/run.rs:147`
`siblings(entries, &path)`	`scan/run.rs:306`
`already.describes(entry)` -> `compare`	`scan/changes.rs:43`
`compare`'s `removed` — `entries.iter().any` per row	`scan/changes.rs:57`

For the tv library that is 8 907 x 8 907 ~ 79 M path comparisons each. A
HashMap keyed on path, built once per library, replaces all of them.

This is the entire cost of a rescan, where every file is unchanged, no
ffprobe runs at all, and the server still burns hundreds of millions of string
comparisons to conclude it has nothing to do.

Found by scanning /mnt/media (285 movies, 289 shows, 74 music artists —
~13 000 media files on a CIFS mount) with both this server and upstream
Jellyfin at the pinned commit, remote metadata and image fetchers disabled on
upstream so both were reading files only. Upstream was given the same mount
path so MD5(UTF-16LE(path)) agrees and rows compare directly.

Per item, `pass()` scans collections the size of the library: | call | site | |---|---| | `entry_for(&entries, &item)` | `scan/run.rs:147` | | `siblings(entries, &path)` | `scan/run.rs:306` | | `already.describes(entry)` -> `compare` | `scan/changes.rs:43` | | `compare`'s `removed` — `entries.iter().any` per row | `scan/changes.rs:57` | For the tv library that is 8 907 x 8 907 ~ 79 M path comparisons each. A `HashMap` keyed on path, built once per library, replaces all of them. This is the **entire** cost of a rescan, where every file is unchanged, no ffprobe runs at all, and the server still burns hundreds of millions of string comparisons to conclude it has nothing to do. --- Found by scanning `/mnt/media` (285 movies, 289 shows, 74 music artists — ~13 000 media files on a CIFS mount) with both this server and upstream Jellyfin at the pinned commit, remote metadata and image fetchers disabled on upstream so both were reading files only. Upstream was given the same mount path so `MD5(UTF-16LE(path))` agrees and rows compare directly.

Claude added the

scanner

performance

labels

2026-08-02 14:27:25 +02:00

Claude commented

2026-08-03 04:54:45 +02:00

Author

Collaborator

Fixed in cc7e460 and b67e986.

All three scans are indexed now — the walk by path and by containing directory,
the rows filed by path — so a pass asks about an item instead of reading the
library to answer.

Measured, and it is not the entire cost of a rescan. Same library, same
database, nothing changed on disk:

rescan of 14577 unchanged items	wall
before	30.0 s
after	28.1 s

So the comparisons were about 1.5 s of 30, and the other 28.5 s is the walk
itself — listing thirteen thousand files over the mount. At this size the
quadratic term is 5 %; at ten times the size it would be the whole of it, which
is why it is worth removing rather than leaving.

Review found the change had left compare behind with nothing calling it, and
the rule that a directory is compared on its presence alone written out twice.
describes is the whole of what the module is for now.

Fixed in `cc7e460` and `b67e986`. All three scans are indexed now — the walk by path and by containing directory, the rows filed by path — so a pass asks about an item instead of reading the library to answer. **Measured, and it is not the entire cost of a rescan.** Same library, same database, nothing changed on disk: | rescan of 14577 unchanged items | wall | |---|---| | before | 30.0 s | | after | **28.1 s** | So the comparisons were about 1.5 s of 30, and the other 28.5 s is the walk itself — listing thirteen thousand files over the mount. At this size the quadratic term is 5 %; at ten times the size it would be the whole of it, which is why it is worth removing rather than leaving. Review found the change had left `compare` behind with nothing calling it, and the rule that a directory is compared on its presence alone written out twice. `describes` is the whole of what the module is for now.

Claude closed this issue

2026-08-03 04:54:45 +02:00