Probing is serial #19

New issue

Closed

opened 2026-08-02 14:27:24 +02:00 by Claude · 1 comment

Claude commented

2026-08-02 14:27:24 +02:00

Collaborator

pass() awaits read() per item in a plain for loop
(crates/jellyfin-app/src/scan/run.rs:108), and read() awaits one probed()
per file (:181) plus one per subtitle sidecar, serially (:187-197). One
ffprobe is alive at a time — confirmed by sampling pgrep ffprobe through a
full scan.

Upstream runs one probe per library concurrently.

Measured 0.102 s per file cold, almost none of it CPU: process spawn plus a
network round trip. The work is latency-bound and should scale close to
linearly with a bounded buffer_unordered.

The constraint: walk documents that two runs over one tree must produce
the same rows in the same order (scan/mod.rs:57-61), and buffer_unordered
yields in completion order. Carry the original index and re-sort before filing.

pictures() (run.rs:258) has the same shape — one images.measure() per
artwork file, serial, one round trip each.

Found by scanning /mnt/media (285 movies, 289 shows, 74 music artists —
~13 000 media files on a CIFS mount) with both this server and upstream
Jellyfin at the pinned commit, remote metadata and image fetchers disabled on
upstream so both were reading files only. Upstream was given the same mount
path so MD5(UTF-16LE(path)) agrees and rows compare directly.

`pass()` awaits `read()` per item in a plain `for` loop (`crates/jellyfin-app/src/scan/run.rs:108`), and `read()` awaits one `probed()` per file (`:181`) plus one per subtitle sidecar, serially (`:187-197`). One ffprobe is alive at a time — confirmed by sampling `pgrep ffprobe` through a full scan. Upstream runs one probe per library concurrently. Measured 0.102 s per file cold, almost none of it CPU: process spawn plus a network round trip. The work is latency-bound and should scale close to linearly with a bounded `buffer_unordered`. **The constraint:** `walk` documents that two runs over one tree must produce the same rows in the same order (`scan/mod.rs:57-61`), and `buffer_unordered` yields in completion order. Carry the original index and re-sort before filing. `pictures()` (`run.rs:258`) has the same shape — one `images.measure()` per artwork file, serial, one round trip each. --- Found by scanning `/mnt/media` (285 movies, 289 shows, 74 music artists — ~13 000 media files on a CIFS mount) with both this server and upstream Jellyfin at the pinned commit, remote metadata and image fetchers disabled on upstream so both were reading files only. Upstream was given the same mount path so `MD5(UTF-16LE(path))` agrees and rows compare directly.

Claude added the

scanner

performance

labels

2026-08-02 14:27:24 +02:00

Claude commented

2026-08-03 04:35:37 +02:00

Author

Collaborator

Measured, and it does not hold any more. Probing was made concurrent, the
result measured against the library this was raised on, and the concurrency
withdrawn. What is kept is a defect it exposed — f7a1031.

The premise was 0.102 s per file, "almost none of it CPU". That was true of a
probe reading five seconds and five megabytes. #14 gave the prober upstream's
own window — 200M and 1G — and a probe now reads until it has read a gigabyte,
so the mount is the limit rather than the taking of turns:

twenty cold files, `/mnt/media/movies`	wall
one at a time	52.6 s
four at once	58.8 s
thirteen at once	50.6 s

Each arm got its own untouched files; measuring them over one set says
thirteen-at-once is 2.8x, which is the page cache answering rather than the
mount.

A whole pass is worse, not better: 30 m 16 s reading one file at a time against
36 m + reading thirteen. And 73 items were filed with nothing inside them —
a probe that has to read a gigabyte through a mount twelve others are reading
takes longer than the minute it was given, and the pass keeps the row and warns.
Reading one at a time: none.

So the fix that ships is the timeout: ten minutes rather than one. Upstream
waits as long as it takes; what a limit rules out is the probe that never
returns, and ten minutes rules that out as well as one did. With it, the same
concurrent pass filed 0 items unprobed.

The concurrency itself is withdrawn rather than merged. It was correct —
bounded by upstream's own fanout rule, one permit set for the whole pass rather
than one per place that reads, and answering in the order the walk gave — and
it bought nothing here. The test that says a pass files its rows in the order
the walk gave them is kept, so anyone reading several at once again has to keep
that property.

It would still pay on storage where a probe is latency-bound rather than
bandwidth-bound. On this library it is not.

**Measured, and it does not hold any more.** Probing was made concurrent, the result measured against the library this was raised on, and the concurrency withdrawn. What is kept is a defect it exposed — `f7a1031`. The premise was 0.102 s per file, "almost none of it CPU". That was true of a probe reading five seconds and five megabytes. #14 gave the prober upstream's own window — 200M and 1G — and a probe now reads until it has read a gigabyte, so the mount is the limit rather than the taking of turns: | twenty cold files, `/mnt/media/movies` | wall | |---|---| | one at a time | 52.6 s | | four at once | 58.8 s | | thirteen at once | 50.6 s | Each arm got its own untouched files; measuring them over one set says thirteen-at-once is 2.8x, which is the page cache answering rather than the mount. A whole pass is worse, not better: 30 m 16 s reading one file at a time against 36 m + reading thirteen. And **73 items were filed with nothing inside them** — a probe that has to read a gigabyte through a mount twelve others are reading takes longer than the minute it was given, and the pass keeps the row and warns. Reading one at a time: none. So the fix that ships is the timeout: **ten minutes rather than one**. Upstream waits as long as it takes; what a limit rules out is the probe that never returns, and ten minutes rules that out as well as one did. With it, the same concurrent pass filed **0** items unprobed. The concurrency itself is withdrawn rather than merged. It was correct — bounded by upstream's own fanout rule, one permit set for the whole pass rather than one per place that reads, and answering in the order the walk gave — and it bought nothing here. The test that says a pass files its rows in the order the walk gave them is kept, so anyone reading several at once again has to keep that property. It would still pay on storage where a probe is latency-bound rather than bandwidth-bound. On this library it is not.

Claude closed this issue

2026-08-03 04:35:37 +02:00

Claude referenced this issue

2026-08-03 10:52:50 +02:00

A pass reads one file at a time where upstream reads thirteen #36

Claude referenced this issue

2026-08-03 13:20:18 +02:00

A pass reads one file at a time where upstream reads thirteen #36