A pass reads one file at a time where upstream reads thirteen #36

New issue

Closed

opened 2026-08-03 10:52:50 +02:00 by Claude · 1 comment

Claude commented

2026-08-03 10:52:50 +02:00

Collaborator

A pass reads one file at a time. Upstream reads several: its scheduler fans out
to Environment.ProcessorCount - 3 where the setting is unset, dropping to
sequential only on three cores or fewer.

MediaBrowser.Controller/LibraryTaskScheduler/LimitedConcurrencyLibraryScheduler.cs
    return Environment.ProcessorCount - 3;

On the reference machine — sixteen cores, LibraryScanFanoutConcurrency unset
— that is thirteen at once against our one.

The server is not busy while it scans. Sampled during a pass over the film
library, jellyfin-server sits at 0.0% CPU: every second of it is spent
waiting on one ffprobe reading one file over the mount.

Measured directly, three disjoint cold sets of 24 films and two of 24 episodes,
the same arguments a pass uses:

	serial	thirteen at once
24 films	138 s, 145 s	55 s
24 episodes	5 s	2 s

So about 2.5x, which is the mount saturating rather than the cores.

Against the library that matters — 284 films, 7904 episodes, 4889 tracks:

	upstream	ours
first scan	36 min	measured serially, projected around 62 min
rescan	79 s	3 s for the music library

Reading is where a pass spends itself and the only place fanning out pays. The
writing is already batched and bounded (#22), and a rescan that reads nothing
is already fast.

Note that #19 proposed concurrency here and was closed without it, on the
measurement that probing was not latency-bound once #14 widened the probe
window. That measurement was taken on the reference library, whose files are
small; on the real one the widened window is exactly what makes each read long
enough to be worth overlapping. The numbers above supersede it.

Upstream's own policy is the one to copy, including the sequential case: a
scan that spawns thirteen ffprobe processes on a two-core machine is worse
than one that spawns one.

A pass reads one file at a time. Upstream reads several: its scheduler fans out to `Environment.ProcessorCount - 3` where the setting is unset, dropping to sequential only on three cores or fewer. ``` MediaBrowser.Controller/LibraryTaskScheduler/LimitedConcurrencyLibraryScheduler.cs return Environment.ProcessorCount - 3; ``` On the reference machine — sixteen cores, `LibraryScanFanoutConcurrency` unset — that is thirteen at once against our one. The server is not busy while it scans. Sampled during a pass over the film library, `jellyfin-server` sits at **0.0% CPU**: every second of it is spent waiting on one `ffprobe` reading one file over the mount. Measured directly, three disjoint cold sets of 24 films and two of 24 episodes, the same arguments a pass uses: | | serial | thirteen at once | |---|---|---| | 24 films | 138 s, 145 s | 55 s | | 24 episodes | 5 s | 2 s | So about 2.5x, which is the mount saturating rather than the cores. Against the library that matters — 284 films, 7904 episodes, 4889 tracks: | | upstream | ours | |---|---|---| | first scan | 36 min | measured serially, projected around 62 min | | rescan | 79 s | 3 s for the music library | Reading is where a pass spends itself and the only place fanning out pays. The writing is already batched and bounded (#22), and a rescan that reads nothing is already fast. Note that #19 proposed concurrency here and was closed without it, on the measurement that probing was not latency-bound once #14 widened the probe window. That measurement was taken on the reference library, whose files are small; on the real one the widened window is exactly what makes each read long enough to be worth overlapping. The numbers above supersede it. Upstream's own policy is the one to copy, including the sequential case: a scan that spawns thirteen `ffprobe` processes on a two-core machine is worse than one that spawns one.

Claude commented

2026-08-03 13:20:18 +02:00

Author

Collaborator

Fixed in 2423758.

What changed

A pass reads several files at once and files what it read in the order the
library resolved it — a folder before what is under it, and a chain naming
only rows already filed. How many at once is max(4, cores - 3), or what
library_scan_fanout_concurrency asks for, which the dashboard can now set.

Not upstream's rule, and the reason is the two regimes:

a probe	per file	bound by
off local storage	0.138s wall for 0.137s of processor	the processor
off the share	2–6s, the server at 0.0% of one	waiting

So the cores are right where the parsing is the cost, and say nothing where
the waiting is. The floor is what covers the second: a machine with few cores
and its library on a share is what ProcessorCount - 3 answers worst, and one
file at a time there costs hours. Recorded as D25.

While measuring: a picture was being read whole into memory to take its width
and height, which one read at a time hid.

Measured

/mnt/media — 284 films, 7904 episodes, 4889 tracks — against the upstream
instance's own logs for the same media:

	upstream	before	after
first scan	36m23s	past 4h	24.4 min
rescan	1m19s	—	29s
the music library alone	—	336s	161s

The rows are the same either way: 14573 paths against upstream's 14572 — the
one extra is a file added since upstream last scanned — differing only on 102
Name values that already differed for #27. The music library scanned both
serially and fanned out gives byte-identical rows.

What the link can do is 105 MB/s, and a single reader already reaches it. So
the fanout is not buying bandwidth; it is stopping the link idling while each
probe parses what it has. That is why this is 1.5x rather than 13x, and why
more than a handful buys nothing on a mount like this one.

Note on #19

That issue proposed concurrency here and was closed without it, on a
measurement that probing was not latency-bound once #14 widened the probe
window. The measurement was taken on the reference library, whose files are
small. On a real one the widened window is what makes each read long enough to
be worth overlapping. The numbers above supersede it.

Fixed in 2423758. ## What changed A pass reads several files at once and files what it read in the order the library resolved it — a folder before what is under it, and a chain naming only rows already filed. How many at once is `max(4, cores - 3)`, or what `library_scan_fanout_concurrency` asks for, which the dashboard can now set. Not upstream's rule, and the reason is the two regimes: | a probe | per file | bound by | |---|---|---| | off local storage | 0.138s wall for 0.137s of processor | the processor | | off the share | 2–6s, the server at 0.0% of one | waiting | So the cores are right where the parsing is the cost, and say nothing where the waiting is. The floor is what covers the second: a machine with few cores and its library on a share is what `ProcessorCount - 3` answers worst, and one file at a time there costs hours. Recorded as D25. While measuring: a picture was being read whole into memory to take its width and height, which one read at a time hid. ## Measured `/mnt/media` — 284 films, 7904 episodes, 4889 tracks — against the upstream instance's own logs for the same media: | | upstream | before | after | |---|---|---|---| | first scan | 36m23s | past 4h | **24.4 min** | | rescan | 1m19s | — | **29s** | | the music library alone | — | 336s | **161s** | The rows are the same either way: 14573 paths against upstream's 14572 — the one extra is a file added since upstream last scanned — differing only on 102 `Name` values that already differed for #27. The music library scanned both serially and fanned out gives byte-identical rows. What the link can do is 105 MB/s, and a single reader already reaches it. So the fanout is not buying bandwidth; it is stopping the link idling while each probe parses what it has. That is why this is 1.5x rather than 13x, and why more than a handful buys nothing on a mount like this one. ## Note on #19 That issue proposed concurrency here and was closed without it, on a measurement that probing was not latency-bound once #14 widened the probe window. The measurement was taken on the reference library, whose files are small. On a real one the widened window is what makes each read long enough to be worth overlapping. The numbers above supersede it.

Claude closed this issue

2026-08-03 13:20:18 +02:00