Streams and images are written one transaction per item #21

New issue

Closed

opened 2026-08-02 14:27:25 +02:00 by Claude · 1 comment

Claude commented

2026-08-02 14:27:25 +02:00

Collaborator

scan_libraries loops save_streams and save_images per detail
(crates/jellyfin-server/src/main.rs:265-275), and each opens and commits its
own transaction (jellyfin-db/src/scan.rs:260, :339).

The item rows themselves are one transaction (:80), so the shape for this
library is 1 + ~26 000 commits. Fold them into the item transaction, or chunk.

Found by scanning /mnt/media (285 movies, 289 shows, 74 music artists —
~13 000 media files on a CIFS mount) with both this server and upstream
Jellyfin at the pinned commit, remote metadata and image fetchers disabled on
upstream so both were reading files only. Upstream was given the same mount
path so MD5(UTF-16LE(path)) agrees and rows compare directly.

`scan_libraries` loops `save_streams` and `save_images` per detail (`crates/jellyfin-server/src/main.rs:265-275`), and each opens and commits its own transaction (`jellyfin-db/src/scan.rs:260`, `:339`). The item rows themselves are one transaction (`:80`), so the shape for this library is 1 + ~26 000 commits. Fold them into the item transaction, or chunk. --- Found by scanning `/mnt/media` (285 movies, 289 shows, 74 music artists — ~13 000 media files on a CIFS mount) with both this server and upstream Jellyfin at the pinned commit, remote metadata and image fetchers disabled on upstream so both were reading files only. Upstream was given the same mount path so `MD5(UTF-16LE(path))` agrees and rows compare directly.

Claude added the

scanner

performance

labels

2026-08-02 14:27:25 +02:00

Claude commented

2026-08-03 06:44:39 +02:00

Author

Collaborator

Fixed in eec99dc and 0b77eca.

Detail moved to core, where its four fields already lived, and a pass is one
transaction: the rows, the ancestor chains, and everything each row carries.

Measured on 14577 items with three streams each, against a file-backed
database:

	wall
four commits per item	1.79 s
one transaction	0.65 s

Not the fsync the issue's shape suggests — the pragmas are WAL with
synchronous = NORMAL, so a commit does not sync; what 58000 of them cost is
the per-transaction work around each.

Review found the change had made something worse before it made it better.
Writing the details in a transaction of their own, after the rows, means a
failure between the two leaves every row filed and none of them carrying
anything — and what decides that an item has already been scanned is the row,
so every later pass reads them as done and never repairs them. Where the old
shape lost the details of the items after the failure, the new one lost the
details of all of them.

0b77eca folds them into the same transaction, so a pass is atomic and this
case leaves #28's repair scope entirely. A detail that cannot be filed takes the
rows with it, which is under test.

Fixed in `eec99dc` and `0b77eca`. `Detail` moved to core, where its four fields already lived, and a pass is one transaction: the rows, the ancestor chains, and everything each row carries. **Measured on 14577 items with three streams each, against a file-backed database:** | | wall | |---|---| | four commits per item | 1.79 s | | one transaction | **0.65 s** | Not the fsync the issue's shape suggests — the pragmas are WAL with `synchronous = NORMAL`, so a commit does not sync; what 58000 of them cost is the per-transaction work around each. --- Review found the change had made something worse before it made it better. Writing the details in a transaction of their own, after the rows, means a failure between the two leaves every row filed and none of them carrying anything — and what decides that an item has already been scanned *is* the row, so every later pass reads them as done and never repairs them. Where the old shape lost the details of the items after the failure, the new one lost the details of all of them. `0b77eca` folds them into the same transaction, so a pass is atomic and this case leaves #28's repair scope entirely. A detail that cannot be filed takes the rows with it, which is under test.

Claude closed this issue

2026-08-03 06:44:39 +02:00