alex/ephyra

Watch

Fork 0

A music library re-asks four databases and re-downloads its covers on every scan #155

New issue

Closed

opened 2026-08-16 21:44:05 +02:00 by Claude · 1 comment

Claude commented

2026-08-16 21:44:05 +02:00

Collaborator

Every scan re-asks all four databases about every album and every artist, and
re-downloads the artwork it already has.

Two independent causes, and either alone is enough.

The folder is never short-circuited. crates/jellyfin-app/src/scan/summary.rs:437
calls folders(&on_record, asking.is_some()), which bypasses the describes()
short-circuit whenever any library has fetch_metadata on. So
plan.rs::describe runs asked_about for every MusicAlbum and MusicArtist
on every pass — MusicBrainz for the release, the release group and the artist,
then AudioDb for each — with MusicBrainz paced at one request a second by
jellyfin_fetch::Live.

The picture is not seen as already there. saved()
(crates/jellyfin-app/src/scan/run/remote.rs:276) builds its already set from
beside, which comes from read()'s pass over the library listing. A cover
fetched on the last scan lives at metadata/library/<shard>/folder.jpg, which
is not in that listing, so the kind is never marked taken and the file is
fetched and rewritten every pass.

A 500-album library therefore spends on the order of 2000 requests and ~35
minutes of pacing per scan, indefinitely, and rewrites every cover it already
had. A film does not have this problem: it goes through read(), whose
already is the pictures found beside the file.

Where to start

The two halves are separable and the second is the cheaper one. For the
pictures, the question is what already should be — the listing plus what the
metadata directory holds for that item, which is what ImageStore::modified
already answers for a portrait in stood_up.

For the folder, the question is what describes() should mean for a row whose
databases have already answered. Note the constraint that makes it non-obvious:
an album is identified by ids its tracks carry, so the row has to be
summarised before anything can be asked — which is why asked_about runs where
it does.

How to know it is done

A second scan of an unchanged music library makes no request and writes no
picture, and a third one after a track's tags change still asks. The pair in
crates/jellyfin-app/tests/scan_enriched.rs that rescans (rescanning/again)
is the shape this is measured in.

House rules

CLAUDE.md is the short version and these are the parts that bite: tests run
under cargo nextest; upstream's source is the checkout in upstream/ at the
pin (v12.0-rc5, 6d501ba4) and not the clone beside the repository;
generated files are regenerated rather than edited; a comment says what the code
does, not how it got there; and Dn references stay out of code and comments.

Every scan re-asks all four databases about every album and every artist, and re-downloads the artwork it already has. Two independent causes, and either alone is enough. **The folder is never short-circuited.** `crates/jellyfin-app/src/scan/summary.rs:437` calls `folders(&on_record, asking.is_some())`, which bypasses the `describes()` short-circuit whenever *any* library has `fetch_metadata` on. So `plan.rs::describe` runs `asked_about` for every `MusicAlbum` and `MusicArtist` on every pass — MusicBrainz for the release, the release group and the artist, then AudioDb for each — with MusicBrainz paced at one request a second by `jellyfin_fetch::Live`. **The picture is not seen as already there.** `saved()` (`crates/jellyfin-app/src/scan/run/remote.rs:276`) builds its `already` set from `beside`, which comes from `read()`'s pass over the *library listing*. A cover fetched on the last scan lives at `metadata/library/<shard>/folder.jpg`, which is not in that listing, so the kind is never marked taken and the file is fetched and rewritten every pass. A 500-album library therefore spends on the order of 2000 requests and ~35 minutes of pacing per scan, indefinitely, and rewrites every cover it already had. A film does not have this problem: it goes through `read()`, whose `already` is the pictures found beside the file. ## Where to start The two halves are separable and the second is the cheaper one. For the pictures, the question is what `already` should be — the listing plus what the metadata directory holds for that item, which is what `ImageStore::modified` already answers for a portrait in `stood_up`. For the folder, the question is what `describes()` should mean for a row whose databases have already answered. Note the constraint that makes it non-obvious: an album is identified by ids its *tracks* carry, so the row has to be summarised before anything can be asked — which is why `asked_about` runs where it does. ## How to know it is done A second scan of an unchanged music library makes no request and writes no picture, and a third one after a track's tags change still asks. The pair in `crates/jellyfin-app/tests/scan_enriched.rs` that rescans (`rescanning`/`again`) is the shape this is measured in. ### House rules `CLAUDE.md` is the short version and these are the parts that bite: tests run under `cargo nextest`; upstream's source is the checkout in `upstream/` **at the pin** (`v12.0-rc5`, `6d501ba4`) and not the clone beside the repository; generated files are regenerated rather than edited; a comment says what the code does, not how it got there; and `Dn` references stay out of code and comments.

Claude referenced this issue

2026-08-16 21:45:09 +02:00

A person has no biography, and a box set takes nothing from TMDb #107

Claude referenced this issue

2026-08-16 21:45:09 +02:00

OMDb is not written: no ratings, no critic scores, no episode fallback #108

Claude referenced this issue

2026-08-16 21:45:09 +02:00

Music metadata is entirely absent: MusicBrainz and AudioDb are both in the box upstream #109

Claude added the

scanner

performance

labels

2026-08-20 17:31:11 +02:00

Claude referenced this issue

2026-08-20 19:48:49 +02:00

A series is asked about on every scan, and its id is resolved again first #158

Claude referenced this issue

2026-08-20 19:49:46 +02:00

A person is asked about on every RefreshPeople run: DateLastRefreshed is never stamped #154

Claude commented

2026-08-20 19:49:46 +02:00

Author

Collaborator

Landed in b45ce06, with corrections in b5de663.

The folder was never short-circuited — and the reason it was not turned out
to be deeper than the asking flag. A folder the pass did not read carries a
bare row, so describes() was comparing provider-enriched stored data against
a summary of the tracks and could never hold again once anything had answered.
The flag was not the cause, it was the workaround.

So the comparison had to change, not just its guard. A row a database has
answered about — DateLastRefreshed, now carried on Tracked — is asked to
carry what the tracks say rather than to equal it, where the pass read nothing
under the folder. Where it read something the lists are compared exactly,
because a track can lose a value as easily as gain one. The running time is
exact either way, which is what a track leaving is noticed by.

The cover was not seen as already there. saved built what is taken from
the library listing, and an album's cover lives under the metadata directory,
which no listing carries. It asks the store now, the way stood_up already did
for a portrait. A kind with nowhere to be written is also settled before the
fetch rather than after, so the bytes are no longer downloaded and dropped.

Two regressions the review caught, both mine and both in the first version
of the gate:

An album that lost a track was never described again. The gate required both
that the pass had disturbed the folder and that the row had drifted — and
nothing reads a file that is gone. The row would have kept the deleted track's
running time forever.
A database that could not be reached stamped the row anyway, and the stamp is
what the short-circuit reads — so one unreachable pass meant that folder was
never asked about again. fetched_music reports whether it reached them and
an unreached folder is left unstamped, which is the distinction #154 already
made for a person.

On measuring it. The first version of the rescan test asserted "asks nothing
and rewrites nothing" on one pass and passed — then passed again with the
picture fix reverted, because once the folder is short-circuited that pass never
reaches the artwork at all. The artwork is asserted on the rescan that does
ask instead. Six tests, each checked against the mutation it exists to catch.

One test was deleted rather than made to pass: it asserted an album drops a
genre when a track is re-tagged, which contradicts a rule this suite already
documents — the genres are the row's, not the tag's.

What is left is filed as #158: a rescan of an unchanged library is now
silent except for three requests, all of them a series. A film is quiet.

Landed in `b45ce06`, with corrections in `b5de663`. **The folder was never short-circuited** — and the reason it was not turned out to be deeper than the `asking` flag. A folder the pass did not read carries a *bare* row, so `describes()` was comparing provider-enriched stored data against a summary of the tracks and could never hold again once anything had answered. The flag was not the cause, it was the workaround. So the comparison had to change, not just its guard. A row a database has answered about — `DateLastRefreshed`, now carried on `Tracked` — is asked to *carry* what the tracks say rather than to equal it, where the pass read nothing under the folder. Where it read something the lists are compared exactly, because a track can lose a value as easily as gain one. The running time is exact either way, which is what a track *leaving* is noticed by. **The cover was not seen as already there.** `saved` built what is taken from the library listing, and an album's cover lives under the metadata directory, which no listing carries. It asks the store now, the way `stood_up` already did for a portrait. A kind with nowhere to be written is also settled before the fetch rather than after, so the bytes are no longer downloaded and dropped. **Two regressions the review caught**, both mine and both in the first version of the gate: - An album that lost a track was never described again. The gate required both that the pass had disturbed the folder and that the row had drifted — and nothing reads a file that is gone. The row would have kept the deleted track's running time forever. - A database that could not be reached stamped the row anyway, and the stamp is what the short-circuit reads — so one unreachable pass meant that folder was never asked about again. `fetched_music` reports whether it reached them and an unreached folder is left unstamped, which is the distinction #154 already made for a person. **On measuring it.** The first version of the rescan test asserted "asks nothing *and* rewrites nothing" on one pass and passed — then passed again with the picture fix reverted, because once the folder is short-circuited that pass never reaches the artwork at all. The artwork is asserted on the rescan that *does* ask instead. Six tests, each checked against the mutation it exists to catch. One test was deleted rather than made to pass: it asserted an album drops a genre when a track is re-tagged, which contradicts a rule this suite already documents — the genres are the row's, not the tag's. What is left is filed as **#158**: a rescan of an unchanged library is now silent except for three requests, all of them a series. A film is quiet.

Claude closed this issue

2026-08-20 19:49:46 +02:00