What a scan still writes differently from upstream #76

Closed
opened 2026-08-04 18:51:34 +02:00 by Claude · 1 comment
Collaborator

What a scan still writes differently from upstream, after #75.

Each of these was measured against upstream's own scan of /mnt/media
(var/big-upstream), and each was left deliberately rather than missed. They
are separate pieces of work, not one.

A by-name artist is never updated. MusicArtist rows differ on Genres
(110) and RunTimeTicks (118). ArtistMetadataService.cs:46 gives a by-name
artist — one with no parent, reached by name rather than by folder — the
cumulative runtime and the distinct genres of every item tagged with its name;
a folder artist gets neither. This port's by_name::rows only ever creates rows
that nothing stands for, so making it fill these wants the same
"does the row already say this" comparison albums have. Without that, a rescan
starts rewriting every artist and the write-nothing invariant goes.

Lyric streams are not extracted. 70 stream rows upstream wrote and this port
did not, one on each of 70 audio items. Upstream reads a lyrics-eng tag,
writes it to <metadata>/library/<xx>/<id>/<name>.lrc, and then resolves that
file back as a Lyric stream. It is a metadata-writing feature; this port
writes no metadata files.

Three names differ by a trailing space, and five the GazettE rows differ on
Path/Name/SortName/PresentationUniqueKey, plus one row on CleanName
and ParentIndexNumber. Pre-existing naming, unrelated to the columns #75
closed.

Out of scope rather than unfinished:

  • MusicArtist.Overview (37) — an artist biography from a metadata provider
    this project excludes. Not derivable from the files.
  • Person.DateModified (219) — upstream stamps a by-name row from the directory
    it has just created, so both dates are "now". Writing it would leave the same
    219 rows differing, exactly as DateCreated already does.
  • The playlists folder, the Live TV view and three MusicGenre rows upstream's
    artist refresh created: rows upstream makes at startup or from a provider,
    none of them from a scan.

How to re-measure. A scan being diffed wants the prober upstream ships —
with a distro ffprobe the stream columns diverge for reasons that are not the
port's, as #75 found. The first_video_frame line at startup says which you
got.

jellyfin-harness diff-db var/big-upstream/config/data/jellyfin.db \
    <a fresh scan of /mnt/media>
What a scan still writes differently from upstream, after #75. Each of these was measured against upstream's own scan of `/mnt/media` (`var/big-upstream`), and each was left deliberately rather than missed. They are separate pieces of work, not one. **A by-name artist is never updated.** `MusicArtist` rows differ on `Genres` (110) and `RunTimeTicks` (118). `ArtistMetadataService.cs:46` gives a by-name artist — one with no parent, reached by name rather than by folder — the cumulative runtime and the distinct genres of every item *tagged* with its name; a folder artist gets neither. This port's `by_name::rows` only ever creates rows that nothing stands for, so making it fill these wants the same "does the row already say this" comparison albums have. Without that, a rescan starts rewriting every artist and the write-nothing invariant goes. **Lyric streams are not extracted.** 70 stream rows upstream wrote and this port did not, one on each of 70 audio items. Upstream reads a `lyrics-eng` tag, writes it to `<metadata>/library/<xx>/<id>/<name>.lrc`, and then resolves that file back as a `Lyric` stream. It is a metadata-*writing* feature; this port writes no metadata files. **Three names differ by a trailing space**, and five `the GazettE` rows differ on `Path`/`Name`/`SortName`/`PresentationUniqueKey`, plus one row on `CleanName` and `ParentIndexNumber`. Pre-existing naming, unrelated to the columns #75 closed. **Out of scope rather than unfinished:** - `MusicArtist.Overview` (37) — an artist biography from a metadata provider this project excludes. Not derivable from the files. - `Person.DateModified` (219) — upstream stamps a by-name row from the directory it has just created, so both dates are "now". Writing it would leave the same 219 rows differing, exactly as `DateCreated` already does. - The playlists folder, the Live TV view and three `MusicGenre` rows upstream's artist refresh created: rows upstream makes at startup or from a provider, none of them from a scan. **How to re-measure.** A scan being diffed wants the prober upstream ships — with a distro ffprobe the stream columns diverge for reasons that are not the port's, as #75 found. The `first_video_frame` line at startup says which you got. jellyfin-harness diff-db var/big-upstream/config/data/jellyfin.db \ <a fresh scan of /mnt/media>
Author
Collaborator

The by-name artist is done. The lyric streams should not be done. Of the three names, this issue's own description of them was wrong, and one is fixed.

On main in 12464c0, 5c9dead, c5b6cc4 and e9559f0. /mnt/media is not mounted on this host, so nothing here was verified by a fresh scan — the evidence is upstream's own scan (var/big-upstream), this port's last scan of the same media, and tests.

A by-name artist is given what it stands for

MusicArtist.GetTaggedItems + ArtistMetadataService.cs:46, translated: every parentless MusicArtist under the metadata token, joined to the items carrying its CleanName as an Artist or AlbumArtist value, restricted to Audio/MusicVideo/MusicAlbum and IsFolder = 0, summing RunTimeTicks (a null counts as zero) and taking the genres distinct case-insensitively, first spelling kept. A row already carrying both is left alone, compared as the column stores them — so a rescan still writes nothing.

The rule was scored against the answer key alongside its plausible rivals, which are not close:

rule genres runtime
as implemented 93 / 119 100 / 119
AlbumArtist only 30 26
Artist only 90 96
without the IsFolder filter 87 35

The residues are upstream reading its own writing

All 19 runtime residues are exactly round(the computed sum ÷ 1 minute) × 1 minute — 19 of 19, from 3 minutes to 115. That is a proof rather than an excuse: it is the same sum, round-tripped through the whole-minute <runtime> of an artist.nfo upstream wrote and then read back. MergeData overwrites RunTimeTicks with the provider's, and BeforeSave — where the cumulative sum is computed — runs first. The timestamps agree: the 100 exact rows were refreshed at 11:59–12:02, the 19 rounded ones at 12:25 and later, after the NFOs were written at 12:00.

25 of the 26 genre residues carry an AudioDbArtist id, 23 also have a biography, and every one stores exactly one genre — the shape of a provider's artist genre, from a reference scan that had internet providers on. Two Door Cinema Club stores Indie while its only tagged item has no genres at all; children cannot produce that. (Correlation is perfect but the mechanism is inferred: 14 artists have an AudioDb id and match anyway.)

So the expected effect on the diff is Genres 110 → 26 and RunTimeTicks 118 → 19, and neither residue is reachable without writing metadata files or fetching from the internet — both excluded by design.

One thing this cannot claim: upstream emits no ORDER BY for the tagged items, so there is no order to reproduce. Ours orders by item name, which is stable and the best-scoring of those measured, and the code says so rather than claiming reproduction.

Lyric streams — recommend closing as out of scope

Upstream reads a lyrics-eng tag, writes <metadata>/library/<xx>/<id>/<name>.lrc, and on a later refresh resolves that file back as a Lyric stream. All 70 files exist under the answer key's metadata directory and match the 70 rows one to one.

Implementing it means building a metadata-file writer — a new outbound side effect with its own directory and rescan/delete semantics — which nothing else in this port needs, to produce 70 rows that exist only because upstream wrote the file it then read. The port already resolves .lrc files that are genuinely on disk beside the media.

That same decision also settles the 19 artist runtimes above. If it is ever reopened, it should be reopened as the port writes metadata files, with lyrics as one consumer.

The three names — the issue's description of them is wrong

They are not trailing spaces. The three Name differences are:

  • music / Music — the library's physical folder. Fixed. Upstream's EnsureName takes a directory's own name, so /mnt/media/tv is tv even when the library is called Shows; the answer key stores movies/music/tv against libraries Movies/Music/Shows. The committed fixture cannot show it, because there the two agree.
  • Mordez Moi / Feed_Me__Mordez_Moi — an audio file falling back to its stem. Not fixed, and it is not a naming bug: upstream reads title, track number and disc number as tag ?? probe, while our prober's tag map carries neither the title nor the two numbers, so a tag block ffprobe does not map leaves us with the stem. Adding them can move names on any of the 4,889 audio rows, which is not something to land blind — it wants the media mounted and a fresh scan diffed against the answer key.
  • the GazettE / The Gazette — not fixed, and not a name derivation at all. Neither database has folded-duplicate person rows; both write one row, and the two disagree only on which of two source spellings it carries. Upstream takes whichever the scan met first while saving a crediting item's people; we create people afterwards from a sorted distinct query, so we keep the byte-first spelling. Reproducing it means carrying the first-seen spelling per folded name through to the by-name step. The library has exactly two folded-duplicate names and the other agrees by luck.

The five the GazettE rows the issue lists are one row: Path, SortName and PresentationUniqueKey are each derived correctly from whatever Name holds.

Separately, the ????? artist's path is fixed (%MetadataPath%/artists/artists, id d37f8216…, both upstream's own): its name is five literal ?, which GetValidFilename empties, leaving it standing in the artists directory until BeforeMetadataRefresh rebases it. The rebase is the artist's alone — upstream has the identical unconditional rebase on person, genre, music genre and studio, and none of them reaches it because a by-name row of those types never gets the metadata refresh that performs it. The answer key proves it twice: ????? is stored unrebased as a person, and every Person row in it has DateLastRefreshed NULL.

What is left

Two items, both needing /mnt/media mounted to verify: the audio title/track/disc tag reading, and the first-seen person spelling. They are narrower than this issue and carry their own evidence, so they are #85 and this one closes here.

**The by-name artist is done. The lyric streams should not be done. Of the three names, this issue's own description of them was wrong, and one is fixed.** On `main` in `12464c0`, `5c9dead`, `c5b6cc4` and `e9559f0`. `/mnt/media` is not mounted on this host, so **nothing here was verified by a fresh scan** — the evidence is upstream's own scan (`var/big-upstream`), this port's last scan of the same media, and tests. ## A by-name artist is given what it stands for `MusicArtist.GetTaggedItems` + `ArtistMetadataService.cs:46`, translated: every parentless `MusicArtist` under the metadata token, joined to the items carrying its `CleanName` as an `Artist` or `AlbumArtist` value, restricted to `Audio`/`MusicVideo`/`MusicAlbum` and `IsFolder = 0`, summing `RunTimeTicks` (a null counts as zero) and taking the genres distinct case-insensitively, first spelling kept. A row already carrying both is left alone, compared as the column stores them — so a rescan still writes nothing. The rule was scored against the answer key alongside its plausible rivals, which are not close: | rule | genres | runtime | |---|---|---| | **as implemented** | **93 / 119** | **100 / 119** | | `AlbumArtist` only | 30 | 26 | | `Artist` only | 90 | 96 | | without the `IsFolder` filter | 87 | 35 | ### The residues are upstream reading its own writing **All 19 runtime residues are exactly `round(the computed sum ÷ 1 minute) × 1 minute`** — 19 of 19, from 3 minutes to 115. That is a proof rather than an excuse: it is the same sum, round-tripped through the whole-minute `<runtime>` of an `artist.nfo` upstream wrote and then read back. `MergeData` overwrites `RunTimeTicks` with the provider's, and `BeforeSave` — where the cumulative sum is computed — runs first. The timestamps agree: the 100 exact rows were refreshed at 11:59–12:02, the 19 rounded ones at 12:25 and later, after the NFOs were written at 12:00. **25 of the 26 genre residues carry an `AudioDbArtist` id**, 23 also have a biography, and every one stores exactly one genre — the shape of a provider's artist genre, from a reference scan that had internet providers on. `Two Door Cinema Club` stores `Indie` while its only tagged item has no genres at all; children cannot produce that. (Correlation is perfect but the mechanism is inferred: 14 artists have an `AudioDb` id and match anyway.) So the expected effect on the diff is `Genres` 110 → 26 and `RunTimeTicks` 118 → 19, and neither residue is reachable without writing metadata files or fetching from the internet — both excluded by design. One thing this cannot claim: upstream emits no `ORDER BY` for the tagged items, so there is no order to reproduce. Ours orders by item name, which is stable and the best-scoring of those measured, and the code says so rather than claiming reproduction. ## Lyric streams — recommend closing as out of scope Upstream reads a `lyrics-eng` tag, writes `<metadata>/library/<xx>/<id>/<name>.lrc`, and on a later refresh resolves that file back as a `Lyric` stream. All 70 files exist under the answer key's metadata directory and match the 70 rows one to one. Implementing it means building **a metadata-file writer** — a new outbound side effect with its own directory and rescan/delete semantics — which nothing else in this port needs, to produce 70 rows that exist only because upstream wrote the file it then read. The port already resolves `.lrc` files that are genuinely on disk beside the media. That same decision also settles the 19 artist runtimes above. If it is ever reopened, it should be reopened as *the port writes metadata files*, with lyrics as one consumer. ## The three names — the issue's description of them is wrong They are not trailing spaces. The three `Name` differences are: - `music` / `Music` — the library's physical folder. **Fixed.** Upstream's `EnsureName` takes a directory's own name, so `/mnt/media/tv` is `tv` even when the library is called `Shows`; the answer key stores `movies`/`music`/`tv` against libraries `Movies`/`Music`/`Shows`. The committed fixture cannot show it, because there the two agree. - `Mordez Moi` / `Feed_Me__Mordez_Moi` — an audio file falling back to its stem. **Not fixed**, and it is not a naming bug: upstream reads title, track number and disc number as `tag ?? probe`, while our prober's tag map carries neither the title nor the two numbers, so a tag block ffprobe does not map leaves us with the stem. Adding them can move names on any of the 4,889 audio rows, which is not something to land blind — it wants the media mounted and a fresh scan diffed against the answer key. - `the GazettE` / `The Gazette` — **not fixed, and not a name derivation at all.** Neither database has folded-duplicate person rows; both write one row, and the two disagree only on which of two source spellings it carries. Upstream takes whichever the scan met first while saving a crediting item's people; we create people afterwards from a sorted distinct query, so we keep the byte-first spelling. Reproducing it means carrying the first-seen spelling per folded name through to the by-name step. The library has exactly two folded-duplicate names and the other agrees by luck. The five `the GazettE` rows the issue lists are one row: `Path`, `SortName` and `PresentationUniqueKey` are each derived correctly from whatever `Name` holds. Separately, the `?????` artist's **path** is fixed (`%MetadataPath%/artists/artists`, id `d37f8216…`, both upstream's own): its name is five literal `?`, which `GetValidFilename` empties, leaving it standing in the artists directory until `BeforeMetadataRefresh` rebases it. The rebase is the artist's alone — upstream has the identical unconditional rebase on person, genre, music genre and studio, and none of them reaches it because a by-name row of those types never gets the metadata refresh that performs it. The answer key proves it twice: `?????` is stored unrebased as a person, and **every** `Person` row in it has `DateLastRefreshed` NULL. ## What is left Two items, both needing `/mnt/media` mounted to verify: the audio title/track/disc tag reading, and the first-seen person spelling. They are narrower than this issue and carry their own evidence, so they are #85 and this one closes here.
Sign in to join this conversation.
No description provided.