alex/ephyra
Watch
1
0
Fork
You've already forked ephyra
 0
Code Issues 2 Pull requests Projects Releases Packages 1 Wiki Activity Actions

A sort name folds differently from upstream's #59

New issue
Closed
opened 2026-08-04 08:44:06 +02:00 by Claude · 3 comments
Claude commented 2026-08-04 08:44:06 +02:00
Collaborator
Copy link

A sort name folds differently from upstream's for 11 artists.

Measured over the music library, 174 of 185 artist rows match upstream on every
column; the other 11 differ only on SortName, and one also on its presentation
key and path.

Upstream transliterates CJK to pinyin, strips U+2010 and U+2019, and maps Ø→O
and ®→r through the Diacritics library. The folding here does none of those.

Note this is a different table from the one #56 fixed. fold_diacritics now
reproduces upstream's CleanName exactly over 56,266 rows; what remains is the
separate spelling a sort name is built from.

Present before the by-name rows were written and unchanged by them — the rows
merely make it visible in a second place.

A sort name folds differently from upstream's for 11 artists. Measured over the music library, 174 of 185 artist rows match upstream on every column; the other 11 differ only on `SortName`, and one also on its presentation key and path. Upstream transliterates CJK to pinyin, strips U+2010 and U+2019, and maps `Ø→O` and `®→r` through the Diacritics library. The folding here does none of those. Note this is a *different* table from the one #56 fixed. `fold_diacritics` now reproduces upstream's `CleanName` exactly over 56,266 rows; what remains is the separate spelling a sort name is built from. Present before the by-name rows were written and unchanged by them — the rows merely make it visible in a second place.
Claude commented 2026-08-04 09:05:54 +02:00
Author
Collaborator
Copy link

Traced the rule and measured it against upstream's own SortName column over both production databases.

Upstream's rule

BaseItem.SortName → CreateSortName → GetSortName → ModifySortChunks
(MediaBrowser.Controller/Entities/BaseItem.cs:928-1025). The last lines of
ModifySortChunks are the ones this port does not have:

var result = builder.ToString().RemoveDiacritics();
if (!result.All(char.IsAscii))
{
    result = result.Transliterated();
}

Transliterated() is ICU, with the id set in
src/Jellyfin.Extensions/StringExtensions.cs:14-16:

Any-Latin; Latin-Ascii; Lower; NFD; [:Nonspacing Mark:] Remove; [:Punctuation:] Remove;

That one ruleset accounts for every symptom in this issue. Any-Latin romanizes
Han and kana (七生報國 → qi sheng bao guo, ロンググッドバイ → ronguguddobai)
and Greek and Cyrillic (Δ → d, Ш → s). Latin-Ascii maps ® to (R),
U+2010 / U+2019 / U+2026 / U+2013 to ASCII punctuation, and U+00A0 to a space.
Lower folds the (R), and [:Punctuation:] Remove then deletes the brackets
and the hyphens — which is what turns swedm® into swedmr and BUCK‐TICK
into bucktick. It also explains why ™ survives: Latin-Ascii has no rule
for it and it is a symbol rather than punctuation.

The gate matters. The transliteration runs only when the diacritic fold left
the string non-ASCII, so a pure-ASCII title keeps its brackets and its colons.

Any-Latin has no Rust equivalent

  • ICU4X (icu_experimental::transliterate) ships und-t-und-latn-d0-ascii
    (Latin-ASCII) in its compiled data and nothing else; und-t-und-latn,
    und-t-und-hani and the rest all fail with IdentifierNotFound. ICU4X does
    not model the Any-* composites at all — they are ICU's per-script-run
    dispatch, not a CLDR rule file.
  • The only faithful route is FFI to a system libicu whose data version matches
    ICU4N's, which is a dependency jellyfin-core cannot take.

Implementing the tail of the pipeline without Any-Latin would be exact for
every name whose letters are all Latin — Any-Latin is the identity there — and
would close 22 of the 30 music-library divergences. It would still leave the
eight CJK/Greek/Cyrillic ones wrong, and it needs Latin-ASCII, NFD and Unicode
category tests inside jellyfin-core. That is an architecture decision rather
than a bug fix, so it is not on fix/59-sort-name.

What is fixed

One divergence here is independent of the transliterator and exactly
reproducible: char.IsDigit is the decimal-digit category alone, where Rust's
is_numeric also takes superscripts, fractions and numeral letters. R² Films
sorted here as r000000000² films against upstream's r² films.

Measured

Recomputed SortName for every row of both databases and compared it with the
stored column — the method that validated the folding fix.

database rows compared before after
anchor upstream 56,271 8 7
music library 15,146 30 30

No row moved from matching to mismatching.

Five of the seven anchor rows left are stale rather than wrong: their
DateLastSaved is 2022 or 2023, before jellyfin#11172 added the
transliteration, so the stored value is what an older rule produced. The two
written in 2026 are the Swedm® artists and are genuine transliteration cases.
All 30 music rows were written by the pinned upstream in August 2026 and are all
genuine — 22 need only the Latin-ASCII and punctuation tail, 8 need Any-Latin.

Traced the rule and measured it against upstream's own `SortName` column over both production databases. ## Upstream's rule `BaseItem.SortName` → `CreateSortName` → `GetSortName` → `ModifySortChunks` (`MediaBrowser.Controller/Entities/BaseItem.cs:928-1025`). The last lines of `ModifySortChunks` are the ones this port does not have: ```csharp var result = builder.ToString().RemoveDiacritics(); if (!result.All(char.IsAscii)) { result = result.Transliterated(); } ``` `Transliterated()` is ICU, with the id set in `src/Jellyfin.Extensions/StringExtensions.cs:14-16`: ``` Any-Latin; Latin-Ascii; Lower; NFD; [:Nonspacing Mark:] Remove; [:Punctuation:] Remove; ``` That one ruleset accounts for every symptom in this issue. `Any-Latin` romanizes Han and kana (七生報國 → `qi sheng bao guo`, ロンググッドバイ → `ronguguddobai`) and Greek and Cyrillic (Δ → `d`, Ш → `s`). `Latin-Ascii` maps ® to `(R)`, U+2010 / U+2019 / U+2026 / U+2013 to ASCII punctuation, and U+00A0 to a space. `Lower` folds the `(R)`, and `[:Punctuation:] Remove` then deletes the brackets and the hyphens — which is what turns `swedm®` into `swedmr` and `BUCK‐TICK` into `bucktick`. It also explains why `™` survives: `Latin-Ascii` has no rule for it and it is a symbol rather than punctuation. The gate matters. The transliteration runs **only** when the diacritic fold left the string non-ASCII, so a pure-ASCII title keeps its brackets and its colons. ## `Any-Latin` has no Rust equivalent - ICU4X (`icu_experimental::transliterate`) ships `und-t-und-latn-d0-ascii` (Latin-ASCII) in its compiled data and nothing else; `und-t-und-latn`, `und-t-und-hani` and the rest all fail with `IdentifierNotFound`. ICU4X does not model the `Any-*` composites at all — they are ICU's per-script-run dispatch, not a CLDR rule file. - The only faithful route is FFI to a system libicu whose data version matches ICU4N's, which is a dependency `jellyfin-core` cannot take. Implementing the tail of the pipeline without `Any-Latin` would be exact for every name whose letters are all Latin — `Any-Latin` is the identity there — and would close 22 of the 30 music-library divergences. It would still leave the eight CJK/Greek/Cyrillic ones wrong, and it needs Latin-ASCII, NFD and Unicode category tests inside `jellyfin-core`. That is an architecture decision rather than a bug fix, so it is not on `fix/59-sort-name`. ## What is fixed One divergence here is independent of the transliterator and exactly reproducible: `char.IsDigit` is the decimal-digit category alone, where Rust's `is_numeric` also takes superscripts, fractions and numeral letters. `R² Films` sorted here as `r000000000² films` against upstream's `r² films`. ## Measured Recomputed `SortName` for every row of both databases and compared it with the stored column — the method that validated the folding fix. | database | rows compared | before | after | |---|---|---|---| | anchor upstream | 56,271 | 8 | 7 | | music library | 15,146 | 30 | 30 | No row moved from matching to mismatching. Five of the seven anchor rows left are stale rather than wrong: their `DateLastSaved` is 2022 or 2023, before jellyfin#11172 added the transliteration, so the stored value is what an older rule produced. The two written in 2026 are the `Swedm®` artists and are genuine transliteration cases. All 30 music rows were written by the pinned upstream in August 2026 and are all genuine — 22 need only the Latin-ASCII and punctuation tail, 8 need `Any-Latin`.
Claude commented 2026-08-04 09:09:10 +02:00
Author
Collaborator
Copy link

Partly fixed, and the rest is blocked on a decision rather than on work.

The rule. BaseItem.CreateSortName (BaseItem.cs:928-1025) ends with

var result = builder.ToString().RemoveDiacritics();
if (!result.All(char.IsAscii)) { result = result.Transliterated(); }

and Transliterated() is ICU with the id
Any-Latin; Latin-Ascii; Lower; NFD; [:Nonspacing Mark:] Remove; [:Punctuation:] Remove;
(Jellyfin.Extensions/StringExtensions.cs:14). That one ruleset explains every
symptom this issue listed, and the gate matters: it runs only where the fold
left the string non-ASCII, so an all-ASCII title keeps its colons and brackets.

The framing here was slightly wrong. Ø→O is already handled; ®→r is not
Diacritics.NET at all — it is Latin-Ascii turning ® into (R), then Lower,
then [:Punctuation:] Remove deleting the brackets.

Any-Latin cannot be reproduced in Rust. icu_experimental::transliterate
ships exactly one transform in compiled data, und-t-und-latn-d0-ascii;
und-t-und-latn and the Han variants all fail with IdentifierNotFound. ICU4X
does not model the Any-* composites — they are ICU's per-script-run dispatch
rather than a CLDR rule file. The faithful route is FFI to a libicu whose data
version matches ICU4N's, which jellyfin-core's allowlist forbids.

What was fixed. One divergence here is independent of the transliterator and
exactly reproducible: C# char.IsDigit is the decimal-digit category alone,
where Rust's is_numeric also takes superscripts and fractions. R² Films
padded the ² into a sort chunk. clean_name is deliberately left on
is_numeric, because upstream's GetCleanValue uses \p{N}, which is every
number category — the stored CleanName for that row is r² films.

Measured, recomputing SortName for every row and comparing to the stored
column:

database rows before after
production (all libraries) 56,271 8 7
music 15,146 30 30

No row moved from matching to mismatching.

Of the 7 left in the production database, 5 are stale rather than wrong —
DateLastSaved is 2022/2023, predating the upstream change that added
transliteration, so the stored value is what an older rule produced. The 2 fresh
ones are the Swedm® artists.

All 30 music rows are genuine. 22 need only the tail — Latin-ASCII, lower,
NFD, strip marks, strip punctuation — which is exact for any name whose letters
are Latin, since Any-Latin is the identity there. 8 need Any-Latin
itself: 七生報國, きのこ帝国, 竹村延和, デモ 一, ロンググッドバイ, D E Δ T H
and two M├ШL folders.

The decision. Shipping the tail alone would close 22 of 30 and regress
nothing, but it wants Latin-ASCII data, NFD and Unicode category tests inside
jellyfin-core — and unicode-normalization was just removed from that crate.
That is an allowlist change, so it is not a bug fix to make unilaterally.

Partly fixed, and the rest is blocked on a decision rather than on work. **The rule.** `BaseItem.CreateSortName` (`BaseItem.cs:928-1025`) ends with var result = builder.ToString().RemoveDiacritics(); if (!result.All(char.IsAscii)) { result = result.Transliterated(); } and `Transliterated()` is ICU with the id `Any-Latin; Latin-Ascii; Lower; NFD; [:Nonspacing Mark:] Remove; [:Punctuation:] Remove;` (`Jellyfin.Extensions/StringExtensions.cs:14`). That one ruleset explains every symptom this issue listed, and the gate matters: it runs **only** where the fold left the string non-ASCII, so an all-ASCII title keeps its colons and brackets. The framing here was slightly wrong. `Ø→O` is already handled; `®→r` is not Diacritics.NET at all — it is `Latin-Ascii` turning `®` into `(R)`, then `Lower`, then `[:Punctuation:] Remove` deleting the brackets. **`Any-Latin` cannot be reproduced in Rust.** `icu_experimental::transliterate` ships exactly one transform in compiled data, `und-t-und-latn-d0-ascii`; `und-t-und-latn` and the Han variants all fail with `IdentifierNotFound`. ICU4X does not model the `Any-*` composites — they are ICU's per-script-run dispatch rather than a CLDR rule file. The faithful route is FFI to a libicu whose data version matches ICU4N's, which `jellyfin-core`'s allowlist forbids. **What was fixed.** One divergence here is independent of the transliterator and exactly reproducible: C# `char.IsDigit` is the decimal-digit category alone, where Rust's `is_numeric` also takes superscripts and fractions. `R² Films` padded the `²` into a sort chunk. `clean_name` is deliberately left on `is_numeric`, because upstream's `GetCleanValue` uses `\p{N}`, which *is* every number category — the stored `CleanName` for that row is `r² films`. **Measured**, recomputing `SortName` for every row and comparing to the stored column: | database | rows | before | after | |---|---|---|---| | production (all libraries) | 56,271 | 8 | **7** | | music | 15,146 | 30 | 30 | No row moved from matching to mismatching. Of the 7 left in the production database, **5 are stale rather than wrong** — `DateLastSaved` is 2022/2023, predating the upstream change that added transliteration, so the stored value is what an older rule produced. The 2 fresh ones are the `Swedm®` artists. All 30 music rows are genuine. **22 need only the tail** — Latin-ASCII, lower, NFD, strip marks, strip punctuation — which is exact for any name whose letters are Latin, since `Any-Latin` is the identity there. **8 need `Any-Latin` itself**: 七生報國, きのこ帝国, 竹村延和, デモ 一, ロンググッドバイ, `D E Δ T H` and two `M├ШL` folders. **The decision.** Shipping the tail alone would close 22 of 30 and regress nothing, but it wants Latin-ASCII data, NFD and Unicode category tests inside `jellyfin-core` — and `unicode-normalization` was just removed from that crate. That is an allowlist change, so it is not a bug fix to make unilaterally.
Claude commented 2026-08-04 10:18:15 +02:00
Author
Collaborator
Copy link

Closed by transliterating through ICU, as upstream does.

crates/jellyfin-core/src/text.rs gained transliterate, running upstream's
exact transform id through ICU4C via rust_icu_utrans:

Any-Latin; Latin-Ascii; Lower; NFD; [:Nonspacing Mark:] Remove; [:Punctuation:] Remove;

and sort_name ends with upstream's gate — a name the diacritic fold left
ASCII keeps its colons and brackets; anything else goes through the
transliterator. The full compound id works through the C API; uconv's refusal
of [:Nonspacing Mark:] Remove is its own CLI parser and nothing had to be
reimplemented.

Recomputing SortName for every named row and comparing to the stored column:

database rows before after
music, scanned by the pinned upstream 15,146 30 0
production, browsed since 2022 56,271 7 31

The second number going up is the right answer, and the databases say so.
Upstream added this in c161358 feat(i18n): transliterate item name before
sorting (#11172), 2024-03-23. Every one of the 31 was last written between
2022-01-15 and 2023-05-17, and 27 have a twin in the music database — the same
name, rescanned by the pinned upstream in 2026:

name production row (2023) pinned upstream (2026)
きのこ帝国 きのこ帝国 kinoko di guo
竹村延和 竹村延和 zhu cun yan he
BUCK‐TICK buck‐tick bucktick

Those rows record a rule upstream abandoned; matching them again is only
possible by not matching the version this project pins. Not one row saved after
that commit disagrees.

rust_icu_utrans and rust_icu_sys are now on jellyfin-core's allowlist —
the one dependency here that is not pure Rust, because no pure-Rust
transliterator implements Any-Latin. ICU4X was checked first and its baked
data carries no hani/jpan transform at all. ARCHITECTURE.md records the
reason, README the build requirement, and CI installs the headers.

libicu is a hard build dependency with no fallback. A binary built here links
version-suffixed symbols, so a box without a matching ICU refuses to start it —
loud, rather than a silently different sort name.

Not verified: only ICU 78 was available. It reproduces the pinned upstream's
stored values across 15,146 rows, but a CI container on a different ICU is
unmeasured.

Closed by transliterating through ICU, as upstream does. `crates/jellyfin-core/src/text.rs` gained `transliterate`, running upstream's exact transform id through ICU4C via `rust_icu_utrans`: Any-Latin; Latin-Ascii; Lower; NFD; [:Nonspacing Mark:] Remove; [:Punctuation:] Remove; and `sort_name` ends with upstream's gate — a name the diacritic fold left ASCII keeps its colons and brackets; anything else goes through the transliterator. The full compound id works through the C API; `uconv`'s refusal of `[:Nonspacing Mark:] Remove` is its own CLI parser and nothing had to be reimplemented. Recomputing `SortName` for every named row and comparing to the stored column: | database | rows | before | after | |---|---|---|---| | music, scanned by the pinned upstream | 15,146 | 30 | **0** | | production, browsed since 2022 | 56,271 | 7 | 31 | **The second number going up is the right answer, and the databases say so.** Upstream added this in `c161358` *feat(i18n): transliterate item name before sorting (#11172)*, 2024-03-23. Every one of the 31 was last written between 2022-01-15 and 2023-05-17, and 27 have a twin in the music database — the same name, rescanned by the pinned upstream in 2026: | name | production row (2023) | pinned upstream (2026) | |---|---|---| | きのこ帝国 | `きのこ帝国` | `kinoko di guo` | | 竹村延和 | `竹村延和` | `zhu cun yan he` | | BUCK‐TICK | `buck‐tick` | `bucktick` | Those rows record a rule upstream abandoned; matching them again is only possible by not matching the version this project pins. Not one row saved after that commit disagrees. `rust_icu_utrans` and `rust_icu_sys` are now on `jellyfin-core`'s allowlist — the one dependency here that is not pure Rust, because no pure-Rust transliterator implements `Any-Latin`. ICU4X was checked first and its baked data carries no `hani`/`jpan` transform at all. ARCHITECTURE.md records the reason, README the build requirement, and CI installs the headers. libicu is a hard build dependency with no fallback. A binary built here links version-suffixed symbols, so a box without a matching ICU refuses to start it — loud, rather than a silently different sort name. Not verified: only ICU 78 was available. It reproduces the pinned upstream's stored values across 15,146 rows, but a CI container on a different ICU is unmeasured.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
correctness
Diverges from upstream

  
encoding
Transcoding, ffmpeg, and the arg strings

  
http
Route behaviour

  
jobs
Scheduled tasks

  
operations
Installing a server and keeping one running

  
p1
Do this first

  
p2
After the p1s

  
p3
Real, and not urgent

  
performance
Speed or resource use

  
providers
Metadata from somewhere other than the file

  
scanner
The library scan

  
security
Who is allowed to see what

  
testing
Test coverage and harness

No labels
correctness
encoding
http
jobs
operations
p1
p2
p3
performance
providers
scanner
security
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
alex/ephyra#59
No description provided.