A title is not searchable by its original name #56

New issue

Closed

opened 2026-08-03 23:37:11 +02:00 by Claude · 1 comment

Claude commented

2026-08-03 23:37:11 +02:00

Collaborator

A title is not searchable by the name it was released under.

Upstream matches OriginalTitle as well as Name, scoring such a row into the
lowest tier. Live: searchTerm=エッジランナーズ finds Cyberpunk: Edgerunners
upstream and nothing here.

Adding the clause alone would not fix that case. This server's
text::fold_diacritics strips every non-spacing mark after NFD, which destroys
Japanese dakuten — エッジ folds to エッシ — where upstream's folding leaves
non-Latin scripts alone. The two are coupled: the missing column and a folding
that damages the term before it is compared.

One title in the measured library has an OriginalTitle differing from its
Name, so the reach today is that single item; the folding is the wider of the
two.

A title is not searchable by the name it was released under. Upstream matches `OriginalTitle` as well as `Name`, scoring such a row into the lowest tier. Live: `searchTerm=エッジランナーズ` finds `Cyberpunk: Edgerunners` upstream and nothing here. Adding the clause alone would not fix that case. This server's `text::fold_diacritics` strips every non-spacing mark after NFD, which destroys Japanese dakuten — `エッジ` folds to `エッシ` — where upstream's folding leaves non-Latin scripts alone. The two are coupled: the missing column and a folding that damages the term before it is compared. One title in the measured library has an `OriginalTitle` differing from its `Name`, so the reach today is that single item; the folding is the wider of the two.

Claude commented

2026-08-04 08:42:42 +02:00

Author

Collaborator

A title is searchable by the name it was released under, and the folding no
longer damages the term.

Two coupled causes. Upstream matches CleanName or
OriginalTitle LIKE %term%, scoring an original-title-only hit into the lowest
tier by falling through the CASE. And its folding is not a decomposition:
StringExtensions.RemoveDiacritics calls Diacritics.NET, whose mapper is a
per-character dictionary lookup — a character not in the table is copied through
unchanged, and there is no NFD anywhere. This port used NFD and stripped every
non-spacing mark, which destroys Japanese dakuten and any script whose marks are
part of its letters.

fold_diacritics is now the same table, and the evidence is the whole library:
recomputing the folded name for all 56,266 rows of a production database
reproduces the stored CleanName with 0 mismatches. The NFD version got
76 wrong — Topsøe, Całun, Skjævesland, Đào, and every kana with a
dakuten.

searchTerm=エッジランナーズ now finds Cyberpunk: Edgerunners, as upstream
does; before it found nothing.

Not verified: the mapping table was extracted from a 4.1.x tree of
Diacritics.NET where upstream pins 4.1.8 and no source tag was fetchable. The
56k-row agreement is the evidence that the two are equivalent for every
character this library holds.

A title is searchable by the name it was released under, and the folding no longer damages the term. Two coupled causes. Upstream matches `CleanName` **or** `OriginalTitle LIKE %term%`, scoring an original-title-only hit into the lowest tier by falling through the `CASE`. And its folding is not a decomposition: `StringExtensions.RemoveDiacritics` calls Diacritics.NET, whose mapper is a per-character dictionary lookup — a character not in the table is copied through unchanged, and there is no NFD anywhere. This port used NFD and stripped every non-spacing mark, which destroys Japanese dakuten and any script whose marks are part of its letters. `fold_diacritics` is now the same table, and the evidence is the whole library: recomputing the folded name for all **56,266** rows of a production database reproduces the stored `CleanName` with **0** mismatches. The NFD version got **76** wrong — `Topsøe`, `Całun`, `Skjævesland`, `Đào`, and every kana with a dakuten. `searchTerm=エッジランナーズ` now finds `Cyberpunk: Edgerunners`, as upstream does; before it found nothing. Not verified: the mapping table was extracted from a 4.1.x tree of Diacritics.NET where upstream pins 4.1.8 and no source tag was fetchable. The 56k-row agreement is the evidence that the two are equivalent for every character this library holds.

Claude closed this issue

2026-08-04 08:42:42 +02:00

Claude referenced this issue

2026-08-04 08:44:06 +02:00

A search that asks for no media is answered with media #58

Claude referenced this issue

2026-08-04 08:44:06 +02:00

A sort name folds differently from upstream's #59